Hi Dwayne,<br><br>Please see inline.<br><br><div class="gmail_quote">2013/2/18 Dwayne Litzenberger <span dir="ltr"><<a href="mailto:dlitz@dlitz.net" target="_blank">dlitz@dlitz.net</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
[Reposted from 'Hash: Remove "oid" attributes; add "name" attribute'<br>
<a href="https://github.com/dlitz/pycrypto/commit/a3ec589b8dcd1c86ddd5f35666e74aa3230801b5" target="_blank">https://github.com/dlitz/<u></u>pycrypto/commit/<u></u>a3ec589b8dcd1c86ddd5f35666e74a<u></u>a3230801b5</a>]:<br>
<br>
Legrandin wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Dwayne,<br>
<br>
The Object ID is an identifier assigned by (inter)national standard bodies (NIST) or recognized private organizations (RSA Inc, Teletrust) to the hash algorithm for use in all the several crypto protocols based on ASN.1 (PKCS#1 signatures, PKCS#7/CMS, PKCS#8 private key encapsulation, SSL/TLS, CA certificates, etc). Nothing stops one from also using without ASN.1, as a stand-along numerical string guaranteed to be unique.<br>
<br>
The fact that a few other protocols don't use it (and prefer to have their own internal identifiers, and therefore not leverage work done by others already) does not look to me as a reason to isolate it in the PKCS#1v1.5 signature module, considering that protocols that use it are the majority, and all hashes currently in pycrypto have it (being them all quite mature).<br>
<br>
The attribute could also remained undefined for those experimental hash that pycrypto ever introduced but that don't have any Object ID assigned yet (e.g. Salsa20 maybe?). That would just mean that the hash cannot be used to make PKCS#1v1.5 signatures (which makes sense). If the OID exists, it can be added to the module. It it doesn't, it is not defined.<br>
</blockquote>
<br>
My response:<br>
<br>
Be very careful with your use of Object Identifiers. In many cases there are a<br>
great many OIDs available for the same algorithm, but the exact OID you're<br>
supposed to use varies somewhat.<br>
-- Peter Gutmann, X.509 Style Guide, <a href="http://www.cs.auckland.ac.nz/%7Epgut001/pubs/x509guide.txt" target="_blank">http://www.cs.auckland.ac.nz/~<u></u>pgut001/pubs/x509guide.txt</a><br>
<br></blockquote><div><br>Does any of the hash algorithm in the library have more than 1 OID?<br>If not, maybe this quote was referring to something else?<br> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Protocols that use [OIDs] are the majority<br>
</blockquote>
<br>
Really? The only widely-used protocols I can think of are SNMP (which is irrelevant here), LDAP (also irrelevant), and the CMS/TLS/PKCS protocol suite. SSH doesn't use them, OpenPGP doesn't use them, DNSSEC doesn't use them, OAuth doesn't use them, OpenID doesn't use them, DKIM doesn't use them, and I'm pretty sure that IPsec/IKEv1/IKEv2 don't use them. Of the protocols that do use them, which ones actually use the OIDs listed in this commit, rather than some ciphersuite identifier like pbeWithSHA1AndDES-CBC? Have there been any major new crypto protocols designed in the last decade that use these OIDs? That use ASN.1? </blockquote>
<div></div><div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
This is going to be a bit of a long rant.<br>
<br>
OIDs and ASN.1 are legacy ITU-T crap, and the protocols built around them are overcomplicated and error-prone. The only reason why I merged any ASN.1 stuff at all is because PKCS#1 uses it. PKCS#1 is a bit of a special case, because it's basically synonymous with RSA; It's even used protocols that don't otherwise use ASN.1 use PKCS#1.<br>
</blockquote><div><br>I think we are throwing the baby with the dirty water here.<br><br>I agree ITU created a lot of bloated standards and protocols.<br>ASN.1 got a bad fame mostly because of that, and even though bells and whistles have been added over time due to design-by-committee, its core remains very simple and elegant. BER/DER encoding in particular is very handy for binary serialization (even outside of the crypto context); it could be summarized in 3 or 4 pages only and still cover 95% of the use cases one could ever need. Even the famous, short layman guide could be trimmed down a lot [2]. To me, BER/DER is just a rock solid binary TLV with a compact schema format (which even XML never had until RELAX NG).<br>
<br>Google's ProtocolBuffers, Facebook's Thrift and several others have been created very recently to cover the same problem space; they ended up re-inventing the same wheel, getting the abstraction wrong, and still without the simplicity of BER/DER [3].<br>
<br>Now, in the crypto world, ASN.1 DER has been the encoding of choice exactly because it is simple, clear, efficient, and unambiguous. I stress "efficient" in that crypto is also done by resource constrained applications like embedded/industrial devices, sensors, smart cards, crypto tokens (all things that have serious trouble processing a bit of HTTP or XML).<br>
<br>Sure, horrible things have been built with ASN.1, but that's true for anything.<br>XML-DSIG is a good example [4], which does *not* prove that XML is bad per se.<br><br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I think it was you who convinced me that the ASN.1 used by PKCS#1 was simple enough that it wouldn't lead to an endless series of bugs. Even so, you *still* got it wrong, as described in LP#1119552 [1] . I'm not blaming you; I'm blaming ASN.1 for being such a terrible, complicated, obfuscatory way to define and describe data formats. Hell, the only reason why you got it wrong was because *so many other people got it wrong early on that the spec was modified to accommodate their errors*. And PKCS#1 is a much *simpler* use-case of ASN.1 compared to the rest of the CMS/TLS/PKCS suite...<br>
</blockquote><div><br>I ignored 3 lines in Appendix B of RFC 3447 (page 54, out of 70+).<br><br>Would have it made any different if the encoding had been XML, Json, or some custom application-specific format? I don't think so.<br>
<br>It has more to do with the fact that any 20+ years old format (PKCS#1) always has some quirks. That, and I was not good enough to read the whole the RFC.<br>But not really a good example for why ASN.1 is bad.<br>
<br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
In contrast, PyCrypto *needs* to be kept simple, because we simply don't have the developer resources to create a secure CMS/TLS/PKCS implementation. Even if we had the resources, getting it right is tricky enough that we *shouldn't* try to make yet another implementation---especially not one that's Python-specific.</blockquote>
<div><br>I think that asn1 module serves the purpose of simplicity, because:<br>a) the code that uses it (PKCS#1/#5/#8) is more compact and readable (at least to me, and compared to what it would be w/o the asn1 module), and more importantly<br>
b) I consider PKCS#1/#5/#8 fundamental for a base crypto library. I consider a library w/o them even harmful.<br><br>Having said that, let me derail a bit to say that I agree that TLS doesn't belong into PyCrypto because it is way above than "basic crypto". I never looked enough into CMS to have an opinion about it, but its RFC is shorter than PKCS1, so I don't have the feeling it's actually complicated.<br>
I don't understand what "PKCS implementation" means though (in the same way I would not know what "RFC" implementation is). All PKCS standards vary in scope and use.<br>
PKCS#1 is just a standardized way to do RSA, because otherwise any
program would do it differently and cryptography in
application would be years behind (ElGamal anybody?). PKCS#5 is a standardized way to
derive keys from passwords. I believe they deserve to be in a basic crypto library, since I could not imagine working w/o them (they also turned into RFC for a reason).<br>
Other PKCS specs don't because they are focused on very specific use cases (PKCS#11, for secure tokens) or are total crap (PKCS#12).<br><br>In short, the expression "CMS/TLS/PKCS" you use all over your email is binding together too many unrelated things.<br>
<br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">It would be better to pool our limited resources with other FOSS crypto developers to improve the existing implementations, or maybe to try to recruit them to work on a new project that would become the successor to the existing implementations. One more insecure, resource-starved FOSS CMS/TLS/PKCS implementation is not good for users.<br>
</blockquote><div><br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
OpenSSL exists today, and there are several ways to use it from Python. The purpose of PyCrypto is not to reimplement everything that OpenSSL already does. What would be gained by doing that? If we just wanted to make a nicer, more Pythonic API for OpenSSL, we could just add OpenSSL as a dependency and be done with it. (Python itself already uses OpenSSL for hashlib, so it's not unprecedented.)<br>
</blockquote><div><br>A hard dependency on OpenSSL would make my life difficult for quite a few reasons:<br>* Its license is neither LGPL-like nor BSD-like and it forces one to advertise its presence (deserved credit, but awkward to do)<br>
* It is cumbersome to cross-compile<br>* Its API is very complex and inconsistent<br>* It is difficult to predict if my target platform will have the openssl library, and if it does, which version of it<br>* It is rather Windows-unfriendly (not that pycrypto itself is much different though...)<br>
* Finally, I don't like to put all eggs in the same basket. Today, a bug in OpenSSL can easily cause unpredictable chain reactions because it is used too much by too many people [7] (in other projects, I prefer other TLS libraries also for that reason). Put differently, I am all for some level of ecosystem diversity when it comes to security.<br>
<br>The reason I started using PyCrypto is that nothing better existed for python (e.g. like BouncyCastle for Java, Crypto++ for C++, or .NET crypto services), apart from odd wrappers to C libraries (if I wanted that, I would stick to C++) which also increased my list of external dependencies (being self-contained is also very valuable).<br>
<br>The only alternative is keyczar, which keeps too much stuff under the bonnet for what I need to do.<br><br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
PyCrypto is used by a lot of folks who are either implementing recently-created protocols (i.e. *not* CMS/TLS/PKCS), or who are---rightly or wrongly---creating new protocols. One of my goals with PyCrypto has been to improve their chances of building something secure, and to me that means that I should steer people to simpler, easy-to-implement building blocks like OpenPGP and SSH, not complex, error-prone things like ASN.1/CMS/TLS/X.509/PKCS#12.<br>
</blockquote><div><br>I think a good deal of PyCrypto users fly under your radar (embedded SW, sys admin scripts, test frameworks, crypto workbenches). They don't develop new protocols, they just implement established ones (and not those web-oriented like OAuth).<br>
<br>I would also not agree in putting the awful PKCS#12 besides all the other protocols you list. If I want to do a PKI, there is no true alternative today to X.509, which - in its PKIX definition - is pretty straightforward actually. You cannot use neither OpenPGP nor SSH in that they adopt different security models (resp. web of trust and opportunistic authentication).<br>
<br>Having said that, isn't the goal of "steer[ing] people to simpler, easy-to-implement building blocks" exactly the same as keyczar (or nacl, not sure if a wrapper exists for it laready)? Why does pycrypto exist then?<br>
<br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I want to avoid turning PyCrypto into something that treats CMS/TLS/PKCS as the gold standard and everything else as a second-class citizen. There have already been a few cases of that (for example, the "oid" attribute here, and the "pkcs" parameter to RSA.exportKey), and I see those things as oversights that need to be fixed, not things that I want to entrench further.<br>
</blockquote><div><br>Just to clear things up, the primary reason I added "oid" was to allow one to pass a hash instance to PKCS115_SigScheme.sign() and have the method to automatically pick the correct OID. Having "oid" as attribute of the hash object seemed to me pretty natural (and neutral) choice. I considered the dictionary with hash names, and I was not thrilled by its elegance, but beauty is in the eye of the beholder. It was truly for practical convenience of the library user; no surreptitious plans to have evil asn.1 take over. ;-)<br>
<br>The "pkcs" parameter came up because PKCS/DER *is* the gold standard for exporting an RSA key. PGP key format exists simply because it was designed at the dawn of time. SSH key format is application specific (to say, is there even a spec for v1?).<br>
Any other key format is truly boutique variety.<br> </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I see that you've been building a PKCS#8 implementation in your fork of the PyCrypto repo. I can only assume that you eventually plan to build a PKCS#7/CMS implementation, too. That's fine, but seeing things like `algos = { 'PBKDF2WithHMAC-SHA1AndDES-EDE3-CBC' : _PBES2_Factory(_PBKDF2_Factory(), _DES_EDE3_CBC_Factory()) }` convinces me that it's beyond the scope of what I want to include in PyCrypto, unless it were in a well-isolated subdirectory that could be easily split into a separate package if the maintenance became too burdensome for me. At a minimum, we'd need to agree that the string "X.509" doesn't belong in the module that implements the raw RSA primitive.<br>
</blockquote><div><br>It actually never crossed my mind to develop any PKCS#7/CMS code (as I say above, I've never looked into it), but it's not clear to me why you despise it so much, apart from being ASN.1 encoded?<br>
<br>The thing is, I see PKCS#8 as belonging more with primitives.<br><br>Two of the biggest limitations of PyCrypto were (and up to a point, still are) interoperability with other systems and basic key management. The former brings value, the latter *must* be done right, because it is critical for security and it is often overlooked in favour of key lengths and algorithms-of-the-day.<br>
<br>In particular, the way keys are encoded (for exchange, storage, etc) plays a big role to both aspects; encoding needs to be agreed upon, secure, platform-independent, free from misunderstandings and so on. I recall that in PyCrypto 2.1 pickling was the only way, and that was wrong on so many levels up to the point it was a security threat by itself.<br>
<br>I decided to spend a good deal of time at making key management (export/import) easier and more secure. I started with PKCS#1 as the low hanging fruit, but it still only allowed storing private keys in the clear, which is pretty bad; it is good practice to have private keys always encrypted at least by a pass-phrase, especially if you plan to share them. PKCS#8 is *the* standard for protecting private asym. keys, so to me is appropriate to have it in a core library, rather than an optional one.<br>
<br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
You've done a lot of good work and I appreciate your contributions, but IMHO you're embedding the PKCS stuff too deeply into the core of PyCrypto when I'd prefer to see it in separate subdirectory, or even a separate library. This is partly my fault: I was a bit too anxious to merge the PKCS#1 stuff after being absent for a while, so I didn't pay close enough attention to the API changes (even though the API is really what differentiates PyCrypto from other libraries). In the future, I'm going to try to be more picky upfront about the API, to avoid backpedaling like I'm doing right now with the .oid stuff.<br>
</blockquote><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
As I see it, the PKCS1 stuff probably should have been consolidated into something like Crypto.Protocol.PKCS1. Going forward, the PKCS8 stuff should probably go into something like Crypto.Protocol.PKCS8, and a future OpenPGP package could go into Crypto.Protocol.OpenPGP. RSA.importKey and RSA.exportKey should probably be deprecated and moved into the PKCS1 and PKCS8 packages, respectively.<br>
<br>
The exact names of the subtrees are debatable, but the idea is create a clear separation between the primitives and the protocols that use them, rather than mixing them all together. This is particularly important for the hash modules, since those could eventually become thin wrappers around the standard hashlib library---I doubt that would ever happen if we insisted on attaching extraneous things like OIDs to them.<br>
</blockquote><div><br>Don't worry, I am happy to be told "move this stuff elsewhere" or even better "this stuff is crap, get it out of my way, you dumbass" when I am proposing some changes.<br>Getting stuck to half-baked APIs is a major pain; they always need careful attention and vetting.<br>
<br>However, the meaning of "protocol" is rather wide. I am afraid that stuffing everything under Crypto.Protocol leads to major confusion.<br> <br>I did some thinking before proposing RSA changes (some ended up on the ML [5]), and I still believe today that:<br>
<br>* Crypto.Signature is a good place for PKCS#1 signature routines. Signatures are protocols, but they are so important that it's debatable they should end up in the generic "bucket" that Crypto.Protocol is. Additionally, Crypto.Signature resembles the JCA and BouncyCastle style.<br>
* Crypto.Cipher is a good place for PKCS#1 encryption routines. "Cipher" is any protocol that performs a keyed transformation aimed at confidentiality. Again it is somewhat similar to JCA.<br>* PKCS#1 data structures (e.g. RSAPublicKey, RSAPrivateKey, etc) and unencrypted PEM are more encodings than protocols because they don't achieve any security objective (see definition of "(cryptographic) protocol" 1.55 in HAC [6]). I proposed them in Crypto.PublicKey.RSA simply because they are basic actions you can perform with a key, and I could get nice one-liners with them. JCA also had something similar (getEncoded). They could have also belonged to another new module (e.g. Crypto.IO?) but I would be wary of having them in something so generic of Crypto.Protocol.<br>
<br>Put differently, I don't think it adds value to have a PKCS1 module, just because all the above things are defined in one standard called PKCS#1. What makes one's code cleaner and easier to understand should be the key factor.<br>
<br>Now, I agree PKCS#8 and Encrypted PEM could be seen as protocols, but something like Crypto.IO is more self-explanatory than Crypto.Protocol.I am not thrilled by PublicKey.PKCS8 either.<br>
<br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Again, sorry for the long message, but I wanted to explain my thinking as clearly as possible. Let me know what you think.<br></blockquote><div><br>Thanks for the time you spend to put it together actually.<br> </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Cheers,<br>
- Dwayne<br>
<br>
[1] <a href="https://bugs.launchpad.net/pycrypto/+bug/1119552" target="_blank">https://bugs.launchpad.net/<u></u>pycrypto/+bug/1119552</a><span><font color="#888888"><br>
<br>
-- <br>
Dwayne C. Litzenberger <<a href="mailto:dlitz@dlitz.net" target="_blank">dlitz@dlitz.net</a>><br>
OpenPGP: 19E1 1FE8 B3CF F273 ED17 4A24 928C EC13 39C2 5CF7<br>
______________________________<u></u>_________________<br>
pycrypto mailing list<br>
<a href="mailto:pycrypto@lists.dlitz.net" target="_blank">pycrypto@lists.dlitz.net</a><br>
<a href="http://lists.dlitz.net/cgi-bin/mailman/listinfo/pycrypto" target="_blank">http://lists.dlitz.net/cgi-<u></u>bin/mailman/listinfo/pycrypto</a><br>
</font></span></blockquote></div><br>[2] <a href="http://luca.ntop.org/Teaching/Appunti/asn1.html" target="_blank">http://luca.ntop.org/Teaching/Appunti/asn1.html</a><br>[3] <a href="http://stackoverflow.com/questions/4633611/what-are-the-key-differences-between-apache-thrift-google-protocol-buffers-mes" target="_blank">http://stackoverflow.com/questions/4633611/what-are-the-key-differences-between-apache-thrift-google-protocol-buffers-mes</a><br>
[4] <a href="http://www.cs.auckland.ac.nz/%7Epgut001/pubs/xmlsec.txt" target="_blank">http://www.cs.auckland.ac.nz/~pgut001/pubs/xmlsec.txt</a><br>
[5] <a href="http://lists.dlitz.net/pipermail/pycrypto/2011q1/000418.html" target="_blank">http://lists.dlitz.net/pipermail/pycrypto/2011q1/000418.html</a><br>
[6] <a href="http://cacr.uwaterloo.ca/hac/about/chap1.pdf" target="_blank">http://cacr.uwaterloo.ca/hac/about/chap1.pdf</a><br>[7] <a href="http://www.schneier.com/blog/archives/2008/05/random_number_b.html" target="_blank">http://www.schneier.com/blog/archives/2008/05/random_number_b.html</a> (Debian OpenSSL branch)<br>
<br>
<br><br>