RSA_sign, RSA_verify - RSA signatures
int RSA_sign(int type, const unsigned char *m, unsigned int m_len, unsigned char *sigret, unsigned int *siglen, RSA *rsa);
int RSA_verify(int type, const unsigned char *m, unsigned int m_len, unsigned char *sigbuf, unsigned int siglen, RSA *rsa);
type denotes the message digest algorithm that was used to generate m. It usually is one of NID_sha1, NID_ripemd160 and NID_md5; see objects(3) for details. If type is NID_md5_sha1, an SSL signature (MD5 and SHA1 message digests with PKCS #1 padding and no algorithm identifier) is created.
The error codes can be obtained by ERR_get_error(3).
Certain signatures with an improper algorithm identifier are accepted for compatibility with SSLeay 0.4.5 :-)
SSL, PKCS #1 v2.0