 
BN_generate_prime(3)
BN_generate_prime, BN_is_prime, BN_is_prime_fasttest  generate primes and
test for primality
#include <openssl/bn.h>
BIGNUM *BN_generate_prime(BIGNUM *ret, int num, int safe, BIGNUM *add,
BIGNUM *rem, void (*callback)(int, int, void *), void *cb_arg);
int BN_is_prime(const BIGNUM *a, int checks, void (*callback)(int, int,
void *), BN_CTX *ctx, void *cb_arg);
int BN_is_prime_fasttest(const BIGNUM *a, int checks,
void (*callback)(int, int, void *), BN_CTX *ctx, void *cb_arg,
int do_trial_division);
BN_generate_prime() generates a pseudorandom prime number of num
bits. If ret is not NULL, it will be used to store the number.
If callback is not NULL, it is called as follows:

callback(0, i, cb_arg) is called after generating the ith potential prime number.

While the number is being tested for primality, callback(1, j,
cb_arg) is called as described below.

When a prime has been found, callback(2, i, cb_arg) is called.
The prime may have to fulfill additional requirements for use in
DiffieHellman key exchange:
If add is not NULL, the prime will fulfill the condition p % add
== rem (p % add == 1 if rem == NULL) in order to suit a given generator.
If safe is true, it will be a safe prime (i.e. a prime p so that (p1)/2 is also
prime).
The PRNG must be seeded prior to calling BN_generate_prime().
The prime number generation has a negligible error probability.
BN_is_prime() and BN_is_prime_fasttest() test if
the number a is prime. The following tests are performed until one of them shows that
a is composite; if a passes all these tests, it is considered prime.
BN_is_prime_fasttest(), when called with do_trial_division == 1, first attempts trial division by a number of small primes; if no divisors
are found by this test and callback is not NULL,
callback(1, 1, cb_arg) is called. If do_trial_division == 0, this test is skipped.
Both BN_is_prime() and BN_is_prime_fasttest()
perform a MillerRabin probabilistic primality test with checks iterations. If
checks == BN_prime_checks, a number of iterations is used that yields a false positive rate of at
most 2^80 for random input.
If callback is not NULL, callback(1, j, cb_arg) is called after the jth iteration (j = 0, 1, ...). ctx is a preallocated BN_CTX (to save the overhead of allocating and freeing the structure in a loop),
or NULL.
BN_generate_prime() returns the prime number on success, NULL otherwise.
BN_is_prime() returns 0 if the number is composite, 1 if it is
prime with an error probability of less than 0.25^checks, and 1 on error.
The error codes can be obtained by ERR_get_error(3).
bn(3), ERR_get_error(3), rand(3)
The cb_arg arguments to BN_generate_prime() and to
BN_is_prime() were added in SSLeay 0.9.0. The ret argument to BN_generate_prime() was added in SSLeay 0.9.1.
BN_is_prime_fasttest() was added in OpenSSL 0.9.5.
  