The public key cryptosystem is considered as a modern direction of cryptography since 1976. It contains two main problems; they are the factorization of the large number problem, and the knapsack problem. One of the representations of the factorization problem is the Rivest, Shamir, Adleman (RSA) public key cryptosystem (that is produced to solve many problems associated with conventional secret key cryptosystems, related to the problem of key generation and key distribution),which is widely used in the special secure commercial communication systems, like the banking systems. Although these systems have higher security in comparison with the conventional cipher systems, like stream cipher system, but they are attacked from many attacking methods. One of the methods is using the Genetic Algorithm in attacking public key cryptosystem. This attacking had a success in the attack of the knapsack public key cryptosystem, but till know; there are no publications about using such attacking method against the RSA public key cryptosystem. This thesis provides (for the first time) two proposals of using genetic algorithm in cryptanalysis of RSA public key, in two main directions. The first direction is by cryptanalysis using cipher text only attack. While the second direction is by factorization of the modular value part of the public key values of RSA cryptosystem to obtain the prime number (p) and (q) that formulate the value of n from their multiplication and they will be used to find the value of secret key (d). Both methods provide at general good result from the point of time implemented view in comparison with the conventional attacking methods.