The Private Key, as the name indicates, should be kept secret. Often, access to it is restricted by further encrypting the key using a passphrase and a SymmetricCipher. Balancing the need to keep the private key both useful and secret is one of the challenges faced in using PublicKeyEncryption.
For example, in order to encrypt or sign content with a private key, the key must be accessible on the system in which those documents are handled. If that content is to be transmitted in some way over a network, though, then that same computer probably has a network connection and might be vulnerable to network-based exploits through which the private key can be discovered by someone other than the legitimate key holder.
The Private Key is used to place a DigitalSignature on a document, to give some verifiable assurance that the document so signed originates from the holder of the Private Key.
The private key is also used by the holder of that key to decrypt messages encoded with the complementary PublicKey, messages intended by the sender only for cleartext viewing by the holder of the private key.