How to Create an SSH Key in Linux: Step by Step Guide for Secure Server Access

Introduction

Managing Linux servers securely is a top priority for system administrators and developers alike. At ServerMO, we always recommend using SSH keys over traditional passwords to connect to your dedicated servers, because security and efficiency matter. SSH (Secure Shell) is a protocol used to access remote servers through an encrypted connection. Instead of typing a password each time you connect, you can use SSH keys for fast, secure, and passwordless logins.

In this guide, we’ll walk you through the simple steps to create and use SSH keys on a Linux system.

What Are SSH Keys and How Do They Work?

SSH servers can authenticate users in multiple ways, with password-based login being the simplest, yet least secure. While passwords are transmitted in a protected manner, they often lack the complexity needed to defend against brute-force attacks. Automated tools can easily crack weak credentials, posing a major risk.

In contrast, SSH key authentication is significantly more secure.

When you try to connect to a remote server, the SSH protocol uses this key pair to verify your identity. If the server confirms that the private key matches the stored public key, you’re granted access — no password needed. The public key is placed inside the user's home directory on the server, specifically in the ~/.ssh/authorized_keys file. When you initiate a connection, the server checks whether your machine holds the correct private key. If validated, the connection is established securely.

To add an extra layer of protection, the private key can also be encrypted on your device with a passphrase. This means even if someone gets hold of your private key, they still need the passphrase to use it.

Step 1 : Generate SSH Key Pair on Your Local Machine

To begin securing access to your Linux server with SSH key authentication, your first task is to generate a new SSH key pair directly on your local system. This can be easily done using the ssh-keygen tool, which comes bundled with most Linux distributions as part of the OpenSSH package. Unless specified otherwise, this utility will generate a 3072-bit RSA key pair, which is a robust standard for secure authentication.

By default, the system suggests saving your new keys in the ~/.ssh/ directory under your home folder. The private key will be saved as id_rsa, and the corresponding public key will be id_rsa.pub.

If you already have an SSH key with the same name, you’ll be prompted like this:

The tool may also output:

Finally, you’ll have the option to secure your private key with a passphrase. This step is optional but recommended. A passphrase encrypts your private key locally, adding another layer of protection in case your machine is compromised.

Why Use an SSH Key with a Passphrase?

At this point, you might be asking yourself, “If I still need to type a passphrase, what’s the real benefit of using SSH keys?” It’s a great question, and here’s why SSH keys, even with a passphrase, offer superior security compared to traditional passwords:

1. Your Private Key Never Leaves Your Machine

The private SSH key, which can be optionally protected with a passphrase, never travels over the network. Instead, the SSH protocol uses cryptographic methods to verify your identity. Since the private key is decrypted locally and never sent out, brute-force attacks over the network become impossible, even if someone is sniffing traffic.

2. Access is Restricted by Permissions

By default, private SSH keys must be stored in a secure directory, typically ~/.ssh/, and with strict file permissions (600 or owner-only read/write). SSH clients are smart enough to ignore insecure private keys, making unauthorized snooping by other local users extremely difficult.

3. Attacker Must Already Have System Access

If someone wants to crack your SSH passphrase, they already need access to your device or server — a major escalation from remote attacks. Even in this case, the passphrase acts as a final defense wall, giving you a small but crucial window to rotate your keys and revoke the compromised one before more damage is done.

4. Passphrase = Extra Security Layer

The passphrase doesn’t replace the private key — it adds an extra layer of protection in case your private key is somehow leaked or stolen. When used correctly, this offers a defense-in-depth approach, which is a cornerstone of modern Linux server security.

When you finish creating the key pair, you’ll see output like:

+---[RSA 3072]----+
|o   ..oo.++o ..  |
| o o +o.o.+...   |
|. . + oE.o.o  .  |
| . . oo.B+  .o   |
|  .   .=S.+ +    |
|      . o..*     |
|        .+= o    |
|        .=.+     |
|       .oo+      |
+----[SHA256]-----+
                                

Congratulations — you now have a fully functional public/private SSH key pair on your local machine.

Step 2 : Uploading Your SSH Public Key to Your ServerMO Server

Before you can enjoy passwordless SSH logins, you’ll need to upload your public SSH key to your remote server. There are a few different ways to do this, and the best option depends on your system setup and available tools. Whether you're using an automated method like ssh-copy-id, a traditional SSH command, or copying it manually, all approaches have the same end goal: store your public key in the ~/.ssh/authorized_keys file on your remote ServerMO server.

Let’s walk through each method in detail.

Option 1: Copy Your Key Using ssh-copy-id (Recommended)

The easiest and most automated way to copy your public key to a remote server is using the ssh-copy-id utility. It’s often pre-installed on most Linux systems (part of the OpenSSH packages).

• You must have password-based SSH access
• The utility must be available on your local machine

• username with your SSH login (commonly root)
• your_server_ip with your ServerMO server’s IP address

First-Time Connection Message:

You may see this prompt when connecting to a new server:

Type yes and press ENTER.

Enter your SSH password (input will be hidden for security). After verification, the utility will automatically:

• Locate your public key (~/.ssh/id_rsa.pub)
• Create the .ssh directory if it doesn’t exist
• Append your key to the authorized_keys file on the remote server

Congratulations! Your key is now installed and ready to use.

Option 2: Copying Your Key via Standard SSH Command

If ssh-copy-id isn’t available, you can upload your key manually using a one-liner that utilizes SSH and shell redirection.

This command:

• Displays your public key
• Connects to the server via SSH
• Ensures the .ssh directory exists
• Appends your key to authorized_keys without overwriting other keys

Just Like Before:

• You’ll be prompted to confirm authenticity and enter your password.
• Once authenticated, your key will be securely added to the server.

Option 3: Manually Upload Your Public Key

If you don’t have direct SSH access with a password (e.g., your server is in recovery mode or accessed via a web console), you can manually copy and paste your public key.

Step 1: Show Your Public Key (Local Machine)

You’ll see output like this:

Copy the entire key string to your clipboard.

Step 2: Access Remote Server Console

Use your hosting provider’s web-based console or direct terminal access to log into your ServerMO server.

Step 3: Prepare the .ssh Directory

Step 4: Paste the Public Key

Run the following command on the remote server:

Be sure to replace the placeholder with the full key you copied earlier (it usually starts with ssh-rsa or ssh-ed25519)

Final Step: Test Your SSH Key Login

From your local system, try logging into your server:

If everything is set up correctly, you’ll connect without being asked for a password.

ServerMO Security Tip:

We always recommend setting strict permissions after uploading SSH keys. On your server, run:

This ensures only the appropriate user can access your SSH configurations.

Step 3 : Logging Into Your Server Using SSH Keys

Once your public SSH key has been successfully copied to your server using any of the previous methods, you’re ready to log in securely, without a password.

Step 3 : Logging Into Your Server Using SSH Keys

Open your terminal and use the following SSH command to initiate the login:

If this is your first time connecting to the server, you may see a message like this:

This simply means your local system hasn't seen this server before. Type yes and press ENTER to continue.

Successful Key-Based Authentication

• If your private key has no passphrase, you’ll be logged in immediately.
• If you set a passphrase when generating your SSH key, you’ll be prompted to enter it now for verification.

Once authenticated, your terminal will open a secure shell session with your server. Now that SSH key authentication is working, it's time to take your server's security a step further.

Step 4 : Securing Your Server by Disabling Password-Based SSH Login

While you now have secure SSH key access, password authentication is still active by default, which leaves your server open to brute-force attacks. Disabling it locks down unauthorized login attempts.

Important: Before You Proceed

• SSH key-based login works flawlessly.
• You have access to a user with sudo privileges.
• You’re not locked out after disabling password login.

Once confirmed, let’s proceed.

1. Log Into Your Server via SSH

Use your SSH key to connect:

2. Edit the SSH Configuration File

Use your favorite text editor to modify the SSH daemon config file:

Locate the line that says #PasswordAuthentication yes and change it to:

If the line is commented out (starts with #), remove the # to uncomment it.

3. Save and Restart the SSH Service

To apply the change, restart the SSH daemon:

Congratulations! Your ServerMO-hosted machine now accepts only key-based authentication, blocking password attacks at the gate.

Optional: Using Hardware Security Modules (HSMs) for Maximum SSH Key Security

If your infrastructure demands military-grade protection, consider using a Hardware Security Module (HSM). HSMs store your private SSH keys in tamper-resistant devices rather than files, providing unmatched security.

Why Use an HSM?

• Private keys never leave the hardware.
• Safe even if the server is compromised.
• Helps meet compliance requirements (e.g., HIPAA, PCI DSS).

How to Set Up SSH Authentication Using an HSM

1. Ensure Your HSM is SSH-Compatible

Confirm your device supports the PKCS#11 standard.

2. Generate SSH Key Inside the HSM

Use the opensc-pkcs11.so module:

ssh-keygen -D /usr/lib/opensc-pkcs11.so -s user-hsm-key
3. Extract the Public Key
ssh-keygen -D /usr/lib/opensc-pkcs11.so -e > ~/.ssh/id_hsm.pub
4. Add the Public Key to Your Remote Server

Append it to the authorized_keys file of your server’s user account.

5. Configure SSH to Use Your HSM

Edit (or create) your SSH config file:

nano ~/.ssh/config
Add:
Host *
IdentityAgent /run/user/1000/gnupg/S.gpg-agent.ssh

Save and exit.

Now, SSH will use the hardware-backed key for authentication by default.

Final Thoughts from ServerMO

Securing your dedicated server is not just a good practice—it’s a necessity. Whether you're running high-performance AMD or Intel systems, hosting storage-heavy applications, or relying on 10 Gbps connectivity, ensuring encrypted, key-based access is foundational. By using SSH keys—and optionally HSMs—you’re fortifying your ServerMO environment against unauthorized access and aligning with enterprise-grade security standards.

Next Up: Learn how to set up a firewall to further protect your infrastructure.