Streamline Remote Access: Disabling Key Authentication for RustDesk Server on Windows

RustDesk emerges as a compelling open-source alternative to TeamViewer, offering a secure and user-friendly self-hosting solution for remote desktop access. This guide focuses on deploying your own RustDesk server and specifically explores how to disable key authentication on a Windows server environment, simplifying access for specific use cases.

Understanding RustDesk and Key Authentication

RustDesk is designed to be a versatile remote access tool, boasting cross-platform compatibility across Windows, macOS, Linux, and Android. It prioritizes security through encrypted connections and delivers high performance with low latency, ensuring a smooth remote operation experience. Key features include multi-display support, file transfer, clipboard sharing, and flexible connection options. Its ease of use is a significant advantage, allowing users to quickly establish connections without complex configurations.

By default, RustDesk employs key authentication for enhanced security. This involves verifying keys between the server and clients to ensure authorized access. However, in certain controlled environments, such as internal networks or for simplified testing, disabling key authentication can streamline the connection process. This article will guide you through deploying a RustDesk server using Docker and configuring it to operate without key authentication.

Prerequisites

Before proceeding, ensure you have the following:

Windows Server: A Windows Server instance to host your RustDesk server.
Docker Desktop for Windows: Docker Desktop installed and configured on your Windows Server.
Basic Docker Knowledge: Familiarity with Docker and Docker Compose is helpful.

Deploying RustDesk Server without Key Authentication on Windows

While the original article uses Linux and Baota Panel, we’ll adapt the Docker deployment for a Windows Server environment. We’ll utilize Docker Compose for straightforward setup.

Create an Application Directory:

On your Windows Server, create a dedicated directory for your RustDesk server files. For example, C:rustdesk-server.

Create docker-compose.yml Configuration File:

Inside the C:rustdesk-server directory, create a file named docker-compose.yml and paste the following configuration. Crucially, we will remove the -k _ parameter to disable key authentication.

version: '3'
networks:
  rustdesk-net:
    external: false
services:
  hbbs:
    container_name: hbbs
    ports:
      - "21115:21115"
      - "21116:21116" # TCP
      - "21116:21116/udp" # UDP
    image: rustdesk/rustdesk-server:latest
    command: hbbs -r your_domain_or_ip:21117 # Replace with your server's domain or IP
    volumes:
      - C:rustdesk-server:/root # Windows path for volume mount
    networks:
      - rustdesk-net
    depends_on:
      - hbbr
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 64M
  hbbr:
    container_name: hbbr
    ports:
      - "21117:21117"
    image: rustdesk/rustdesk-server:latest
    command: hbbr # -k _ removed to disable key auth
    volumes:
      - C:rustdesk-server:/root # Windows path for volume mount
    networks:
      - rustdesk-net
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 64M

Important Notes:

your_domain_or_ip: Replace your_domain_or_ip with your Windows Server’s public IP address or domain name that resolves to your server.
-k _ Removal: Notice the command: hbbr line in the hbbr service. We have removed -k _ which is responsible for enabling key authentication. This is the key step to disable it.
Volume Mount: C:rustdesk-server:/root maps the directory on your Windows host to the /root directory inside the Docker containers. Ensure the paths are correctly configured for Windows.
Port Mapping: The ports 21115, 21116 (TCP and UDP), and 21117 are mapped. Ensure these ports are open in your Windows Server firewall and any network security groups.

Open Firewall Ports on Windows Server:

Allow inbound traffic on your Windows Server firewall for the following ports: 21115, 21116 (TCP and UDP), and 21117. This is crucial for RustDesk clients to connect to your server.
Start Docker Containers:

Navigate to the C:rustdesk-server directory in your command prompt or PowerShell and run the command:
```
docker-compose up -d
```
This command will download the RustDesk server Docker image and start the hbbs and hbbr containers in detached mode.
Verify Container Status:

Check if the containers are running correctly using:
```
docker ps -a
```
You should see hbbs and hbbr containers in the Up status.

Connecting RustDesk Clients

Download RustDesk Client:

Download the RustDesk client application from the official RustDesk website: https://rustdesk.com/ and install it on the devices you want to remotely access.
Configure Server Address:

Open the RustDesk client and navigate to the “Network” settings (usually a gear icon).

Network Settings in RustDesk Client

In the Network settings:
- ID Server: Enter your Windows Server’s public IP address or domain name (the same one you used in the docker-compose.yml file).
- Key: Leave this field blank. Since we disabled key authentication, no key is required.
Network Settings in RustDesk Client

Click “OK” to save the settings.
Connect:

Now, you can use the RustDesk client as usual. The client will connect to your self-hosted server without requiring key authentication.

Troubleshooting Connection Issues

If you encounter problems connecting, consider these common issues:

ID Not Found: If you are prompted with “ID does not exist,” double-check that your RustDesk server containers are running correctly and that the ID Server address in your client is configured accurately.
Connection Closed by Peer: This error, as mentioned in the original article, can sometimes occur if the volume mount paths for hbbr and hbbs are inconsistent. In our Windows setup, we’ve used C:rustdesk-server:/root for both, ensuring they are the same. Also, verify that the firewall rules on your Windows Server are correctly configured to allow connections on the necessary ports.

Conclusion

Self-hosting a RustDesk server offers greater control and security for your remote access needs. By disabling key authentication, as outlined in this guide for Windows Server, you can simplify connections in specific scenarios where ease of access is prioritized within a controlled environment. Remember to carefully consider the security implications of disabling key authentication based on your specific use case. RustDesk provides a robust and flexible open-source solution for remote desktop management, adaptable to various deployment scenarios.