How to deploy the Malcolm network traffic analysis tool with Ubuntu Server 22.04

Jack Wallen walks you through the steps of deploying a powerful, easy-to-use network analysis tool on Ubuntu Server 22.04.

Malcolm is an open-source network traffic analysis tool that uses a framework of tools to create a robust analysis tool for network admins. Malcolm accepts network traffic data in the form of PCAP (full packet capture) files and Zeek logs.

SEE: Hiring Kit: Network Engineer (TechRepublic Premium)

Malcolm includes two different interfaces:

• OpenSearch Dashboards: A flexible data visualization plugin with dozens of prebuilt dashboards.
• Arkime: A powerful tool for finding and identifying network sessions made up of suspected security incidents.

Malcolm is easy to use, containerized, secure and in very active development. I want to walk you through the process of deploying this tool on Ubuntu Server 22.04.

What you’ll need for Malcolm

To get Malcolm up and running, you’ll need an instance of Ubuntu Server 22.10 and a user with sudo privileges. That’s it: Let’s get to work.

How to create a new user

The first thing we’ll do is create a new user. SSH or log in to your Ubuntu Server instance and issue the command:

sudo useradd -m -d /opt/malcolm -s /bin/bash -G sudo malcolm

Change the password for the new user with:

sudo passwd malcolm

Log in as that user with:

su - malcolm

How to clone Malcolm and run the installer

Using git, clone the latest release of Malcolm with:

git clone https://github.com/idaholab/Malcolm

Change into the newly created directory with:

cd Malcolm

Run the installer with:

sudo ./scripts/install.py

During this first stage of the installation, you’ll be asked a few questions. For every Y/N question, answer with Y. The only non-Y/N question is:

Enter user account:

To that, answer with:

malcolm

How to configure Malcolm

Once you’ve answered the installer questions, you need to configure Malcolm. Open the configuration file with:

sudo ./scripts/install.py –configure

Once again, you’ll be asked several questions. Here are the questions and the answers you should give:

• Malcolm processes will run as UID 1000 and GID 1000. Is this OK? (Y/n): Y
• Setting 10g for OpenSearch and 3g for Logstash. Is this OK? yes
• Setting 3 workers for Logstash pipelines. Is this OK? (Y/n): yes
• Restart Malcolm upon system or Docker daemon restart: Yes — make sure to choose the default option, unless-stopped.
• Choose whether to set up Malcolm with HTTPS: Yes
• Choose whether Malcolm will run behind any proxy: No
• Choose networking: Hit Enter
• Choose LDAP: No
• Store OpenSearch index snapshosts locally in /opt/malcolm/Malcom/opensearch-backup? Yes
• Choose to Compress OpenSearch index snapshots: Yes
• Automatically analyze all PCAP files with Suricata: Yes
• Download updated Suricata signatures periodically: Yes
• Automatically analyze all PCAP files with Zeek: Yes
• Whether you want to delete the oldest indices when the database exceeds a certain size: No
• Reverse DNS lookup locally for source and destination IP addresses in logs: No
• Hardware vendor OUI lookups for MAC addresses: Yes
• Perform string randomness scoring on some fields: yes
• Expose OpenSearch port to external hosts: no
• Expose Logstash port to external hosts: no
• Forward Logstash logs to external OpenSearch instance: no
• Expose Filebeat TCP port to external hosts: no
• Expose SFTP server (for PCAP upload) to external hosts: No
• Enable file extraction with Zeek: yes
• Choose interesting as the extraction behavior (Figure A).
• Choose file preservation method: quarantine
• Scan extracted files/PE files with ClamAV: yes
• Scan extracted files/PE files with Yara: yes
• Scan extracted files/PE files with Capa: yes
• Lookup extracted file hashes with VirusTotal: no
• Download updated scanner signatures periodically: yes
• Should Malcolm capture network traffic to PCAP files for analysis with Arkime: yes
• Specify capture interface(s) (comma-separated) on which Malcolm will use to network traffic: eth0
• Capture packets using netsniff-ng (Y/n): yes
• Capture packets using tcpdump (y/N): no
• Should Malcolm analyze traffic with Suricata: No
• Capture filter (tcpdump-like filter expression; leave blank to capture all traffic) NOTE: You can disable traffic related to Elasticsearch (port 9200), Logstash (5044), Arkime(8005): not port 9200 and not port 5044 and not port 8005
• Disable capture interface hardware offloading and adjust ring buffer sizes: (y/N): n

Figure A

Once you’ve done this, reboot the system with:

sudo reboot

How to create an admin account for Malcolm

Once the system has rebooted, log back in and change into the Malcolm account with:

su – malcolm

Change into the Malcolm directory for the user:

cd ~/Malcolm

Run the admin account setup script with:

./scripts/auth_setup

Answer all of the required questions as such:

• Store administrator username/password for local Malcolm access? yes
• Create a new admin user and give that user a password.
• (Re)generate self-signed SSL certs for web traffic HTTPS: yes
• (Re)generate self-signed certificates for a remote log forwarder: yes
• Store username/password for forwarding Logstash events to a secondary, external OpenSearch instance: no
• Store username/password for email alert sender account: no

How to pull the required Docker image

Malcolm is deployed with Docker, so first, we must pull the official image with:

docker-compose pull

The pull will take some time, so sit back and enjoy the passing output or go do something else. Give between two to 10 minutes for this to complete.

How to start and access Malcolm

To start the Malcolm service, issue the command:

./scripts/start

The above command will deploy the Docker container. Give the containers enough time to deploy and you’re ready to go. Malcolm has a few different URLs for different tasks. For each component, make sure to log in with the admin account you created during the configuration step.

• For the OpenSearch Dashboard, the address is https://SERVER/dashbaords, where SERVER is the IP address of the hosting server.
• For the Malcolm Capture File and Log Archive Upload screen, the address is https://SERVER/upload, where SERVER is the IP address of the hosting server.
• For the Host and Subnet Mapping Editor, the address is https://SERVER/name-map-ui, where SERVER is the IP address of the hosting server.
• For the Account Management screen, the address is https://SERVER:488, where SERVER is the IP address of the hosting server.

And that’s all there is to deploying the Malcolm Network Traffic Analyzer. Hopefully, you’ll get plenty of use from this powerful tool.

Subscribe to TechRepublic’s How To Make Tech Work on YouTube for all the latest tech advice for business pros from Jack Wallen.