Cortex integration in TheHive

[Davdavidid]

| OS version (server) | Debian 11 bullseye |
| Python version | 3.9.2 |
| Type of email address used | Gmail |
| Browser type & version | Firefox |
| Virtualized Env. | True |
| Dedicated RAM | 8 GB |
| vCPU | 2 |
| ThePhish version | - |
| TheHive version | 4.1.9-1 |
| Cortex version | 3.1.1-1 |
| MISP version | 2.4.148 |
| Installed using Docker and Docker Compose | True |
| Docker Version | 20.10.16 |
| Docker Compose version | 2.5.0 |

Hi Emalderson,
i really like your Platform but im having problems setting it up. I used your Guide for installation with docker compose and keep getting "AUTH_ERROR" on theHive. I did the integration of the API-keys in theHive application-conf aswell as thephish_conf_files config. In Hive though i get this message:

Are there any other steps i can take to assure a connection between Hive and Cortex? Fetching Emails on ThePhish doesn't work (An error occured). I tried doing a stacktrace but im not used to Linux and couldn't quiet figure it out. Also i don't know how to view thePhish version.

[badspoiler]

Would love to read an Answer for this one ! Very exciting!

[emalderson]

Hi all, sorry for the delay in the response. The problem experienced by Davdavidid may be due to a configuration problem. I know well that the installation and configuration procedure is fairly tedious, but maybe you have skipped some step anywhere in the guide or something like that. The fact that Cortex is not recognized is a problem related to the KEY that you set in the configuration file for sure.

[Davdavidid]

Hallo again and thanks for your answer. On Friday we set up 2 systems using Ubuntu, configured them on cli and they went up and running instantly :) Both used the docker method but different hardware and 1 HyperV, 1 ESXI.
Today though when we wanted to do further tests, both systems were unable to operate because the diskspace ran full (129GB and 120GB).

They were running over the weekend but didn't get any emails beside some test Emails on Friday. Do you know where this might be coming from? There is no harm since i got a Snapshot but i’m worried it’ll happen again.

I would also like your help on a Topic with the Analysis. Is it possible to add the IP of the initial sender as an observable (from the SMTP-header)? If not do you maybe know of an analyzer that will do that?
The things i tried sadly didn’t work and a malicious phishing mail i tested was marked as SAFE. When i manually checked for that IP the services marked it as a spam IP though.
I also have problems adjusting the URLs in the „index.html“. I tried changing the href links to the machines IP-address, but when opening the site the links still point to the old adresses.
href=http://thehive:9000/ -> href="http://192.168.188.62:9000"
Is there another location beside the „configuration.json“ and „index.html“ that i need to adjust?

Thanks for your help in advance!

[emalderson]

Hi, the problem of the disk running full is strange. I do know that every analyzed email may occupy more space than one would expect due to the fact that an entire case is created for each email, but if you just analyzed a couple of emails your disk shouldn't be full. The only thing I can think about now as the root of the problem is that you may have enabled the fetching functionality in your MISP instance, so all the information contained in the various feeds you enable are also ingested and stored on your machine.

Regarding the second problem, what do you mean by "the IP of the sender"? Every IP in the header should be captured by the regular expression engine running in the backend, so if the IP is in the header, it will appear among the observables as well. Are you sure that the IP you are lloking for is actually present in the email, or you just know the IP via other means? I'm asking this question because often the outgoing SMTP servers or the client application don't include the information about the effective sender IP for privacy or security reasons.

Regarding the last problem, the files you have mentioned are the only locations where the URL is written.

[Davdavidid]

You'r totally right regarding the Header. It got cut off by the software when attaching as an .eml. After a manual test everything worked out perfectly!

Regarding the disk space, we just made a clean docker installation as per guide and both had the same issue after letting the machine run over the weekend. After rolling back i constantly check the ressources and haven't seen any high demand yet but i'll report back if i catch anything. If you have any other tips for me what to check for that would be awesome.

About the URL i'm unsure if i'm making a major mistake here but when i use ther index.html to open thePhish everything is fine. When im browsing there via IP, the links are still the default ones.

[Davdavidid]

Hello again,
i let ThePhish run overnight and the size grew to 39GB. Before i left it yesterday it was at ~250MB.

Do you have any idea how to get to the bottom of this problem?

I also did some more testing and found out that as soon as an Email got a DKIM-Signature it will not get fetched by thePhish (will get an error when trying to list). After editing those 2 Emails and removing the Signature it could get processed.
Do you think there is a possibility to fetch Emails with a Signature?

[emalderson]

Hello, sorry for the delay in the response. Regarding the space problem, I have never experienced this problem. I think it may be a MISP problem, since it fetches several feeds. For the DKIM problem, I have never experienced it either. I should test this behavior, but it strange that a DKIM signature prevents the email from being processed, since it is not directly checked by ThePhish.

[Davdavidid]

Hello again. I couldn't get the space problem under control. If i can assist you with the search i'd be happy to.
Regarding the DKIM-signature i found the workaround to forward the Email from the Phish inbox to itself, which removed the Signature and solved the problem.

[emalderson]

Hi, how many emails have you fed to ThePhish? From what I see in your screenshots, it seems like there is a problem related to the size of the logs and correlations that MISP produces. If this is just how MISP works, I don't think you can do anything but disabling MISP if you want to test it on a machine with limited storage.

[Davdavidid]

I probably fed 20 Emails or somehting like that. The size of the vm also doesn't really change in that timeframe. It writes those logs over time (without getting Emails) until to the point that it's not operational anymore. i get that MISP needs a lot of storage to run but there seems to be a problem with it disabling the system. The system has 200GB storage right now but i'll try and extend that even more and reduce feeds.

[emalderson]

I fed hundreds of emails to the tool and never had this problem on different VMs with 50GB storage at max. Have you checked that you have just enabled the feeds and not fetched them in their entirety in the MISP instance?

[Davdavidid]

I don't understand exactly what you mean by enabling fetching but i stuck to the Guide and enabled the misp feeds like this:
Enable MISP feeds:
Sync Actions -> List Feeds -> Load default feed metadata -> All feeds
Select the feeds to enable (filtered for "misp" and enabled them)
Click on "Enable selected"

The "Fetch and store" button seems to be enabled but i dont't think i changed anything there.

[emalderson]

That's weird. I just found an issue for MISP that mentions the same problem that you have and it seems the issuer solved its problem. Now I can't test that in my environment, both because I don't have time now and because i can't reproduce it, but this may help you: MISP/MISP#2800