fix: changed threadpool -> malebolgia #88

fukusuket · 2024-01-27T05:40:15Z

What Changed

Fixed Deprecated usage of spawn in VT lookup commands #81
- Changed threadpool to malebolgia

fukusuket · 2024-01-27T05:45:22Z

@YamatoSecurity
Unfortunately, due to a issue nim-lang/Nim#23129 in Nim, other wanings have started appearing😢
I think it will probably be resolved in the next Nim release, so it might be better to wait for the next version of Nim🤔
What do you think?

fukusuke@fukusukenoMacBook-Air takajo % nimble build -d:release --threads:on
  Verifying dependencies for [email protected]
     Info:  Dependency on cligen@>= 1.5 already satisfied
  Verifying dependencies for [email protected]
     Info:  Dependency on suru@#f6f1e607c585b2bc2f71309996643f0555ff6349 already satisfied
  Verifying dependencies for [email protected]
     Info:  Dependency on puppy@>= 2.1.0 already satisfied
  Verifying dependencies for [email protected]
     Info:  Dependency on libcurl@>= 1.0.0 already satisfied
  Verifying dependencies for [email protected]
     Info:  Dependency on zippy@>= 0.10.0 already satisfied
  Verifying dependencies for [email protected]
     Info:  Dependency on webby@>= 0.1.6 already satisfied
  Verifying dependencies for [email protected]
     Info:  Dependency on termstyle@any version already satisfied
  Verifying dependencies for [email protected]
     Info:  Dependency on nancy@any version already satisfied
  Verifying dependencies for [email protected]
     Info:  Dependency on ansiparse@>= 0.2.0 already satisfied
  Verifying dependencies for [email protected]
     Info:  Dependency on malebolgia@any version already satisfied
  Verifying dependencies for [email protected]
   Building takajo/takajo using c backend
/Users/fukusuke/Scripts/Nim/takajo/src/takajopkg/vtDomainLookup.nim(92, 6) template/generic instantiation of `awaitAll` from here
/Users/fukusuke/.nimble/pkgs2/malebolgia-0.1.0-5c404addea46d485f94915a897fba9906feff92d/malebolgia.nim(244, 5) template/generic instantiation of `checkBody` from here
/Users/fukusuke/Scripts/Nim/takajo/src/takajopkg/vtDomainLookup.nim(96, 12) template/generic instantiation of `spawn` from here
/Users/fukusuke/.nimble/pkgs2/malebolgia-0.1.0-5c404addea46d485f94915a897fba9906feff92d/malebolgia.nim(177, 47) template/generic instantiation of `toTask` from here
/Users/fukusuke/.choosenim/toolchains/nim-2.0.2/lib/std/tasks.nim(132, 20) template/generic instantiation of `isolate` from here
/Users/fukusuke/.choosenim/toolchains/nim-2.0.2/lib/std/isolation.nim(37, 14) template/generic instantiation of `=destroy` from here
/Users/fukusuke/.choosenim/toolchains/nim-2.0.2/lib/std/isolation.nim(27, 6) template/generic instantiation from here
/Users/fukusuke/.choosenim/toolchains/nim-2.0.2/lib/std/isolation.nim(29, 13) Warning: `=destroy`(dest.value) can raise an unlisted exception: Exception [Effect]
/Users/fukusuke/Scripts/Nim/takajo/src/takajopkg/vtDomainLookup.nim(92, 6) template/generic instantiation of `awaitAll` from here
/Users/fukusuke/.nimble/pkgs2/malebolgia-0.1.0-5c404addea46d485f94915a897fba9906feff92d/malebolgia.nim(244, 5) template/generic instantiation of `checkBody` from here
/Users/fukusuke/Scripts/Nim/takajo/src/takajopkg/vtDomainLookup.nim(96, 12) template/generic instantiation of `spawn` from here
/Users/fukusuke/.nimble/pkgs2/malebolgia-0.1.0-5c404addea46d485f94915a897fba9906feff92d/malebolgia.nim(177, 47) template/generic instantiation of `toTask` from here
/Users/fukusuke/.choosenim/toolchains/nim-2.0.2/lib/std/tasks.nim(132, 20) template/generic instantiation of `isolate` from here
/Users/fukusuke/.choosenim/toolchains/nim-2.0.2/lib/std/isolation.nim(37, 14) template/generic instantiation of `=destroy` from here
/Users/fukusuke/.choosenim/toolchains/nim-2.0.2/lib/std/isolation.nim(27, 6) template/generic instantiation from here
/Users/fukusuke/.choosenim/toolchains/nim-2.0.2/lib/std/isolation.nim(29, 13) Warning: `=destroy`(dest.value) can raise an unlisted exception: Exception [Effect]
ld: warning: ignoring duplicate libraries: '-lm'

fukusuket · 2024-01-27T06:09:13Z

I have opened the following issue.
Araq/malebolgia#29

fukusuket · 2024-01-27T07:35:24Z

It's already fixed in the development branch(not released yet)😁

Warning: =destroy (dest.value) can raise an unlisted exception: Exception [Effect] displayed when running README's example Araq/malebolgia#29 (comment)

I'm starting to think it's okay to merge...😅 Anyway, I'll test it first!

fukusuket · 2024-01-27T08:24:34Z

Environment

OS: macOS Sonoma version 14.2.1
Hayabusa v2.12.0
Nim: 2.0.2

TEST(macOS)

I confirmed that all VT commands are successful as shown below.

vt-domain-lookup

./takajo vt-domain-lookup -a key --domainList domains.txt -r 1000 -o res-domain.csv
...
Loaded domains: 5
Rate limit per minute: 1000

Estimated time: 0 hours, 0 minutes, 0 seconds

100%|█████████████████████████| 5/5 [ 1.6s< 0.0s,  15.38 /sec]

Finished querying domains. 0 Malicious domains found.

Domain not found:

Saved CSV results to res-domain.csv (8.00 KB)

Elapsed time: 0 hours, 0 minutes, 1 seconds

vt-ip-lookup

% ./takajo vt-ip-lookup -q -a key --ipList ip.txt -r 1000 -o re-ip.csv --jsonOutput responses.json
Started the VirusTotal IP Lookup command

This command will lookup a list of IP addresses on VirusTotal.
Specify -j results.json to save the original JSON responses.
The default rate is 4 requests per minute so increase this if you have a premium membership.

Loading IP addresses. Please wait.

Loaded IP addresses: 4
Rate limit per minute: 1000

Estimated time: 0 hours, 0 minutes, 0 seconds

100%|█████████████████████████| 4/4 [ 1.1s< 0.0s,  15.49 /sec]

Finished querying IP addresses. 1 Malicious IP addresses found.

Found malicious IP address: 185.199.111.153 (Malicious count: 3)

Saved CSV results to re-ip.csv (4.00 KB)
Saved JSON responses to responses.json (83.49 KB)

Elapsed time: 0 hours, 0 minutes, 1 seconds

vt-hash-lookup

% ./takajo vt-hash-lookup -q -a 27b3a40b2e9a4334810befa7690f2ffbfbb2b040d9098519dbd574b4ad1521cc --hashList hash.txt-MD5-hashes.txt -r 1000 -o res-hash.csv --jsonOutput res-hash.json
Started the VirusTotal Hash Lookup command

This command will lookup a list of file hashes on VirusTotal.
Specify -j results.json to save the original JSON responses.
The default rate is 4 requests per minute so increase this if you have a premium membership.

Loading hashes. Please wait.

Loaded hashes: 4
Rate limit per minute: 1000

Estimated time: 0 hours, 0 minutes, 0 seconds

100%|█████████████████████████| 4/4 [ 1.2s< 0.0s,  15.37 /sec]

Finished querying hashes. 0 Malicious hashes found.


Saved CSV results to res-hash.csv (815 Bytes)
Saved JSON responses to res-hash.json (178.30 KB)

Elapsed time: 0 hours, 0 minutes, 1 seconds

fukusuket · 2024-01-27T08:35:14Z

Environment

OS: Ubuntu 22.04.3 LTS
Hayabusa v2.12.0
Nim: 2.0.0

TEST(Linux)

I confirmed that all VT commands are successful as shown below.

vt-domain-lookup

./takajo vt-domain-lookup -a key --domainList domains.txt -r 1000 -o res-domain.csv
...
Started the VirusTotal Domain Lookup command

This command will lookup a list of domains on VirusTotal.
Specify -j results.json to save the original JSON responses.
The default rate is 4 requests per minute so increase this if you have a premium membership.

Loading domains. Please wait.

Loaded domains: 5
Rate limit per minute: 1000

Estimated time: 0 hours, 0 minutes, 0 seconds

100%|█████████████████████████| 5/5 [ 2.4s< 0.0s,  16.59 /sec]

Finished querying domains. 0 Malicious domains found.

Domain not found:

Saved CSV results to res-domain.csv (8.00 KB)

Elapsed time: 0 hours, 0 minutes, 2 seconds

vt-ip-lookup

./takajo vt-ip-lookup -q -a key --ipList ip.txt -r 1000 -o re-ip.csv --jsonOutput responses.json
Started the VirusTotal IP Lookup command

This command will lookup a list of IP addresses on VirusTotal.
Specify -j results.json to save the original JSON responses.
The default rate is 4 requests per minute so increase this if you have a premium membership.

Loading IP addresses. Please wait.

Loaded IP addresses: 4
Rate limit per minute: 1000

Estimated time: 0 hours, 0 minutes, 0 seconds

100%|█████████████████████████| 4/4 [ 2.1s< 0.0s,  16.59 /sec]

Finished querying IP addresses. 1 Malicious IP addresses found.

Found malicious IP address: 185.199.111.153 (Malicious count: 3)

Saved CSV results to re-ip.csv (4.00 KB)
Saved JSON responses to responses.json (83.49 KB)

Elapsed time: 0 hours, 0 minutes, 2 seconds

vt-hash-lookup

./takajo vt-hash-lookup -q -a key --hashList hash.txt-MD5-hashes.txt -r 1000 -o re-hash.csv --jsonOutput res-hash.json
Started the VirusTotal Hash Lookup command

This command will lookup a list of file hashes on VirusTotal.
Specify -j results.json to save the original JSON responses.
The default rate is 4 requests per minute so increase this if you have a premium membership.

Loading hashes. Please wait.

Loaded hashes: 4
Rate limit per minute: 1000

Estimated time: 0 hours, 0 minutes, 0 seconds

100%|█████████████████████████| 4/4 [ 1.6s< 0.0s,  16.59 /sec]

Finished querying hashes. 0 Malicious hashes found.


Saved CSV results to re-hash.csv (0 Bytes)
Saved JSON responses to res-hash.json (178.30 KB)

fukusuket · 2024-01-27T10:50:21Z

Environment

OS: Windows11
Hayabusa v2.12.0
Nim: 2.0.0

TEST(Windows)

I confirmed that all VT commands are successful as shown below.

vt-domain-lookup

.\takajo.exe vt-domain-lookup -a key --domainList .\domains.txt -r 1000 -o res-domain.csv --jsonOutput res-domain.json
...
Started the VirusTotal Domain Lookup command

This command will lookup a list of domains on VirusTotal.
Specify -j results.json to save the original JSON responses.
The default rate is 4 requests per minute so increase this if you have a premium membership.

Loading domains. Please wait.

Loaded domains: 4
Rate limit per minute: 1000

Estimated time: 0 hours, 0 minutes, 0 seconds

100%|█████████████████████████| 4/4 [ 1.3s< 0.0s,  15.83 /sec]

Finished querying domains. 1 Malicious domains found.

Found malicious domains: facebook.com (Malicious count: 1)

Saved CSV results to res-domain.csv (0 Bytes)
Saved JSON responses to res-domain.json (115.14 KB)

Elapsed time: 0 hours, 0 minutes, 1 seconds

vt-ip-lookup

./takajo  vt-ip-lookup -a key --ipList ipAddresses.txt -r 1000 -o results.csv --jsonOutput responses.json

Started the VirusTotal IP Lookup command

This command will lookup a list of IP addresses on VirusTotal.
Specify -j results.json to save the original JSON responses.
The default rate is 4 requests per minute so increase this if you have a premium membership.

Loading IP addresses. Please wait.

Loaded IP addresses: 5
Rate limit per minute: 1000

Estimated time: 0 hours, 0 minutes, 0 seconds

100%|█████████████████████████| 5/5 [ 1.0s< 0.0s,  13.30 /sec]

Finished querying IP addresses. 1 Malicious IP addresses found.

IP address not found:
Found malicious IP address: 185.199.111.153 (Malicious count: 3)

Saved CSV results to results.csv (0 Bytes)
Saved JSON responses to responses.json (83.50 KB)

Elapsed time: 0 hours, 0 minutes, 1 seconds

vt-hash-lookup

 .\takajo.exe vt-hash-lookup -a key --hashList .\hash-MD5-hashes.txt -r 1000 -o results.csv --jsonOutput responses.json
...
Started the VirusTotal Hash Lookup command

This command will lookup a list of file hashes on VirusTotal.
Specify -j results.json to save the original JSON responses.
The default rate is 4 requests per minute so increase this if you have a premium membership.

Loading hashes. Please wait.

Loaded hashes: 4
Rate limit per minute: 1000

Estimated time: 0 hours, 0 minutes, 0 seconds

100%|█████████████████████████| 4/4 [ 1.8s< 0.0s,  15.04 /sec]

Finished querying hashes. 0 Malicious hashes found.


Saved CSV results to results.csv (820 Bytes)
Saved JSON responses to responses.json (178.30 KB)

Elapsed time: 0 hours, 0 minutes, 1 seconds

fukusuket · 2024-01-27T10:51:56Z

I have confirmed normal operation on Windows, Ubuntu, and macOS!
I would appreciate it if you could check it out when you have time🙏

YamatoSecurity · 2024-01-28T09:28:48Z

@fukusuket
Thanks so much! I am glad that it is working but since it results in many more warnings with the current stable nim version, let's wait until it gets fixed on the nim side before merging.

fukusuket · 2024-01-28T10:42:16Z

@YamatoSecurity
I agree! I will keep this PR as Draft until Nim is released :)

fukusuket · 2024-05-03T00:58:58Z

I tried v2.0.4 but the warning issue was not resolved :(
https://nim-lang.org/blog/2024/04/16/versions-1620-204-released.html

YamatoSecurity · 2024-05-03T08:00:33Z

@fukusuket That's too bad. Since this seems to be a nim issue, can you raise an issue in the nim repository and include error logs for them to look at?

fukusuket · 2024-05-03T10:28:20Z

@YamatoSecurity
I checked the source code of Nim's Compiler and found that the fix has been merged into the dev branch (but not into the 2.0.4 tag). So I'm looking forward to the next release!

issue:
nim-lang/Nim#23129

2.0.4:
https://github.com/nim-lang/Nim/blob/b47747d31844c6bd9af4322efe55e24fefea544c/compiler/liftdestructors.nim#L1157-L1158

dev:
https://github.com/nim-lang/Nim/blob/d772186b2dbacdbeceaf2cd3ed3bd583f912e6e3/compiler/liftdestructors.nim#L1190C10-L1196

fukusuket · 2024-06-27T11:00:32Z

I tried v2.0.6 but the warning issue was not resolved :(
https://nim-lang.org/blog/2024/06/17/version-206-released.html

fix: changed threadpool -> malebolgia

33c1e2d

fukusuket self-assigned this Jan 27, 2024

fukusuket added the bug Something isn't working label Jan 27, 2024

fix: close before getFileSize.

59872dc

fukusuket marked this pull request as ready for review January 27, 2024 10:50

fukusuket requested a review from YamatoSecurity January 27, 2024 10:52

fukusuket added this to the v2.3.1 milestone Jan 27, 2024

fukusuket added invalid This doesn't seem right and removed bug Something isn't working labels Jan 27, 2024

fukusuket marked this pull request as draft January 28, 2024 10:42

fukusuket removed this from the v2.3.1 milestone Jan 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: changed threadpool -> malebolgia #88

fix: changed threadpool -> malebolgia #88

fukusuket commented Jan 27, 2024 •

edited

Loading

fukusuket commented Jan 27, 2024 •

edited

Loading

fukusuket commented Jan 27, 2024

fukusuket commented Jan 27, 2024 •

edited

Loading

fukusuket commented Jan 27, 2024 •

edited

Loading

fukusuket commented Jan 27, 2024

fukusuket commented Jan 27, 2024

fukusuket commented Jan 27, 2024

YamatoSecurity commented Jan 28, 2024

fukusuket commented Jan 28, 2024

fukusuket commented May 3, 2024 •

edited

Loading

YamatoSecurity commented May 3, 2024

fukusuket commented May 3, 2024

fukusuket commented Jun 27, 2024

fix: changed threadpool -> malebolgia #88

Are you sure you want to change the base?

fix: changed threadpool -> malebolgia #88

Conversation

fukusuket commented Jan 27, 2024 • edited Loading

What Changed

fukusuket commented Jan 27, 2024 • edited Loading

fukusuket commented Jan 27, 2024

fukusuket commented Jan 27, 2024 • edited Loading

fukusuket commented Jan 27, 2024 • edited Loading

Environment

TEST(macOS)

vt-domain-lookup

vt-ip-lookup

vt-hash-lookup

fukusuket commented Jan 27, 2024

Environment

TEST(Linux)

vt-domain-lookup

vt-ip-lookup

vt-hash-lookup

fukusuket commented Jan 27, 2024

Environment

TEST(Windows)

vt-domain-lookup

vt-ip-lookup

vt-hash-lookup

fukusuket commented Jan 27, 2024

YamatoSecurity commented Jan 28, 2024

fukusuket commented Jan 28, 2024

fukusuket commented May 3, 2024 • edited Loading

YamatoSecurity commented May 3, 2024

fukusuket commented May 3, 2024

fukusuket commented Jun 27, 2024

fukusuket commented Jan 27, 2024 •

edited

Loading

fukusuket commented Jan 27, 2024 •

edited

Loading

fukusuket commented Jan 27, 2024 •

edited

Loading

fukusuket commented Jan 27, 2024 •

edited

Loading

fukusuket commented May 3, 2024 •

edited

Loading