Refactor batch generation logic to allow large reports to be generated

[aequitas]

Related to #1395

During batch result report generation the result is stored in a variable before being written to a file:

Internet.nl/interface/batch/util.py

Lines 270 to 274 in 9e4d250

	results = gather_batch_results(user, batch_request, site_url)
	save_batch_results_to_file(user, batch_request, results)
	del results
	results = gather_batch_results_technical(user, batch_request, site_url)
	save_batch_results_to_file(user, batch_request, results, technical=True)

For batch requests with 5000 domains this results in a memory usage of 1GB, for 10k domain almost 2GB, etc. Requiring the worker performing this task to have this much memory available for this short time it takes to generate the reports. Furthermore this memory is retained by the worker until the next report generation is run where the memory will be reused but not freed.

Suggest to refactor the generation logic to write the report file to disk in a streaming fashon in gather_batch_results to eliminate the dom_results (

Internet.nl/interface/batch/util.py

Line 291 in 9e4d250

dom_results = {}

) variable which contains the bulk of the memory used.

Because the dom_results (domains field in the report

Internet.nl/interface/batch/util.py

Line 340 in 9e4d250

data["domains"] = dom_results

) is a dictionary/object, existing JSON encoders might not be able to handle this in a streaming manner. But because the report structure is simple enough it might be best to just write a custom encoder or write the JSON directory without any encoder/library.