Add ray tracing benchmark #842

Rombur · 2023-03-13T13:29:07Z

This is a benchmark based on an adamantine use case. We have an IR camera which produces a file of rays and temperatures. We then need to find the first intersection of the rays with the mesh. In adamantine, we are interested in the exact coordinates of the intersections but I've removed that computation here since it doesn't use ArborX. The image contains about 300,000 rays with some of them missing the mesh. By increasing the number of processors we increase the number of cells but the number of rays is constant. All the processors read the camera file.

Here is part of the profiling (bottom-up and top-bottom trees) using 4 processors

BOTTOM-UP TIME TREE:
<average time> <percent of total time> <percent time in Kokkos> <percent MPI imbalance> <number of calls> <name> [type]
===================
|-> 3.52e+00 sec 14.7% 0.0% 0.0% 0.0% 0.00e+00 2 ArborX::DistributedTree::sendAcrossNetwork (ArborX::DistributedTree::query::forwardQueries::exports) [region]
|   |-> 3.52e+00 sec 14.7% 0.0% 0.0% 0.0% 0.00e+00 2 ArborX::DistributedTree::forwardQueries [region]
|       |-> 3.52e+00 sec 14.7% 0.0% 0.0% 0.0% 0.00e+00 2 ArborX::DistributedTree::query::nearest [region] 
|           |-> 3.52e+00 sec 14.7% 0.0% 0.0% 0.0% 0.00e+00 2 Ray::Query [region]
|-> 3.27e+00 sec 13.7% 100.0% 0.0% ------ 3 ArborX::TreeTraversal::nearest [for]
|   |-> 3.27e+00 sec 13.7% 100.0% 0.0% 0.0% 0.00e+00 3 ArborX::BVH::query::nearest [region]
|       |-> 3.27e+00 sec 13.7% 100.0% 0.0% 0.0% 0.00e+00 3 ArborX::CrsGraphWrapper::two_pass::first_pass [region]
|           |-> 3.27e+00 sec 13.7% 100.0% 0.0% 0.0% 0.00e+00 3 ArborX::CrsGraphWrapper::two_pass [region] 
|               |-> 3.27e+00 sec 13.7% 100.0% 0.0% 0.0% 0.00e+00 3 ArborX::CrsGraphWrapper::query::nearest [region]
|                   |-> 3.27e+00 sec 13.7% 100.0% 0.0% 0.0% 0.00e+00 3 ArborX::query [region]
|                       |-> 3.03e+00 sec 12.7% 100.0% 0.0% 0.0% 0.00e+00 2 ArborX::DistributedTree::query::nearest [region]
|                       |   |-> 3.03e+00 sec 12.7% 100.0% 0.0% 0.0% 0.00e+00 2 Ray::Query [region]
|                       |-> 2.45e-01 sec 1.0% 100.0% 0.0% 0.0% 0.00e+00 1 ArborX::DistributedTree::deviseStrategy [region]
|                           |-> 2.45e-01 sec 1.0% 100.0% 0.0% 0.0% 0.00e+00 1 ArborX::DistributedTree::query::nearest [region]
|                               |-> 2.45e-01 sec 1.0% 100.0% 0.0% 0.0% 0.00e+00 1 Ray::Query [region]

TOP-DOWN TIME TREE:                                                                                                                                                                                                
<average time> <percent of total time> <percent time in Kokkos> <percent MPI imbalance> <remainder> <kernels per second> <number of calls> <name> [type]                                                           
===================                                                                                                                                                                                                
|-> 3.47e+00 sec 58.0% 30.4% 0.5% 0.0% 4.53e+01 1 Ray::Query [region]                                                                                                                                              
|   |-> 3.47e+00 sec 58.0% 30.4% 0.5% 0.0% 4.47e+01 1 ArborX::DistributedTree::query::nearest [region]                                                                                                             
|   |   |-> 1.39e+00 sec 23.2% 0.8% 0.7% 7.2% 2.59e+01 2 ArborX::DistributedTree::forwardQueries [region]                                                                                                          
|   |   |   |-> 8.86e-01 sec 14.8% 0.6% 0.9% 99.4% 6.77e+00 2 ArborX::DistributedTree::sendAcrossNetwork (ArborX::DistributedTree::query::forwardQueries::exports) [region]                                        
|   |   |   |-> 2.06e-01 sec 3.4% 0.1% 8.7% 99.9% 9.71e+00 2 ArborX::DistributedTree::sendAcrossNetwork (ArborX::DistributedTree::query::forwardQueries::export_ranks) [region]                                    
|   |   |   |-> 1.91e-01 sec 3.2% 0.2% 0.0% 99.8% 1.05e+01 2 ArborX::DistributedTree::sendAcrossNetwork (ArborX::DistributedTree::query::forwardQueries::export_ids) [region]                                      
|   |   |-> 1.02e+00 sec 17.1% 0.1% 13.0% 22.1% 3.91e+00 2 ArborX::DistributedTree::communicateResultsBack [region]                                                                                                
|   |   |   |-> 2.16e-01 sec 3.6% 0.0% 7.4% 100.0% 0.00e+00 2 ArborX::DistributedTree::sendAcrossNetwork (ArborX::DistributedTree::query::nearest::indices) [region]                                               
|   |   |   |-> 2.08e-01 sec 3.5% 0.0% 7.7% 100.0% 0.00e+00 2 ArborX::DistributedTree::sendAcrossNetwork (ArborX::DistributedTree::query::nearest::distances) [region]                                             
|   |   |   |-> 1.92e-01 sec 3.2% 0.0% 0.0% 100.0% 0.00e+00 2 ArborX::DistributedTree::sendAcrossNetwork (ArborX::DistributedTree::query::forwardQueries::import_ids) [region]                                     
|   |   |   |-> 1.82e-01 sec 3.0% 0.0% 5.7% 100.0% 0.00e+00 2 ArborX::DistributedTree::sendAcrossNetwork (ArborX::DistributedTree::query::forwardQueries::import_ranks) [region]                                   
|   |   |-> 7.63e-01 sec 12.8% 100.0% 18.9% 0.0% 3.93e+01 2 ArborX::query [region]                                                                                                                                 
|   |   |   |-> 7.63e-01 sec 12.8% 100.0% 18.9% 0.0% 3.93e+01 2 ArborX::CrsGraphWrapper::query::nearest [region]                                                                                                   
|   |   |       |-> 7.59e-01 sec 12.7% 100.0% 18.8% 0.0% 2.37e+01 2 ArborX::CrsGraphWrapper::two_pass [region]                                                                                                     
|   |   |       |   |-> 7.58e-01 sec 12.7% 100.0% 18.8% 0.0% 1.58e+01 2 ArborX::CrsGraphWrapper::two_pass::first_pass [region]                                                                                     
|   |   |       |   |   |-> 7.58e-01 sec 12.7% 100.0% 18.8% 0.0% 1.06e+01 2 ArborX::BVH::query::nearest [region]                                                                                                   
|   |   |       |   |   |   |-> 7.57e-01 sec 12.7% 100.0% 18.9% ------ 2 ArborX::TreeTraversal::nearest [for]

aprokop

The 33MB csv file is too large for merging.

benchmarks/CMakeLists.txt

benchmarks/rays/rays.cpp

aprokop · 2023-03-14T20:32:38Z

benchmarks/rays/rays.cpp

+std::pair<Kokkos::View<ArborX::Experimental::Ray[n_rays], MemorySpace>,
+          Kokkos::View<double[n_rays], MemorySpace>>


Is there a reason to use non-dynamic views?

aprokop · 2023-03-14T20:34:54Z

benchmarks/rays/rays.cpp

+    Kokkos::View<ArborX::Nearest<ArborX::Experimental::Ray> *, MemorySpace>
+        queries(Kokkos::view_alloc("queries", Kokkos::WithoutInitializing),
+                rays.size());
+    Kokkos::parallel_for(
+        "Ray::Build_queries",
+        Kokkos::RangePolicy<ExecutionSpace>(0, rays.extent(0)),
+        KOKKOS_LAMBDA(int i) {
+          queries(i) = ArborX::nearest<ArborX::Experimental::Ray>(rays(i), 1);
+        });


Maybe use access traits instead?

aprokop · 2023-03-14T20:35:46Z

benchmarks/rays/rays.cpp

+    Kokkos::View<ArborX::Box[n_divisions_x * n_divisions_y * n_divisions_z],
+                 MemorySpace>
+        bounding_boxes(
+            Kokkos::view_alloc("bounding_boxes", Kokkos::WithoutInitializing));


You would want to use execution space here too, I think.

aprokop · 2023-03-14T20:36:14Z

benchmarks/rays/rays.cpp

+    constexpr int n_divisions_x = 100;
+    constexpr int n_divisions_y = 100;
+    constexpr int n_divisions_z = 100;


Why not simply nx, ny, nz?

aprokop · 2023-03-14T20:37:16Z

benchmarks/rays/rays.cpp

+    MPI_Barrier(comm);
+
+    // Build the distributed tree
+    Kokkos::Profiling::pushRegion("Ray::Setup");


Maybe prefix labels in the file with Example::?

aprokop · 2023-03-14T21:32:19Z

benchmarks/rays/rays.cpp

+  std::ifstream file;
+  file.open(filename);


Suggested change

std::ifstream file;

file.open(filename);

std::ifstream file(filename);

You would want to have a check for failure.

aprokop · 2023-03-14T21:32:52Z

benchmarks/rays/rays.cpp

+  std::ifstream file;
+  file.open(filename);
+  std::string line;
+  std::getline(file, line);


Add a comment about why the first line is ignored.

Rombur added 3 commits March 6, 2023 18:39

Add camera data ouput

658e2b9

Add ray benchmark

6109e18

Workaround for clang

cbd5626

aprokop requested changes Mar 14, 2023

View reviewed changes

Rombur added 3 commits April 24, 2023 16:05

Fix reviewer's comment

28e5c7e

Update csv file

73806f1

More fixes from reviewer's comments

64fa7be

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ray tracing benchmark #842

Add ray tracing benchmark #842

Rombur commented Mar 13, 2023

aprokop left a comment

aprokop Mar 14, 2023

aprokop Mar 14, 2023

aprokop Mar 14, 2023

aprokop Mar 14, 2023

aprokop Mar 14, 2023

aprokop Mar 14, 2023

aprokop Mar 14, 2023

		std::pair<Kokkos::View<ArborX::Experimental::Ray[n_rays], MemorySpace>,
		Kokkos::View<double[n_rays], MemorySpace>>

	std::ifstream file;
	file.open(filename);
	std::ifstream file(filename);

Add ray tracing benchmark #842

Are you sure you want to change the base?

Add ray tracing benchmark #842

Conversation

Rombur commented Mar 13, 2023

aprokop left a comment

Choose a reason for hiding this comment

aprokop Mar 14, 2023

Choose a reason for hiding this comment

aprokop Mar 14, 2023

Choose a reason for hiding this comment

aprokop Mar 14, 2023

Choose a reason for hiding this comment

aprokop Mar 14, 2023

Choose a reason for hiding this comment

aprokop Mar 14, 2023

Choose a reason for hiding this comment

aprokop Mar 14, 2023

Choose a reason for hiding this comment

aprokop Mar 14, 2023

Choose a reason for hiding this comment