Distribution
If you don't want to run the Simplygon processing on your local machine you can use distribution to offload the work to another machine.
Simplygon support three different distribution methods: Simplygon Grid, Incredibuild and FASTBuild and you can use distribution in our integrations or directly in the Simplygon API.
Simplygon process distribution works using pipelines and pipeline batches. Any pipeline and pipeline batch can be distributed directly through the API by using the run mode parameter in the run scene API call. Distribution is handled by the SimplygonBatch tool in a separate sandboxed process.
Paths
By default the batch tool is located using the SIMPLYGON_10_PATH
environment variable. If you wish to place the executable in a custom location, use the SetSimplygonBatchPath setting on the pipeline to the path where the batch tool executable is located.
License
If you have a node locked license you must ensure that each machine that wants to act as a worker node in distribution must have its own license installed, since the license is unique for each machine. Otherwise the license will be included in the work data and the target machine does not need to have the license pre-installed.
IMPORTANT NOTE
Distribution is not supported in Simplygon Free. Instead it will always fallback to local processing.
Simplygon Grid
The easiest way to get distribution of pipelines and pipeline batches running is to use the Simplygon Grid. There is no need to setup coordinators or network details, just run the grid agent executable on any number of machines on the local network and everything is setup for distribution.
Running a distribution agent
To start a grid agent run the SimplygonGridAgent.exe tool on the machine. There is no coordinator or network setup needed, the pipeline execution locates available processing nodes using mDNS/DNS-SD.
Check Grid capabilities
Grid agent availability is currently based on compatible version, vulkan capabilities and load. If you want to check the status of your grid agents, run SimplygonBatch.exe -Discover
. That will list all your agents that are discoverable and also print information about each agent, here is an example output:
❯ & "C:\Program Files\Simplygon\10\SimplygonBatch.exe" -Discover
SimplygonBatch discover remote agents
...
[2023-09-08 10:38:03] [INFO] Found 6 agents:
[2023-09-08 10:38:03] [INFO] =========================================
[2023-09-08 10:38:03] [INFO] 9.9.9.120:49157 [10.2.2802, load 0]
[2023-09-08 10:38:03] [INFO] =========================================
[2023-09-08 10:38:03] [INFO] 9.9.9.203:49157 [10.2.2802, load 10]
[2023-09-08 10:38:03] [INFO] * Agent (10.2.2802) is not running the same version as client (10.2.3502)
[2023-09-08 10:38:03] [WARNING] * No Vulkan capabilities. Pipelines requiring Vulkan cannot be distributed to this agent.
[2023-09-08 10:38:03] [INFO] =========================================
[2023-09-08 10:38:03] [INFO] 9.9.9.248:49157 [10.2.2802, load 43]
[2023-09-08 10:38:03] [WARNING] * No Vulkan capabilities. Pipelines requiring Vulkan cannot be distributed to this agent.
In the above example we can see that two of the agents are missing vulkan capabilites and won't pick up pipelines that requires vulkan. Also, one of the agent is running an older version and needs an upgrade in order to be availabe and pick up distributed jobs.
Bridging subnets
If your network is segmented and multicast UDP packets cannot traverse subnet boundaries you can bridge these subnets by allowing the batch tool to connect to a Grid agent on a different subnet over TCP/IP and routing discovery of hosts from that subnet back to the batch tool over the TCP/IP connection. This is done by setting the environment variable SIMPLYGON_<MajorVersion>_GRID_HOSTS
to a semicolon separated list of machines (either IP address or FQDN) running Grid agents on different subnets. This way you do not need to have every machine in the list, only one on each subnet that can act as the bridge during discovery. The bridge connection utilizes TCP/IP port 55001 which must be open in any firewalls. The environment variable needs to be set only on the machine that is the distributor.
![Bridging Grid](/SimplygonSDK_10.2.5200.0/assets/bridging_grid.bef3f882.png)
Replace <MajorVersion>
with the major version of Simplygon, e.g. SIMPLYGON_10_GRID_HOSTS
.
Running as a Windows service
The grid agent can be started as a Windows service. To install the service, run the grid agent executable with the -InstallService
argument. To uninstall the service, run the grid agent executable with the -UninstallService
argument. Errors will be posted to the Windows event log. Note that services are run in session 0 and cannot display any UI to the logged in user.
Log to file
If you want the grid agent log output as a file, add the -LogFile <path-to-file>
argument to the command line.
Distributed pipeline execution
Distribution is enabled by passing the appropriate run mode to the pipeline run scene API
spReductionPipeline reductionPipeline = sg->CreateReductionPipeline();
// Process a scene from file directly using distribution
reductionPipeline->RunSceneFromFile( "input.obj", "output.obj", EPipelineRunMode::RunDistributedUsingSimplygonGrid );
// Or if you have a scene object, process it using distribution
reductionPipeline->RunScene( scene, EPipelineRunMode::RunDistributedUsingSimplygonGrid )
spReductionPipeline reductionPipeline = sg.CreateReductionPipeline();
// Process a scene from file directly using distribution
reductionPipeline.RunSceneFromFile( "input.obj", "output.obj", EPipelineRunMode.RunDistributedUsingSimplygonGrid );
// Or if you have a scene object, process it using distribution
reductionPipeline.RunScene( scene, EPipelineRunMode.RunDistributedUsingSimplygonGrid )
reductionPipeline = sg.CreateReductionPipeline()
# Process a scene from file directly using distribution
reductionPipeline.RunSceneFromFile( "input.obj", "output.obj", EPipelineRunMode_RunDistributedUsingSimplygonGrid )
# Or if you have a scene object, process it using distribution
reductionPipeline.RunScene( scene, EPipelineRunMode_RunDistributedUsingSimplygonGrid )
Distributed batch tool
Distribution can be enabled when invoking the batch tool (SimplygonBatch.exe
) by passing the -Distribute
parameter on the command line
SimplygonBatch.exe -Distribute <path/to/pipeline.json> <input.scene> <output.scene>
SimplygonBatch.exe -Distribute <path/to/pipelinebatch.json>
If you wish to integrate the batch tool in an asset processing tool and read progress for an UI, you can pass and additional parameters to the batch tool executable, -Progress
. It will suppress the command line progress bar and instead print progress as a number between 0 and 100, one line at a time, to the standard output.
SimplygonBatch.exe -Distribute -Progress <path/to/pipeline.json> <input.scene> <output.scene>
SimplygonBatch.exe -Distribute -Progress <path/to/pipelinebatch.json>
Here is an example how to do batch processing using Simplygon Grid.
How it works
The pipeline execution usees the batch tool (SimplygonBatch.exe
) to do the distribution. The batch tool find a suitable node using mDNS/DNS-SD (multicast UDP port 5353), picks the node with the lowest overall load and sends/receives files using TCP/IP (by default using an ephemeral port). Progress and errors are propagated back to the originating host.
Cascaded distribution
If a pipeline execution is distributed, and the pipeline is cascaded with multiple child pipelines, the execution of the child pipeline will be re-distributed to parallelize work as much as possible. This is handled internally by the pipeline execution.
Fallback
If the pipeline execution is unable to locate a suitable node for distributed execution it will fall back to local processing.
Settings
BatchLogPath
Output SimplygonBatch.exe's log to file. This setting can be very useful when you want to troubleshoot pipelines executed using RunInNewProcess
or RunDistributedUsingSimplygonGrid
sg->SetGlobalBatchLogPathSetting("path.log");
> SimplygonBatch.exe -Log "path.log"
GridDiscoverTimeout
The time in milliseconds SimplygonBatch waits before presenting agents found during -Discover
sg->SetGlobalGridDiscoverTimeoutSetting(1000);
> SimplygonBatch.exe -DiscoverTimeout 1000
GridAgentLoadLimit
The maximum allowed agent load between 1-100%
sg->SetGlobalGridAgentLoadLimitSetting(50);
> SimplygonBatch.exe -AgentLoadLimit 50
Network troubleshooting
In case the distribution batch tool has trouble distributing processing work you can run the SimplygonBatch.exe
tool with a -Discover
command line argument to make it list the remote grid agents it is able to find. If you are running agents on your local network and this call does not list any remote hosts, check that your network is not blocking multicast UDP packets between machines.
If the Grid agent reports the wrong IP address, it is possible to select which network interface the agent should listen on by setting the SIMPLYGON_<MajorVersion>_GRID_IFINDEX
environment variable before starting the Grid agent.
See more information under Installation -> Environment variables.
FASTBuild
Simplygon can use FASTBuild to distribute pipeline and pipeline batch execution. Setup of FASTBuild is outside the scope of this documentation, please refer to the FASTBuild documentation. You must make sure FBuild.exe
tool path must be set in the search paths and that the FASTBUILD_BROKERAGE_PATH
environment variable is set in the environment where you execute the Simplygon processing or SimplygonBatch tool.
Distributed pipeline execution
Distribution using FASTBuild is enabled by passing the appropriate run mode to the pipeline run scene API.
Progress
Due to limitations in FASTBuild there are no progress events when using FASTBuild distribution. The execution will block on the run call until processing has completed
spReductionPipeline reductionPipeline = sg->CreateReductionPipeline();
// Process a scene from file directly using distribution
reductionPipeline->RunSceneFromFile( "input.obj", "output.obj", EPipelineRunMode::RunDistributedUsingFastbuild );
// Or if you have a scene object, process it using distribution
reductionPipeline->RunScene( scene, EPipelineRunMode::RunDistributedUsingFastbuild )
spReductionPipeline reductionPipeline = sg.CreateReductionPipeline();
// Process a scene from file directly using distribution
reductionPipeline.RunSceneFromFile( "input.obj", "output.obj", EPipelineRunMode.RunDistributedUsingFastbuild );
// Or if you have a scene object, process it using distribution
reductionPipeline.RunScene( scene, EPipelineRunMode.RunDistributedUsingFastbuild )
reductionPipeline = sg.CreateReductionPipeline()
# Process a scene from file directly using distribution
reductionPipeline.RunSceneFromFile( "input.obj", "output.obj", EPipelineRunMode_RunDistributedUsingFastbuild )
# Or if you have a scene object, process it using distribution
reductionPipeline.RunScene( scene, EPipelineRunMode_RunDistributedUsingFastbuild )
Distributed using batch tool
Distribution using FASTBuild can be enabled when invoking the batch tool (SimplygonBatch.exe
) by passing the -Fastbuild
parameter on the command line
SimplygonBatch.exe -Fastbuild <path/to/pipeline.json> <input.scene> <output.scene>
SimplygonBatch.exe -Fastbuild <path/to/pipelinebatch.json>
Note that due to FASTBuild limitations, the batch tool cannot report any progress.
Incredibuild
Simplygon can use Incredibuild to distribute pipeline and pipeline batch execution. Setup of Incredibuild is outside the scope of this documentation, please refer to the Incredibuild documentation. The IBConsole.exe
tool path must be set in the search paths in the environment where Simplygon is executing.
Multiple pipelines
Due to limitations in Incredibuild where only a single execution of IBConsole is allowed on a machine, calls to process multiple pipelines in multiple threads or processes will cause the IBConsole to block and execute the processin in a sequence rather than in parallel. If you plan on distributing multiple pipelines and scenes in parallel using Incredibuild you must use the pipeline batch API. A pipeline batch will allow Incredibuild to launch the processing of all pipelines and scenes in parallel, as it can be done in a single invocation of IBConsole
Distributed pipeline execution
Distribution using Incredibuild is enabled by passing the appropriate run mode to the pipeline run scene API.
Progress
Due to limitations in Incredibuild there are no progress events when using Incredibuild distribution. The execution will block on the run call until processing has completed
spReductionPipeline reductionPipeline = sg->CreateReductionPipeline();
// Process a scene from file directly using distribution
reductionPipeline->RunSceneFromFile( "input.obj", "output.obj", EPipelineRunMode::RunDistributedUsingIncredibuild );
// Or if you have a scene object, process it using distribution
reductionPipeline->RunScene( scene, EPipelineRunMode::RunDistributedUsingIncredibuild )
spReductionPipeline reductionPipeline = sg.CreateReductionPipeline();
// Process a scene from file directly using distribution
reductionPipeline.RunSceneFromFile( "input.obj", "output.obj", EPipelineRunMode.RunDistributedUsingIncredibuild );
// Or if you have a scene object, process it using distribution
reductionPipeline.RunScene( scene, EPipelineRunMode.RunDistributedUsingIncredibuild )
reductionPipeline = sg.CreateReductionPipeline()
# Process a scene from file directly using distribution
reductionPipeline.RunSceneFromFile( "input.obj", "output.obj", EPipelineRunMode_RunDistributedUsingIncredibuild )
# Or if you have a scene object, process it using distribution
reductionPipeline.RunScene( scene, EPipelineRunMode_RunDistributedUsingIncredibuild )
Distributed batch tool
Distribution using Incredibuild can be enabled when invoking the batch tool (SimplygonBatch.exe
) by passing the -Incredibuild
parameter on the command line
SimplygonBatch.exe -Incredibuild <path/to/pipeline.json> <input.scene> <output.scene>
SimplygonBatch.exe -Incredibuild <path/to/pipelinebatch.json>
Note that due to Incredibuild limitations, the batch tool cannot report any progress. Also note that due to the limit of a single IBConsole active process, launching multiple SimplygonBatch instances with Incredibuild distribution will cause any second process to block until the first process has completed, effectively executing the pipelines in serial rather than parallel.