ZED SDK and Fusion API

I am currently using the ZED SDK 5.0 in conjunction with two ZED 2i’s. I am experimenting using the Fusion API in C++ to perform some basic spatial mapping with the two cameras before I move on to more advanced use cases through the Fusion API. I’ve written some basic code, which I will attach below, with some minor tweaks while referring to the examples provided in the Git repository. However, every time I run the code, I get an unhandled exception saying “ucrtbase.pdb not included”. I predominantly use the Python version of the API to develop, and I believe not all of the Fusion API is ported to Python as of yet? As I could not access certain methods that are accessible through the C++ API. I have more experience using the Python API and very little using the C++ side of things. I wanted to know if anyone else was facing this issue, and if so, what they did to counter it. Any help would be appreciated.

Kind regards

# The header files ClientPublisher and utils are directly copied from the ZED SDK repository on GitHub
#include "ClientPublisher.hpp"
#include "utils.hpp"
#include <chrono>
#include <thread>

static const int CAPTURE_DURATION = 1;

static void FusionCapture()
{
	constexpr sl::COORDINATE_SYSTEM COORDINATE_SYSTEM = sl::COORDINATE_SYSTEM::RIGHT_HANDED_Y_UP;
	constexpr sl::UNIT UNIT = sl::UNIT::METER;

	auto configuration = sl::readFusionConfigurationFile("TestCalibration.json", COORDINATE_SYSTEM, UNIT);

	if (configuration.empty())
	{
		std::cout << "Empty configuration file" << std::endl;
		return;
	}

	std::vector<ClientPublisher> clients(configuration.size());
	int id = 0;
	std::map<int, std::string> svoFiles;

	for (auto conf : configuration)
	{
		if (conf.communication_parameters.getType() == sl::CommunicationParameters::COMM_TYPE::INTRA_PROCESS)
		{
			std::cout << "Try to open ZED " << conf.serial_number << ".." << std::flush;
			auto state = clients[id].open(conf.input_type);

			if (!state)
			{
				std::cerr << "Could not open ZED: " << conf.input_type.getConfiguration() << ". Skipping..." << std::endl;
				continue;
			}

			if (conf.input_type.getType() == sl::InputType::INPUT_TYPE::SVO_FILE)
			{
				svoFiles.insert(std::make_pair(id, conf.input_type.getConfiguration()));
			}

			std::cout << "Ready!" << std::endl;
			id++;
		}
	}

	bool enableSVOSync = (svoFiles.size() > 1);
	if (enableSVOSync)
	{
		std::cout << "Starting SVO Sync Process..." << std::endl;
		std::map <int, int> cam_idx_to_svo_frame_idx = syncDATA(svoFiles);

		for (auto& it : cam_idx_to_svo_frame_idx)
		{
			std::cout << "Setting camera " << it.first << " to frame " << it.second << std::endl;
			clients[it.first].setStartSVOPosition(it.second);
		}
	}

	for (auto& it : clients)
	{
		it.start();
	}

	sl::InitFusionParameters initParameters;
	initParameters.coordinate_units = UNIT;
	initParameters.coordinate_system = COORDINATE_SYSTEM;
	initParameters.verbose = true;

	sl::Fusion fusion;
	auto state = fusion.init(initParameters);
	if (state != sl::FUSION_ERROR_CODE::SUCCESS)
	{
		std::cout << "Error Init " << state << std::endl;
		return;
	}

	std::vector <sl::CameraIdentifier> cameras;
	for (auto& it : configuration)
	{
		sl::CameraIdentifier uuid(it.serial_number);
		auto state = fusion.subscribe(uuid, it.communication_parameters, it.pose, it.override_gravity);
		if (state != sl::FUSION_ERROR_CODE::SUCCESS)
		{
			std::cout << "Unable to subscribe to " << std::to_string(uuid.sn) << " . " << state << std::endl;
		}
		else
		{
			cameras.push_back(uuid);
		}
	}

	if (cameras.empty())
	{
		std::cout << "No connections" << std::endl;
		return;
	}

	sl::PositionalTrackingFusionParameters positionalTrackingParameters;
	state = fusion.enablePositionalTracking(positionalTrackingParameters);
	if (state != sl::FUSION_ERROR_CODE::SUCCESS)
	{
		std::cout << "Error Positional Tracking" << state << std::endl;
		return;
	}

	sl::SpatialMappingFusionParameters spatialMappingParameters;
	spatialMappingParameters.map_type = sl::SpatialMappingParameters::SPATIAL_MAP_TYPE::MESH;
	sl::Mesh map;

	spatialMappingParameters.set(sl::SpatialMappingParameters::MAPPING_RANGE::SHORT);
	spatialMappingParameters.set(sl::SpatialMappingParameters::MAPPING_RESOLUTION::HIGH);
	spatialMappingParameters.use_chunk_only = true;
	spatialMappingParameters.stability_counter = 4;

	state = fusion.enableSpatialMapping(spatialMappingParameters);
	if (state != sl::FUSION_ERROR_CODE::SUCCESS)
	{
		std::cout << "Error Spatial Mapping" << state << std::endl;
		return;
	}

	sl::Timestamp lastUpdate = 0;
	bool waitForMesh = false;

	auto startTime = std::chrono::high_resolution_clock::now();

	while (true)
	{
		auto now = std::chrono::high_resolution_clock::now();
		double elapsedSeconds = std::chrono::duration<double>(now - startTime).count();

		if (elapsedSeconds > CAPTURE_DURATION)
		{
			break;
		}

		if (fusion.process() == sl::FUSION_ERROR_CODE::SUCCESS)
		{
			auto ts = sl::getCurrentTimeStamp();
			if (!waitForMesh && (ts.getMilliseconds() - lastUpdate.getMilliseconds() > 100))
			{
				fusion.requestSpatialMapAsync();
				waitForMesh = true;
			}

			if (waitForMesh && fusion.getSpatialMapRequestStatusAsync() == sl::FUSION_ERROR_CODE::SUCCESS)
			{
				fusion.retrieveSpatialMapAsync(map);
				waitForMesh = false;
				lastUpdate = ts;
			}
		}
	}
	
	map.save("Map.ply", sl::MESH_FILE_FORMAT::PLY);

	for (auto& it : clients)
	{
		it.stop();
	}

	fusion.close();

	return;

}

Hi,

Are you on Windows? Can you make sure you are not compiling your program in Debug and not Release?

And yes, you can use python, everything is available as well.

I have tried doing it in Python. However, it does not allow me to access the sl.SpatialMappingFusionParameters() method. I get the error module 'pyzed.sl' has no attribute 'SpatialMappingFusionParameters'. I have made use of the Fusion API for body tracking with multiple cameras, however this one, I was struggling with. Do advise if I was calling the wrong function.

Yes, I am on Windows. And I was compiling in Release mode and not Debug mode, mainly because I get further in the program when I run the program in Release mode. For example, starting the cameras through the Fusion configuration file does not happen when I run the program in debug mode. It just tells me that I have an empty configuration file, which is not true. It must be that I am doing something wrong. Is there a reason why it has to be in Debug mode?

No, on the contrary, you must run it in Release.
I just double check and it seems the SpatialMappingFusionParameters struct is missing in the Python wrapper. The issue has been logged on our side, and we will fix it for the next patch.

Great. That sounds helpful! I’m going to continue looking into why the C++ program doesn’t run in that case. It must be something on my end. Would it be possible to get a time estimate on when the next patch will be out in the meantime? Even if it’s a rough one. Thanks in advance.

Unfortunately, we just released a new patch this week, so there is no new update in the near future, unless a major issue is discovered.

I can help you fix your cpp issue. Did you try to use the “official” c++ sample (https://github.com/stereolabs/zed-sdk/tree/master/spatial%20mapping/multi%20camera/cpp/) ? Do you have the same issue?

Could you tell me where the code crashes?

Thanks.

Stereolabs Support