Luxonis OAK-D Camera Integration
This tutorial will guide you through the process of connecting an OAK-D camera to your Leo Rover.
The OAK-D is a camera for robotic vision that perceives the world like a human by combining a stereo depth camera and a high-resolution color camera with an on-device Neural Network inferencing and Computer Vision capabilities. In robotics, stereo cameras are commonly used for mapping the environment, object recognition, 3D scanning and in many other applications.
This tutorial is based on the Luxonis OAK-D Pro W IMX378 camera, but it will work with other OAK-D models as well (such as OAK-D S2, OAK-D Lite, OAK-D Pro).
What to expect?​
After completing this tutorial, you will have successfully connected an OAK-D stereo camera to the Leo Rover in ROS, and you will be able to visualize and use the data from the camera.
You can find all available topics and configuration options here.
Prerequisites​
Referenced products​
Hardware integration​
Mounting​
If you bought the integration from us, you can use the mounting adapter included in the package. It is designed to mount the OAK-D camera on top of the rover.
Required components:
- OAK-D camera
- Camera adapter
- M4 x 8 Allen head screw (x2)
- M5 x 10 Allen head screw (x2)
- USB-C right-angle adapter
- OAK Y-adapter
To mount the sensor:
- Insert the right-angle adapter into the USB-C port so that it faces backwards. Then, connect the OAK Y-adapter to the angle adapter.
- Insert the OAK-D camera into the adapter and secure it with the M4 x 8 bolts. Ensure that the camera's USB port faces downwards.
- Place the assembly on top of the rover and align the mounting holes with those in the front row of the rover. The camera should be perfectly centered.
At the bottom of the Wiring section, you will find a picture of the properly assembled product.
You can also mount the camera elsewhere on the rover, but you will have to adjust the camera's position and orientation in the URDF file accordingly.
You can also 3D print the adapter yourself. Get the files from the addon adapters page.
Wiring​
The OAK-D camera requires a relatively high supply current, so it can't be powered solely via the USB connection to the rover's computer. Therefore, it is necessary to provide additional 5V power supply. We strongly recommend using the Powerbox addon to do that. Without the additional power supply, the camera will not operate reliably.
To connect the camera to the Leo Rover, simply connect the data port (marked with the USB symbol) of the OAK Y-adapter to the rover's USB-C interface.
Next, connect the 5V power supply using the USB power cable provided in the package. The USB-C end should be connected to the OAK Y-adapter's power port (indicated by the lightning bolt symbol), and the other end should be connected to the Powerbox 5V array.
Ensure that the cable is connected to the 5V array (indicated by the "5V" symbol, not "BAT"). Double-check the polarity: the red cable must connect to "+" and the black cable to "-".
After assembly, the rover with the integrated OAK-D camera should look like the picture below.
Software integration​
All the steps in this section are performed on the rover's computer, so start by logging in to your rover via SSH:
Installing the packages​
The first thing you can do is to make sure your device has the correct
permissions. In /etc/udev/rules.d create a luxonis.rules file and paste the
following line into this file:
SUBSYSTEM=="usb", ATTRS{idVendor}=="03e7", MODE="0666"
Then, reload udev rules by typing:
sudo udevadm control --reload-rules && sudo udevadm trigger
Next, install a ROS package that provides a node for the camera.
sudo apt install ros-${ROS_DISTRO}-depthai-ros
Modifying the URDF model​
As we want to have the camera model visible with the rover model, we need to
create an URDF file with the OAK-D included. Package depthai_descriptions is
installed alongside depthai-ros and it contains OAK-D camera models and xacro
macros that we can easily add (list of available models
here).
In /etc/ros/urdf create the oak.urdf.xacro file that attaches the camera
model to the rover. This example uses the OAK-D Pro W model.
<?xml version="1.0"?>
<robot xmlns:xacro="http://www.ros.org/wiki/xacro">
<xacro:include filename="$(find depthai_descriptions)/urdf/include/depthai_macro.urdf.xacro"/>
<xacro:depthai_camera camera_name="oak"
camera_model="OAK-D-PRO-W"
base_frame="oak-d-base_frame"
parent="base_link"
cam_pos_x="0.09675"
cam_pos_y="-0.0485"
cam_pos_z="0.01575"
cam_roll="0.0"
cam_pitch="0.0"
cam_yaw="0.0"/>
</robot>
The parameters named cam_<> determine camera's position and orientation in the
robot model. In the file above they are set to the values that correspond to the
default mounting position of the camera on the rover. If you mounted the camera
differently, you will have to adjust these values accordingly.
Now we have to include our camera model in robot.urdf.xacro file - the
description that is uploaded at boot.
In /etc/ros/urdf/robot.urdf.xacro add this line somewhere before the closing
</robot> tag:
<xacro:include filename="/etc/ros/urdf/oak.urdf.xacro"/>
Creating launch and configuration files​
To launch the camera, you can create your own launch file, use one of the
example packages from depthai-ros package, or use the provided launch file
which provides basic functionalities such as camera image and depth data.
In /etc/ros create a .yaml configuration file. This example contains
configuration for OAK-D Pro Wide RGB and depth camera.
/oak:
ros__parameters:
oak.rgb.image_raw.enable_pub_plugins:
['image_transport/raw', 'image_transport/compressed']
oak.stereo.image_raw.enable_pub_plugins:
['image_transport/raw', 'image_transport/compressedDepth']
camera:
i_pipeline_type: RGBD
i_nn_type: none
i_laser_dot_brightness: 1000
imu:
i_acc_freq: 50
i_rot_freq: 50
i_gyro_freq: 50
rgb:
i_publish_topic: true
i_fps: 30.0
i_resolution: 12MP
i_isp_num: 1
i_isp_den: 3
i_output_isp: true
i_width: 1344
i_height: 1008
left:
i_publish_topic: false
i_fps: 10.0
i_resolution: 800P
right:
i_publish_topic: false
i_fps: 10.0
i_resolution: 800P
stereo:
i_align_depth: true
i_subpixel: true
i_stereo_conf_threshold: 110
i_enable_brightness_filter: false
i_brightness_filter_min_brightness: 1
i_brightness_filter_max_brightness: 225
i_enable_decimation_filter: true
i_publish_left_rect: false
i_publish_right_rect: false
This is a basic configuration for the OAK-D camera. Feel free to fine-tune the parameters to your needs.
Next, also in /etc/ros create a launch file that will use the previously added
configuration for the OAK-D camera node.
<launch version="0.1.1">
<node name="oak" namespace="" pkg="depthai_ros_driver" exec="camera_node">
<param from="/etc/ros/oak.yaml" />
</node>
</launch>
You can set the namespace property of the node tag to whatever you want, but
the name has to be set to the same value as the camera_name in urdf file.
It's required for the camera data to be placed in correct TF frame.
For more information about OAK-D configuration and usage examples you can check out the depthai-ros github repository.
Launching camera nodes on system startup​
Include your launch file in the robot.launch.xml file, so that your node will
start at boot.
In /etc/ros/robot.launch.xml add this line somewhere before the closing
</launch> tag:
<include file="/etc/ros/oak.launch.xml"/>
Now your OAK-D camera ROS node will start at launch. You can also start it now by typing:
ros-nodes-restart
Example usage​
Reading and visualizing the data​
The topics from the camera node should now be visible. You can check them with
the ros2 topic tool - topics from your node will have the prefix specified by
you in the name and namespace node tag properties.
If you have ROS installed on your computer, you can view the camera image with
rqt.
If you don't have ROS, you can follow this guide:
Before starting any of the programs on your computer, make sure you are connected to the rover's network:
Now you can start rqt, and from Plugins -> Visualization choose
Image View, then select your topic.
Don't worry if the quality is low, and the image is lagging. It is due to transfer of the data through your rover and computer.
You can also start RViz to see the robot model. Type rviz2 in the terminal,
or, if you have the leo_viz package installed, type:
ros2 launch leo_viz rviz.launch.xml
This will start RViz with visualization of the current robot model.
You can easily install the leo_viz package with this command:
sudo apt install ros-${ROS_DISTRO}-leo-desktop
You will also have to install the depthai-descriptions package on your
computer:
sudo apt install ros-${ROS_DISTRO}-depthai-descriptions
With RViz opened, add robot model in the display view. Chose also
base_link as fixed frame, and you should see the rover model with camera
attached.
You can also display tf tree in rqt, to see if everything is configured
correctly (plugins -> visualization -> TF Tree)
What's next?​
Stereo cameras are commonly used in projects involving autonomous navigation, you might be interested in a tutorial about it.
They are however, not the only way of teaching a Leo Rover how to move on it's own. Check out our line follower tutorial if you want to learn more. You can also check out our Integrations page for more instructions.