AI Target Recognition
Overview
Once target recognition is enabled, the aircraft detects targets in the live view in real time. Pedestrians, cyclists, vehicles, boats, drones, smoke and fire can be recognized. See "Target Types" for the complete list.
The following capabilities are built on top of target recognition:
| Capability | Description |
|---|---|
| Target tracking | Tap or draw a box on a target in the live view, and the aircraft keeps following it. Simple tracking and orbit tracking are supported |
| Target lock auto zoom | After a target is locked, the zoom ratio is adjusted automatically to keep the target at a suitable size in the view |
| Area search | The aircraft flies along a route within a specified polygonal area and recognizes targets |
| AI arming | Recording or photo capture is triggered automatically when a target of the specified type is recognized |
| Illegal parking evidence | Evidence is collected for the selected vehicle, and the license plate, body color and position are reported |
Recognition results are delivered over two channels: the video stream SEI carries target boxes, target types, confidence and tracking IDs frame by frame, so that the pull side can overlay them on the live view in real time; MQTT events carry detection frame image URLs, illegally parked vehicle information and evidence collection progress for business records and further processing.
The MQTT events are listed below. For their field definitions, see "Detailed Interface Implementation":
| Method | Description |
|---|---|
| target_detect_result_report | Target recognition result report, including the detection frame image URL and the target list |
| evidence_collection_status | Task progress and per-vehicle results of illegal parking evidence collection |
| ai_service_illegal_parking_info | License plate, color, position, detection time and evidence image URLs of the illegally parked vehicle |
If any error occurs, check the corresponding error description using the error codes listed in the Error Codes section.
AI_detect_shoot_notify(AI detection shooting result report) is deprecated. These results are now carried by the video stream SEI.
Video Stream SEI Parsing
SEI (Supplemental Enhancement Information) is frame-aligned metadata in the H.264/H.265 bitstream and stays strictly aligned with the video content. Recognition results are periodically written into the bitstream as SEI of the user_data_unregistered type (payloadType = 5). The payload is Protobuf serialized data (proto3, package video.sei, message SeiPayload).
Encapsulation
| Section | Length | Content |
|---|---|---|
| Start code | 4 bytes | 00 00 00 01 |
| NAL header | 1 byte for H.264, 2 bytes for H.265 | H.264 06 (nal_unit_type = 6); H.265 4E 01 (nal_unit_type = 39, PREFIX_SEI) |
| payloadType | 1 byte | 05, that is user_data_unregistered |
| payloadSize | Variable | Each 0xFF adds 255 and the last byte is the remainder. Its value is the length of the RBSP region, that is 16 + 4 + the Protobuf byte count, excluding emulation prevention bytes |
| UUID | 16 bytes | dc45e9bd-e6d9-48b7-962c-d820d923aaaa |
| Identifier | 4 bytes | 41 74 41 69, that is AtAi |
| Protobuf data | payloadSize - 20 | SeiPayload serialized bytes |
| Trailing byte | 1 byte | 80 |
The two encodings differ only in the NAL header length; everything after it is identical.
Parsing Flow
- Split NAL units by the start code
00 00 00 01. - Read the NAL header to determine the type.
nal_unit_type = 6for H.264 andnal_unit_type = 39for H.265 are SEI; skip everything else. - Read payloadType, which must be 5.
- Read the variable-length payloadSize to get the length of the RBSP region.
- Remove the emulation prevention byte
0x03(inserted when consecutive00 00is followed by a value<= 03). - Verify that the first 16 bytes are the UUID
dc45e9bd-e6d9-48b7-962c-d820d923aaaaand the following 4 bytes are the identifier41 74 41 69. User data SEI that does not match belongs to another purpose and must be ignored. - Deserialize the remaining bytes as
SeiPayload.
Step 5 must come before step 7: emulation prevention bytes may be inserted in the middle of the Protobuf data, and float fields are a frequent location in practice. See "Sample Bytes" for a byte-level comparison.
Message Definitions
syntax = "proto3";
package video.sei;
message SeiPayload {
// Frame time
int32 year = 1;
int32 month = 2;
int32 day = 3;
int32 hour = 4;
int32 minute = 5;
int32 second = 6;
int32 millisecond = 7;
SeiPayloadCamera camera = 8;
SeiPayloadUav uav = 9;
SeiPayloadSpeed speed = 10;
AIServiceDetectResult detect_result = 11;
ARTrajectoryProjection trajectory = 12;
}
message SeiPayloadCamera {
int32 sei_type = 1;
int32 length = 2;
int32 gimbal_type = 3;
int32 zoom_value = 4;
int32 focus_length = 5;
int32 focus_length_35mm = 6;
int32 h_fov = 7;
int32 v_fov = 8;
}
message SeiPayloadUav {
int32 sei_type = 1;
int32 length = 2;
int32 lon = 3;
int32 lat = 4;
int32 abs_alt = 5;
int32 sea_alt = 6;
int32 rel_alt = 7;
int32 uav_yaw = 8;
int32 uav_roll = 9;
int32 uav_pitch = 10;
int32 gimbal_pitch = 11;
int32 gimbal_roll = 12;
int32 gimbal_yaw = 13;
int32 lon_center = 14;
int32 lat_center = 15;
int32 lon_tl = 16;
int32 lat_tl = 17;
int32 lon_tr = 18;
int32 lat_tr = 19;
int32 lon_bl = 20;
int32 lat_bl = 21;
int32 lon_br = 22;
int32 lat_br = 23;
}
message SeiPayloadSpeed {
int32 sei_type = 1;
int32 length = 2;
int32 x_speed = 3;
int32 y_speed = 4;
int32 z_speed = 5;
}
message AIServiceDetectResult {
int32 sei_type = 1;
int32 length = 2;
int32 len_id = 3;
int32 mode_id = 4;
int32 frame_id = 5;
int32 resolution_width = 6;
int32 resolution_height = 7;
int32 obj_num = 8;
repeated DetectObjInfo detect_obj_infos = 9;
}
message DetectObjInfo {
float start_x = 1;
float start_y = 2;
float width = 3;
float height = 4;
int32 type = 5;
int32 confidence = 6;
int32 object_id = 7;
int32 status = 8;
uint32 car_color = 9;
uint32 car_type = 10;
uint32 gender = 11;
uint32 age = 12;
uint32 top_type = 13;
uint32 lower_type = 14;
uint32 top_color = 15;
uint32 lower_color = 16;
uint32 military = 17;
uint32 danger_level = 18;
}
message ARTrajectoryProjection {
int32 sei_type = 1;
int32 length = 2;
bytes traj_px = 3;
bytes traj_py = 4;
bytes mission_px = 5;
bytes mission_py = 6;
bytes home_px = 7;
bytes home_py = 8;
int32 camera_id = 9;
}
Four rules to keep in mind when decoding:
start_x,start_y,widthandheightare float32 (wire type 5, fixed 4-byte little-endian IEEE 754), while all other numeric fields are int32 varints. Parsing these four fields as varints yields incorrect results.- proto3 does not serialize zero values. A missing field must always be treated as 0, and neither the frame nor the target may be discarded because a field is absent. In practice
focus_length_35mmandgimbal_rollnever appear, and the three speed axes appear in only some of the frames. - Negative int32 values are encoded with 64-bit sign extension and take 10 bytes. For example, a
rel_altof -100 (that is -10 m) is encoded as9C FF FF FF FF FF FF FF FF 01. When decoding varints manually, read them as signed 64-bit values and then truncate to int32. - In each sub-message, the measured
lengthdoes not match the actual byte count. Use the length prefixes provided by Protobuf itself and ignore that field.
Field Reference
The top level carries the frame timestamp (year through millisecond) and five sub-messages, each identified by its sei_type. The first four are populated in every frame. trajectory is injected only during autonomous flight when a valid trajectory exists, so check its presence in every frame and do not cache the previous frame. Where a table below has a "Scale" column, it gives the integer scaling factor: the real value equals the raw value divided by the scale.
Camera Parameters (camera, sei_type = 1)
| Field | Description | Scale |
|---|---|---|
| gimbal_type | Gimbal form | — |
| zoom_value | Zoom ratio | 100 |
| focus_length / focus_length_35mm | Focal length and 35mm equivalent focal length (mm) | 10 |
| h_fov / v_fov | Horizontal and vertical field of view (°) | 10 |
Position and Attitude (uav, sei_type = 2)
| Field | Description | Scale |
|---|---|---|
| lon / lat | Aircraft longitude and latitude (°) | 10⁷ |
| abs_alt / sea_alt / rel_alt | Absolute, sea-level and takeoff-relative altitude (m) | 10 |
| uav_yaw / uav_roll / uav_pitch | Airframe yaw, roll and pitch (°) | 100 |
| gimbal_pitch / gimbal_roll / gimbal_yaw | Gimbal pitch, roll and yaw (°) | 100 |
| lon_center / lat_center | Projected longitude and latitude of the view center (°) | 10⁷ |
| lon_tl to lat_br | Projected longitude and latitude of the four view corners (°), in top-left, top-right, bottom-left, bottom-right order | 10⁷ |
Speed (speed, sei_type = 3)
| Field | Description | Scale |
|---|---|---|
| x_speed / y_speed / z_speed | North, east and down speed (m/s) | 100 |
Detection and Tracking (detect_result, sei_type = 4)
| Field | Description |
|---|---|
| len_id | Lens type: 0 wide, 1 zoom, 2 infrared, 3 night vision |
| resolution_width / resolution_height | Width and height of the image the detection is based on (px) |
| obj_num / detect_obj_infos | Number of targets and the target array, whose length equals obj_num |
Elements of the target array, DetectObjInfo:
| Field | Description |
|---|---|
| start_x / start_y / width / height | Top-left coordinates, width and height of the target box, normalized to 0-1 |
| type / confidence / object_id | Target type (see "Target Types"), confidence 0-100 and tracking ID |
Convert the target box to pixels with pixel = ratio × the corresponding detection image dimension. The detection image dimensions may differ from the playback resolution; if the playback side scales the video, convert once more against the playback dimensions. Extended attributes from status to danger_level are not reported by the current firmware.
AR Trajectory Projection (trajectory, sei_type = 5)
The positions of trajectory points, mission points and the home point in the current view, for the playback side to overlay.
| Field | Description |
|---|---|
| traj_px / traj_py | Sequences of trajectory point x and y values, of equal length, connected point by point into a polyline |
| mission_px / mission_py | Mission point x and y |
| home_px / home_py | Home point x and y |
| camera_id | Lens type, using the same values as len_id |
Coordinates are ratio values from 0 to 255, so pixel = value / 255 × view dimension. For mission_* and home_*, a length of 1 means the point is inside the view and gives its projected position, while a length of 2 means it is outside the view and the two values indicate its approximate bearing.
Target Types
The table below lists the values of the type field in the recognition results. The same values are used in target_type_list to specify the target types to be recognized, and an empty list means all types supported by the current scene are recognized.
| Target Type ID | Target Name |
|---|---|
| 0 | Background(Not supported yet) |
| 1 | Animal(Not supported yet) |
| 2 | Boat |
| 3 | Car |
| 4 | Pedestrian |
| 5 | Cyclist |
| 6 | Large Vehicle |
| 34 | Drone |
| 35 | Smoke |
| 36 | Fire |
| 99 | No Target Detected |
Sample Bytes
One H.265 SEI frame: an 887-byte NAL containing 1 emulation prevention byte, an 886-byte RBSP after unescaping, of which 858 bytes are Protobuf.
The first 32 bytes as they appear in the bitstream, in order: start code, NAL header, payloadType, payloadSize, UUID and identifier. Offsets here include the start code and are therefore 4 larger than in the table below:
00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F |ASCII|
00000000 00 00 00 01 4E 01 05 FF FF FF 71 DC 45 E9 BD E6 |....N.....q.E...|
00000010 D9 48 B7 96 2C D8 20 D9 23 AA AA 41 74 41 69 08 |.H..,. .#..AtAi.|
The only emulation prevention byte in this frame falls inside obj[2].height. Before and after unescaping:
Bitstream … 3C 25 00 00 03 00 3D 28 03 30 …
RBSP … 3C 25 00 00 00 3D 28 03 30 …
25 is the tag of height, and only the following 4 bytes 00 00 00 3D are the float32 value 0.03125. Leaving the 0x03 in place misreads not just this value: every subsequent field boundary shifts as well.
A field-by-field breakdown after unescaping. The timestamp fields and the identically shaped view corners are folded, and only the first element of the target array is listed:
Offset Section Field No. Raw bytes Raw value Scaled / meaning
- Envelope start code - 00 00 00 01 4 bytes before RBSP, not counted in offset
0x00 Envelope nal_unit_header - 4E 01 2 bytes H.265 header, nal_unit_type = 39
0x02 Envelope payloadType - 05 5 user_data_unregistered
0x03 Envelope payloadSize - FF FF FF 71 878 = 16 + 4 + 858
0x07 Envelope uuid - DC 45 E9 BD E6 D9 48 B7 … 16 bytes ✓ matches this protocol
0x17 Envelope identifier - 41 74 41 69 "AtAi" ✓
0x1B SeiPayload year ~ millisecond 1~7 08 EA 0F … 38 AA 05 7 fields frame time 2026-08-21 14:23:57.682
0x2B camera camera 8 42 10 16 bytes submessage SeiPayloadCamera
0x2D camera sei_type 1 08 01 1 camera
0x2F camera length 2 10 0C 12 16 bytes in fact, do not use for boundaries
0x31 camera gimbal_type 3 18 01 1
0x33 camera zoom_value 4 20 64 100 1 x
0x35 camera focus_length 5 28 58 88 8.8 mm
0x37 camera h_fov 7 38 C0 04 576 57.6 deg
0x3A camera v_fov 8 40 C9 03 457 45.7 deg
0x3D uav uav 9 4A 72 114 bytes submessage SeiPayloadUav
0x3F uav sei_type 1 08 02 2 pose
0x41 uav length 2 10 42 66
0x43 uav lon 3 18 BF E0 D8 A0 04 1142304831 114.2304831 deg
0x49 uav lat 4 20 C4 F8 EB 6D 230358084 23.0358084 deg
0x4E uav abs_alt 5 28 CB 59 11467 1146.7 m
0x51 uav sea_alt 6 30 8E 5C 11790 1179 m
0x54 uav rel_alt 7 38 9C FF FF FF FF FF FF FF FF 01 -100 -10 m
0x5F uav uav_yaw 8 40 42 66 0.66 deg
0x61 uav uav_roll 9 48 01 1 0.01 deg
0x63 uav uav_pitch 10 50 0D 13 0.13 deg
0x65 uav gimbal_pitch 11 58 B2 FF FF FF FF FF FF FF FF 01 -78 -0.78 deg
0x70 uav gimbal_yaw 13 68 3F 63 0.63 deg
0x72 uav lon_center 14 70 D6 E3 D8 A0 04 1142305238 114.2305238 deg
0x78 uav lat_center 15 78 C2 FC EB 6D 230358594 23.0358594 deg
0x7D uav lon_tl 16 80 01 9B E2 D8 A0 04 1142305051 114.2305051 deg
the other 6 corner fields share the same shape, field 17-22
0xAB uav lat_br 23 B8 01 88 F9 EB 6D 230358152 23.0358152 deg
0xB1 speed speed 10 52 0F 15 bytes submessage SeiPayloadSpeed
0xB3 speed sei_type 1 08 03 3 speed
0xB5 speed length 2 10 06 6
0xB7 speed z_speed 5 28 FF FF FF FF FF FF FF FF FF 01 -1 -0.01 m/s
0xC2 detect_result detect_result 11 5A B0 05 688 bytes submessage AIServiceDetectResult
0xC5 detect_result sei_type 1 08 04 4 detection
0xC7 detect_result length 2 10 A3 06 803
0xCA detect_result len_id 3 18 01 1 zoom
0xCC detect_result mode_id 4 20 01 1
0xCE detect_result frame_id 5 28 FF FF 03 65535
0xD2 detect_result resolution_width 6 30 80 0A 1280
0xD5 detect_result resolution_height 7 38 C0 07 960
0xD8 detect_result obj_num 8 40 18 24
0xDA detect_result.obj[0] detect_obj_infos 9 4A 1A 26 bytes submessage DetectObjInfo
0xDC detect_result.obj[0] start_x 1 0D 5E 13 0B 3F 0.543264 float32
0xE1 detect_result.obj[0] start_y 2 15 9A 99 39 3F 0.725 float32
0xE6 detect_result.obj[0] width 3 1D 86 4E F8 3C 0.030311 float32
0xEB detect_result.obj[0] height 4 25 9A 99 19 3D 0.0375 float32
0xF0 detect_result.obj[0] type 5 28 03 3
0xF2 detect_result.obj[0] confidence 6 30 5A 90 0.9
0xF4 detect_result.obj[0] object_id 7 38 02 2
the other 23 targets share the same structure, 24 in total
The frame carries 24 targets. With resolution_width = 1280 and resolution_height = 960, the first target box is located at (695, 696) and measures 39 × 36 pixels. An H.264 frame differs only in the single-byte NAL header 06; everything after it is identical.
Target Tracking
Target tracking commands are issued in the order intelligent_lock_enter → select_tracking_target → start_track. The aircraft must already be airborne when start_track is issued.
A target can be selected in two ways:
| Method | Parameters |
|---|---|
| Tap | width and height are 0, and start_x and start_y are the coordinates of the target point |
| Box | width and height are greater than 0 and form the target box together with the start point |
The tracking mode is specified by track_type:
| track_type | Description |
|---|---|
| 1 | Simple tracking. The aircraft follows the moving target |
| 3 | Orbit tracking. The aircraft orbits around the target |
locked determines the relationship between the aircraft nose and the gimbal: when it is false, only the gimbal rotates and the airframe stays still; when it is true, the nose rotates together with the gimbal.
There are three levels of exit: target_tracking_close exits the target lock but keeps detecting targets, intelligent_lock_exit exits intelligent tracking, and target_detect_close disables target recognition.
Interaction Sequence Diagram
Detailed Interface Implementation
- Enable Target Recognition
- Disable Target Recognition
- Enter Intelligent Tracking
- Exit Intelligent Tracking
- Select Locked Target Information
- Exit Target Lock
- Start Tracking
- Target Lock Auto Zoom
- Area Search
- Select Illegal Parking Target
- AI Arming Settings
- Target Recognition Result Report
- Evidence Collection Status Report
- Illegal Parking Detection Info Report