Skip to content

Commit

Permalink
Get rid of implementation names (#56)
Browse files Browse the repository at this point in the history
  • Loading branch information
MatthijsBurgh authored Mar 19, 2024
2 parents bccbc89 + 12a7931 commit 780b2f1
Show file tree
Hide file tree
Showing 9 changed files with 104 additions and 140 deletions.
32 changes: 16 additions & 16 deletions people_recognition_2d/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,21 @@
Node for people recognition in 2D. Defined as a service and not a publisher/subscriber.

## Custom dependencies
- [image_recognition_openpose](https://github.com/tue-robotics/image_recognition/tree/master/image_recognition_openpose)
- [image_recognition_openface](https://github.com/tue-robotics/image_recognition/tree/master/image_recognition_openpose)
- [image_recognition_keras](https://github.com/tue-robotics/image_recognition/tree/master/image_recognition_keras)
- [image_recognition_pose_estimation](https://github.com/tue-robotics/image_recognition/tree/master/image_recognition_pose_estimation)
- [image_recognition_face_recognition](https://github.com/tue-robotics/image_recognition/tree/master/image_recognition_pose_estimation)
- [image_recognition_age_gender](https://github.com/tue-robotics/image_recognition/tree/master/image_recognition_age_gender)
- [image_recognition_color_extractor](https://github.com/tue-robotics/image_recognition/tree/master/image_recognition_color_extractor)
- [image_recognition_msgs](https://github.com/tue-robotics/image_recognition/tree/master/image_recognition_msgs)

# How To
### Start the dependent nodes
The relative namespaces of the dependent nodes must be as follows so that their advertised services are the same as the values of the [parameters](#parameters) of people recognition node:
```
rosrun image_recognition_openpose openpose_node __ns:=openpose
rosrun image_recognition_pose_estimation pose_estimation_node __ns:=pose_estimation
rosrun image_recognition_openface face_recognition_node __ns:=face_recognition
rosrun image_recognition_face_recognition face_recognition_node __ns:=face_recognition
rosrun image_recognition_keras face_properties_node __ns:=face_recognition
rosrun image_recognition_age_gender face_properties_node __ns:=face_recognition
rosrun image_recognition_color_extractor color_extractor_node
```
Expand All @@ -30,12 +30,12 @@ This will create a service `detect_people` of type `RecognizePeople2D` and which
custom message type `people_recognition_msgs/Person2D`

### Parameters
| Name | Default Value |
|------------------------------|---------------------------------------|
| `~openpose_srv_name` | `openpose/recognize` |
| `~openface_srv_name` | `face_recognition/recognize` |
| `~keras_srv_name` | `face_recognition/get_face_properties`|
| `~color_extractor_srv_name` | `extract_color` |
| Name | Default Value |
|------------------------------|----------------------------------------|
| `~pose_estimation_srv_name` | `pose_estimation/recognize` |
| `~face_recognition_srv_name` | `face_recognition/recognize` |
| `~face_properties_srv_name` | `face_recognition/get_face_properties` |
| `~color_extractor_srv_name` | `extract_color` |

### Message definition of Person2D
```
Expand Down Expand Up @@ -67,9 +67,9 @@ roscd people_recognition_2d/scripts
```

# Work Flow
The node first calls the recognize services of the openpose and openface nodes. ROIs of faces are extracted from
the recognitions returned by OpenPose and are associated with the recognitions returned by OpenFace through the
face ROIs to create a `Person2D` object. The ROIs of body parts returned by OpenPose are associated with each
`Person2D` object. Face images are sent to the Keras node and properties (age and gender) are extracted and
The node first calls the recognize services of the pose_estimation and face_recognition nodes. ROIs of faces are extracted from
the recognitions returned by pose_estimation and are associated with the recognitions returned by face_recognition through the
face ROIs to create a `Person2D` object. The ROIs of body parts returned by pose_estimation are associated with each
`Person2D` object. Face images are sent to the face properties node and properties (age and gender) are extracted and
assoicated with each 'Person2D' object. The ROIs of the faces are shifted vertically to approximate the ROIs of
the shirts. These are sent to the color extractor to get the dominant colors.
12 changes: 6 additions & 6 deletions people_recognition_2d/scripts/get_people_recognition_2d
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ from people_recognition_2d import PeopleRecognizer2D

parser = argparse.ArgumentParser(description="Get 2D people recognition")
parser.add_argument("-v", "--verbose", help="Increase output verbosity", action="store_true")
parser.add_argument("--openpose_srv_name", default="openpose/recognize")
parser.add_argument("--openface_srv_name", default="people_recognition/face_recognition/recognize")
parser.add_argument("--keras_srv_name", default="people_recognition/face_recognition/get_face_properties")
parser.add_argument("--pose_estimation_srv_name", default="pose_estimation/recognize")
parser.add_argument("--face_recognition_srv_name", default="people_recognition/face_recognition/recognize")
parser.add_argument("--face_properties_srv_name", default="people_recognition/face_recognition/get_face_properties")
parser.add_argument("--color_extractor_srv_name", default="people_recognition/extract_color")

parser.add_argument("--enable-age-gender", action="store_const", default=True, const=True, dest="age_gender")
Expand All @@ -36,9 +36,9 @@ args = parser.parse_args(myargv)
rospy.init_node("get_people_recognition_2d")

recognizer = PeopleRecognizer2D(
args.openpose_srv_name,
args.openface_srv_name,
args.keras_srv_name,
args.pose_estimation_srv_name,
args.face_recognition_srv_name,
args.face_properties_srv_name,
args.color_extractor_srv_name,
args.age_gender,
args.shirt_color,
Expand Down
12 changes: 6 additions & 6 deletions people_recognition_2d/scripts/people_recognition_2d_node
Original file line number Diff line number Diff line change
Expand Up @@ -11,16 +11,16 @@ from sensor_msgs.msg import Image
class PeopleRecognition2DNode:
def __init__(self):

openpose_srv_name = self._get_param('~openpose_srv_name', "openpose/recognize")
openface_srv_name = self._get_param('~openface_srv_name', "face_recognition/recognize")
keras_srv_name = self._get_param('~keras_srv_name', "face_recognition/get_face_properties")
pose_estimation_srv_name = self._get_param('~pose_estimation_srv_name', "pose_estimation/recognize")
face_recognition_srv_name = self._get_param('~face_recognition_srv_name', "face_recognition/recognize")
face_properties_srv_name = self._get_param('~face_properties_srv_name', "face_recognition/get_face_properties")
color_extractor_srv_name = self._get_param('~color_extractor_srv_name', "extract_color")
enable_age_gender_detection = self._get_param('~enable_age_gender_detection', True)
enable_shirt_color_extraction = self._get_param('~enable_shirt_color_extraction', True)

self._people_recognizer = PeopleRecognizer2D(openpose_srv_name,
openface_srv_name,
keras_srv_name,
self._people_recognizer = PeopleRecognizer2D(pose_estimation_srv_name,
face_recognition_srv_name,
face_properties_srv_name,
color_extractor_srv_name,
enable_age_gender_detection,
enable_shirt_color_extraction,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,23 +48,22 @@ def _get_service_response(srv, args):


class PeopleRecognizer2D(object):
def __init__(self, openpose_srv_name, openface_srv_name,
keras_srv_name, color_extractor_srv_name,
def __init__(self, pose_estimation_srv_name, face_recognition_srv_name,
face_properties_srv_name, color_extractor_srv_name,
enable_age_gender_detection, enable_shirt_color_extraction):

self._openpose_srv = _get_and_wait_for_service(openpose_srv_name, Recognize)
self._openface_srv = _get_and_wait_for_service(openface_srv_name, Recognize)
self._pose_estimation_srv = _get_and_wait_for_service(pose_estimation_srv_name, Recognize)
self._face_recognition_srv = _get_and_wait_for_service(face_recognition_srv_name, Recognize)

self._enable_age_gender_detection = enable_age_gender_detection
self._enable_shirt_color_extraction = enable_shirt_color_extraction

if self._enable_age_gender_detection:
self._keras_srv = _get_and_wait_for_service(keras_srv_name, GetFaceProperties)
self._face_properties_srv = _get_and_wait_for_service(face_properties_srv_name, GetFaceProperties)

if self._enable_shirt_color_extraction:
self._color_extractor_srv = _get_and_wait_for_service(color_extractor_srv_name, Recognize)


self._bridge = CvBridge()

rospy.loginfo("People recognizer 2D initialized")
Expand All @@ -80,12 +79,12 @@ def _is_label_recognition(recognition):
return [r for r in recognitions if _is_label_recognition(r)]

@staticmethod
def _get_face_rois_ids_openpose(recognitions):
def _get_face_rois_ids_pose_estimation(recognitions):
"""
Get ROIs of faces from openpose recognitions using the nose, left ear
Get ROIs of faces from pose estimation using the nose, left ear
and right ear
:param: recognitions from openpose
:param: recognitions from pose estimation
"""
nose_recognitions = PeopleRecognizer2D._get_recognitions_with_label("Nose", recognitions)
left_ear_recognitions = PeopleRecognizer2D._get_recognitions_with_label("LEar", recognitions)
Expand Down Expand Up @@ -129,23 +128,23 @@ def _get_face_rois_ids_openpose(recognitions):
return rois, group_ids

@staticmethod
def _get_body_parts_openpose(group_id, recognitions):
def _get_body_parts_pose_estimation(group_id, recognitions):
"""
Get a list of all bodyparts associated with a particular group ID
:param: group_id: The group ID of the bodyparts to be fetched
:param: recognitions: All bodyparts recieved from openpose
:param: recognitions: All bodyparts received from pose estimation
:return: List of body_parts
"""
return [r for r in recognitions if r.group_id == group_id]

@staticmethod
def _get_container_recognition(roi, recognitions, padding_factor=0.1):
"""
Associate OpenPose ROI with best OpenPose face ROI
Associate pose estimation ROI with best pose estimation face ROI
:param: roi: openpose face roi
:recognitions: openface recognitions
:param: roi: pose estimation face roi
:param: recognitions: face recognitions
"""
x = roi.x_offset + .5 * roi.width
y = roi.y_offset + .5 * roi.height
Expand Down Expand Up @@ -244,35 +243,38 @@ def recognize(self, image_msg):
image_annotations = []
people = []

# OpenPose and OpenFace service calls
# Pose estimation and face recognition service calls
rospy.loginfo("Starting pose and face recognition...")
start_recognize = time.time()
openpose_response = _get_service_response(self._openpose_srv, image_msg)
openface_response = _get_service_response(self._openface_srv, image_msg)
pose_estimation_response = _get_service_response(self._pose_estimation_srv, image_msg)
face_recognition_response = _get_service_response(self._face_recognition_srv, image_msg)
rospy.logdebug("Recognize took %.4f seconds", time.time() - start_recognize)
rospy.loginfo("_get_face_rois_ids_openpose...")
rospy.loginfo("_get_face_rois_ids_pose_estimation...")

# Extract face ROIs and their corresponding group ids from recognitions of openpose
openpose_face_rois, openpose_face_group_ids = PeopleRecognizer2D._get_face_rois_ids_openpose(
openpose_response.recognitions)
# Extract face ROIs and their corresponding group ids from recognitions of pose estimation
pose_estimation_face_rois, pose_estimation_face_group_ids = PeopleRecognizer2D._get_face_rois_ids_pose_estimation(
pose_estimation_response.recognitions
)

body_parts_array = [PeopleRecognizer2D._get_body_parts_openpose(group_id,
openpose_response.recognitions) for group_id in
openpose_face_group_ids]
body_parts_array = [
PeopleRecognizer2D._get_body_parts_pose_estimation(group_id, pose_estimation_response.recognitions)
for group_id in pose_estimation_face_group_ids
]

face_recognitions = [PeopleRecognizer2D._get_container_recognition(openpose_face_roi,
openface_response.recognitions)
for openpose_face_roi in openpose_face_rois]
face_recognitions = [
PeopleRecognizer2D._get_container_recognition(pose_estimation_face_roi, face_recognition_response.recognitions)
for pose_estimation_face_roi in pose_estimation_face_rois
]

face_labels = [PeopleRecognizer2D._get_best_label(r) for r in face_recognitions]

# Keras service call
# Face properties service call
if self._enable_age_gender_detection:
rospy.loginfo("_get_face_properties...")
face_image_msg_array = [self._bridge.cv2_to_imgmsg(PeopleRecognizer2D._image_from_roi(image, r.roi), "bgr8") for
r in face_recognitions]
keras_response = _get_service_response(self._keras_srv, face_image_msg_array)
face_properties_array = keras_response.properties_array
face_properties_response = _get_service_response(self._face_properties_srv, face_image_msg_array)
face_properties_array = face_properties_response.properties_array
else:
face_properties_array = [FaceProperties()] * len(face_recognitions)

Expand Down
2 changes: 1 addition & 1 deletion people_recognition_3d/config/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ link_threshold: 0.5
heuristic: "shoulder"
arm_norm_threshold: 0.5
neck_norm_threshold: 0.7
wave_threshold: 0.2
waving_threshold: 0.2
vert_threshold: 0.7
hor_threshold: 0.4
padding: 5
2 changes: 2 additions & 0 deletions people_recognition_3d/package.xml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@
<exec_depend>cv_bridge</exec_depend>
<exec_depend>geometry_msgs</exec_depend>
<exec_depend>image_geometry</exec_depend>
<exec_depend>image_recognition_pose_estimation</exec_depend>
<exec_depend>image_recognition_util</exec_depend>
<exec_depend>message_filters</exec_depend>
<exec_depend>people_recognition_2d</exec_depend>
<exec_depend>people_recognition_msgs</exec_depend>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ def _generate_dummy_person3d(self, rgb, depth, cam_info, name=None):
colors = ['black', 'orange', 'yellow']
random.shuffle(colors)
person.shirt_colors = colors
person.tags = ['LWave', 'RWave'] + random.choice([[], ["is_pointing"]])
person.tags = ['LWaving', 'RWaving'] + random.choice([[], ["is_pointing"]])
# person.body_parts_pose
person.position = Point()
person.face.roi = RegionOfInterest(width=200, height=200)
Expand Down
4 changes: 2 additions & 2 deletions people_recognition_3d/scripts/people_recognition_3d_node
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ class PeopleRecognition3DNode:
heuristic = self._get_param('~heuristic', 'shoulder')
arm_norm_threshold = self._get_param('~arm_norm_threshold', 0.5)
neck_norm_threshold = self._get_param('~neck_norm_threshold', 0.7)
wave_threshold = self._get_param('~wave_threshold', 0.2)
waving_threshold = self._get_param('~waving_threshold', 0.2)
vert_threshold = self._get_param('~vert_threshold', 0.7)
hor_threshold = self._get_param('~hor_threshold', 0.4)
padding = self._get_param('~padding', 5)
Expand All @@ -33,7 +33,7 @@ class PeopleRecognition3DNode:

self._people_recognizer_3d = PeopleRecognizer3D(
recognize_people_srv_name, probability_threshold, link_threshold,
heuristic, arm_norm_threshold, neck_norm_threshold, wave_threshold,
heuristic, arm_norm_threshold, neck_norm_threshold, waving_threshold,
vert_threshold, hor_threshold, padding)

self._markers_pub = rospy.Publisher('~viz',
Expand Down
Loading

0 comments on commit 780b2f1

Please sign in to comment.