Color filtering a mesh .OBJ with .MTL


I am trying to filter out a 3D scene based on color (e.g. RGB or HSV) from the .OBJ and .MTL files. For example, to localize a colored balloon. I have tried using Open3D in Python, however it does not yet support extracting face information or .MTL file.

Any suggestions on how I can get color info with the mesh using .OBJ and .MTL?