← Back to packages

Package biopb.image

Service ObjectDetection

Service ProcessImage

Message BinData

Message DetectionRequest

Message DetectionResponse

Message DetectionSettings

Message Ellipse

Message Face

Message ImageAnnotation

Message ImageData

Message Mask

Message Mesh

Message OpNames

Message Pixels

Message Point

Message Polygon

Message ProcessRequest

Message ProcessResponse

Message ROI

Message Rectangle

Message ScoredROI

Enum Endianness

Services

service ObjectDetection

Unitary call for computing cell detection / segmentation.

The streaming version allows dynamic updating of the DetectionRequest and getting updated results. For example, a client may request analyses at several different settings on the same image. After initial request, the following streaming requests no long need to transmit the image pixel data anymore, but only the new parameter settings, which saves network bandwidth.

Send the image dataset as a stream of (overlapping or non-overlapping) image patches, instead of one large chunk of binary data. There are two use cases for this call: (1) The image dataset is too large therefore sending all data in one message would exceed GRPC message size limit; (2) The original data were taken as a grid scan so it is convinent to read/write as patches.

rpc RunModelAdaptation
Request: biopb.image.DetectionRequest
Response: google.protobuf.Any

Run model adaptation based on the submitted stream of DetectionRequest, which may or may not contain ROI annotations. The all returns an opaque data strcuture representing the model configuration after model adaptation, which in turn should be included in future inference reuest as a DetectionSetting field.

service ProcessImage

A generic image process protocol that returns a new image. Examples include denoising, super-resolution, and semantic segmentation.

Process the input image

rpc GetOpNames
Request: google.protobuf.Empty
Response: biopb.image.OpNames

Query operation names. Server with a single op may not implement this call.

Messages

message BinData

A binary stream

Field Type Description
1 data bytes
2 endianness biopb.image.BinData.Endianness

message DetectionRequest

Field Type Description
1 image_data biopb.image.ImageData

Encapsulate all pixel data and associated metadata. In a streaming call, this field is allowed to be null, which simply is a request to re-analyze the last submitted image with the updated settings. If this is the first call in a stream, the server will not return results, but simply update and remember the settings.

2 detection_settings biopb.image.DetectionSettings

Other parameters associated with the request, such as score threshold etc. In a streaming call, this field is allowed to be null, in which case the server will use the setings of the previous request.

3 field_mask google.protobuf.FieldMask

A field_mask to indicate the requested fields in the reply. Default is to return the full response.

message DetectionResponse

Field Type Description
1 detections repeated biopb.image.ScoredROI

message DetectionSettings

Field Type Description
1 min_cell_area optional float

Minimal cell area to be consider for detection. The unit is decided by the physical unit of the image, i.e., if the physical pixel size unit is "nm", then the unit here is nm^2 (2D) or nm^3 (3D). However, if the physical pixel size field is not set, then the unit here is pixels.

2 min_score optional float

Minimal detection score in order to be considered. If unset, the default value is chosen server-side.

3 segmentation_threshold optional float

Threshold for binarize segmentation logits. If unset, default to 0.

4 nms_iou optional float

A threshold value for remove overlapping detections. If two detection has an IOU > this threshold, the one with the lower score will be eliminated from the detection list. Nothing will be removed if the field is not set.

8 model_name string

Specify the model name for servers that support more than one model variants.

9 model_configuration google.protobuf.Any

An opaque data structure that specify a server side model configuration. An important use case is to perform few-shot model adaptations. The model adaptation call returns a new model configuration. This stucture should be included in all new inference requests, so that the server understand that it should perform computation using the adapted model.

oneof resize_info
6 cell_diameter_hint float

A hint for the typical cell diameter. If the physical pixel size is known, the value takes the same physical unit. Otherwise, the unit is pixel.

7 scaling_hint float

A direct suggestion for image scaling. A nonnegative value. However, requesting a very large scaling factor will likely be refused by the server due to the potential out-of-memory problem.

message Ellipse

A simple ellipse object.

Field Type Description
1 center biopb.image.Point
2 radius biopb.image.Point

If the center`` has no z field, neither shouldradius`

message Face (Nested in biopb.image.Mesh )

A trangle defined three indices of the verts array

Field Type Description
1 p1 uint32
2 p2 uint32
3 p3 uint32
4 normals repeated biopb.image.Point

Optionally one can cache the computed normal vectors here

5 values repeated float

Optionally one can cache the associated face values here

message ImageAnnotation

Additional annotation of the image, which may be used for the analysis.

Field Type Description
1 rois repeated biopb.image.ROI

a list of region of interest

2 annotation string

additional text annotation

message ImageData

Field Type Description
1 pixels biopb.image.Pixels

Encapsulated image data

2 image_annotation biopb.image.ImageAnnotation

Optional additional information regarding the image that will be utilized by the RPC call. For example, some SegmentAnything type models may ask the user to provide a tentative bounding box for the objects that the users are interested in.

message Mask

The Mask ROI shape is a link to a BinData object that is a BIT mask drawn on top of the image as an ROI. It is applied at the same scale, pixel to pixel, as the Image the ROI is applied to. The dimension order of the mask is the same as the that of the image data the Roi is applied to.

Field Type Description
1 rectangle biopb.image.Rectangle
2 bin_data biopb.image.BinData

message Mesh

A closed 3D shape repesented by triangular meshes.

Field Type Description
1 verts repeated biopb.image.Point

All vertices of the mesh

2 faces repeated biopb.image.Mesh.Face

message OpNames

Field Type Description
1 names repeated string

a list of op names

message Pixels

The Pixels element represent generic 5D pixel data. The time axis is currently not being utilized, but is defined here for futire extensibility. This is modeled after OME-XML's Pixel structure.

Field Type Description
3 bindata biopb.image.BinData

Pixel values

4 dimension_order string

A string describing the dimension order of the pixel data, e.g. "XYZCT" The string must be a permutaion of these 5 letters even if some of the dimension doesn't exist in the data.

5 dtype string

A string describing the data type using numpy's convention. E.g., 'i8' represent int64. Endianness indicator, e.g. '=i8', is allowed but will be ignored. Instead the endianness will be determined by the field within the Bindata structure.

9 size_x uint32

Dimensional size of pixel data array [units:none]

10 size_y uint32
11 size_z uint32
12 size_c uint32
13 size_t uint32
14 physical_size_x optional float

Physical size of a pixel.

15 physical_size_y optional float
16 physical_size_z optional float
17 physical_size_unit string

physical size unit, default "µm"

18 time_increment optional float

time_increment is used for time series that have a global timing specification instead of per-timepoint timing info. For example in a video stream. Units are set by TimeIncrementUnit.

21 time_increment_unit string

time unit, default "s"

22 offset_x optional uint32

for large image dataset, it is necessary to send a stream of image patches instead of one monolithic data structure of the whole dataset. The offset values allows one to specify the relative locations of each patch.

23 offset_y optional uint32
24 offset_z optional uint32
25 offset_t optional uint32

message Point

/////////// ROI classes are desgined roughly after OME-XML's related datatypes, but with a emphasis on 3D data

Field Type Description
1 x float
2 y float
3 z optional float

message Polygon

The Polygon defines closed shapes formed of straight lines. Note: Polygon uses counterclockwise winding (this is the default OpenGL behavior)

Field Type Description
1 points repeated biopb.image.Point

message ProcessRequest

Field Type Description
1 image_data biopb.image.ImageData

Input image for processing

2 op_name string

Specify a operator name because the server may implement multiple operations.

3 field_mask google.protobuf.FieldMask

A field_mask to indicate the requested fields in the reply. Default is to return the full response. May be ignored by the server.

message ProcessResponse

Field Type Description
1 image_data biopb.image.ImageData

Output image after processing

2 annotation string

Additonal information the server may return as a text string

message ROI

A three dimensional 'Region of Interest'.

Field Type Description
oneof shape
1 point biopb.image.Point
2 rectangle biopb.image.Rectangle
3 ellipse biopb.image.Ellipse
4 polygon biopb.image.Polygon
5 mesh biopb.image.Mesh
6 mask biopb.image.Mask

message Rectangle

A simple rectangle object.

Field Type Description
1 top_left biopb.image.Point
2 bottom_right biopb.image.Point

message ScoredROI

Field Type Description
1 roi biopb.image.ROI
2 score float

Enums

enum Endianness

Data endianness. Note OME-XML use a bool. We change that to a Enum that default to BigEndian for slightly better efficiency and allowing potential extension to other states: e.g. system default.

Name Number Description
BIG 0
LITTLE 1