pyhealth.processors.TextProcessor#

Processor for text data.

class pyhealth.processors.TextProcessor[source]#

Bases: FeatureProcessor

Processor for text data.

process(value)[source]#

Process an individual field value.

Parameters:: value (Any) – Raw field value.
Return type:: str
Returns:: Processed value.

size()[source]#

dim()#

Number of dimensions (Tensor.dim()) for each output tensor, in the same order as the output tuple.

Return type:: tuple[int, ...]
Returns:: Tuple of integers corresponding to the number of dimensions of each output tensor.

fit(samples, field)#

Fit the processor to the samples.

Parameters:: samples (Iterable[Dict[str, Any]]) – List of sample dictionaries.
Return type:: None

is_token()#

Returns whether the output (in particular, the value tensor) of the processor represents discrete token indices (True) or continuous values (False). This is used to determine whether to apply token-based transformations (e.g. nn.Embedding) or value-based augmentations (e.g. nn.Linear).

Return type:: bool
Returns:: True if the output of the processor represents discrete token indices, False otherwise.

load(path)#

Optional: Load processor state from disk.

Parameters:: path (str) – File path to load processor state from.
Return type:: None

save(path)#

Optional: Save processor state to disk.

Parameters:: path (str) – File path to save processor state.
Return type:: None

schema()#

Returns the schema of the processed feature. For a processor that emits a single tensor, this should just return [“value”]. For a processor that emits a tuple of tensors, this should return a tuple of the same length as the tuple, with the semantic name of each tensor, such as [“time”, “value”], [“value”, “mask”], etc.

Typical semantic names include:

“value”: the main processed tensor output of the processor
“time”: the time tensor output of the processor (mostly for StageNet)
“mask”: the mask tensor output of the processor (if applicable)

Return type:: tuple[str, ...]
Returns:: Tuple of semantic names corresponding to the output of the processor.

spatial()#

Whether each dimension (axis) of the value tensor is spatial (i.e. corresponds to a spatial axis like time, height, width, etc.) or not. This is used to determine how to apply augmentations and other transformations that should only be applied to spatial dimensions.

E.g. for CNN or RNN features, this would help determine which dimensions to apply spatial augmentations to, and which dimensions to treat as channels or features.

Return type:: tuple[bool, ...]
Returns:: Tuple of booleans corresponding to whether each axis of the value tensor is spatial or not.