Data Extraction

class ledsa.data_extraction.DataExtractor.DataExtractor(channels=0, load_config_file=True, build_experiment_infos=True, fit_leds=True)[source]

Bases: object

A class responsible for extracting data from experiment images.

Variables:

  • config (ConfigData) – Configuration data object.

  • channels (Tuple) – Channels to be processed.

  • fit_leds (bool :ivar fit_leds: Whether to fit LEDs or not.) – Whether to fit LEDs or not.

  • threshold (float, optional) – The threshold value used for LED detection.

  • line_indices (list[list[int]], optional) – 2D list with dimension (# of LED arrays) x (# of LEDs per array) or None.

find_search_areas() → None[source]

Identify all LEDs in the reference image and define the areas where LEDs will be searched in the experiment images.

load_led_array_indices() → None[source]

Load LED array indices from the ‘led_array_indices_{…}.csv’ files.

load_search_areas() → None[source]

Load LED search areas from the ‘led_search_areas.csv’ file. #TODO be consistent with search areas and ROIS

match_leds_to_led_arrays() → None[source]

Analyze which LEDs belong to which LED array and save this mapping.

plot_led_arrays(merge_led_arrays=False) → None[source]

Plots the arrangement of LEDs as identified in the LED arrays and saves the plot as a PDF file.

Parameters:

merge_led_arrays (bool) – A flag indicating whether LED arrays have been merged. Affects the naming of the output file.

plot_search_areas(reorder_leds=False) → None[source]

Plot the identified LED search areas with their ID labels.

Parameters:

reorder_leds (bool) – A flag indicating whether the LED IDs have been reordered. Affects the name of the output file.

process_image_data() → None[source]

Process all the image data to detect changes in light intensity in the search areas across the images. Removes ‘images_to_process.csv’ file afterward.

process_img_file(img_filename: str) → CorruptedImageError | None[source]

Process a single image file to extract relevant data. This is a workaround for pool.map.

An image that cannot be read is skipped instead of aborting the analysis of the remaining images. The error is returned rather than reported here, because the multicore analysis processes the images in worker processes, which would write to the report file concurrently.

Parameters:

img_filename (str) – The name of the image file to be processed.

Returns:

The error raised for a skipped image, or None if the image was processed.

Return type:

Optional[CorruptedImageError]

setup_restart() → None[source]

Setup a restart in case the data extraction process was interrupted earlier. Every image that misses the result file of one of the channels is analysed again, for all of them.

setup_step3() → None[source]

Setup the third step of the data extraction process by creating ‘image_infos_analysis.csv’ and ‘images_to_process.csv’ files.

write_search_areas(reorder_leds=False) → None[source]

Writes LED search areas to a CSV file.

Parameters:

reorder_leds (bool) – A flag indicating whether the LED IDs have been reordered. Affects the header of the output file.

class ledsa.data_extraction.LEDAnalysisData.LEDAnalysisData(led_id, led_array, fit_leds)[source]

Bases: object

Represents LED analysis data, including its physical properties and fit results.

Variables:

  • led_id (int) – The ID of the LED.

  • led_array (int) – The array index the LED belongs to.

  • fit_leds (bool) – Indicates whether to fit LEDs or not.

  • led_center_x (float) – X-coordinate of LED’s center.

  • led_center_y (float) – Y-coordinate of LED’s center.

  • mean_color_value (float) – Mean color value of the LED over the search area.

  • sum_color_value (float) – Integrated color value of the LED over the search area.

  • max_color_value (float) – Maximum color value observed for the LED.

  • farbg_sum_color_value (float or None) – Integrated color value minus the far-field pedestal of the search area, None if the pedestal is not estimated.

  • farbg_pedestal (float or None) – Far-field pedestal of the search area, None if the pedestal is not estimated.

  • fit_results (OptimizeResult) – Fit results after fitting.

  • fit_time (float) – Time taken for fitting.

get_fit_data_string() → str[source]

Provides the fitted LED data as a formatted string.

Returns:

Formatted string of fitted LED data.

Return type:

str

get_main_data_string() → str[source]

Provides the main LED data as a formatted string.

Returns:

Formatted string containing led_id, led_array and pixel values.

Return type:

str

Pedestal of the LED search areas, estimated in the far field of the LED strips.

The sum over a search area (sum_col_val) holds, besides the light of the LED, light that does not dim with the LED: the dark level of the sensor where it differs from the optical black the black level is taken from, ambient light, and light that smoke scatters into the line of sight. In dense smoke the LED dims while this pedestal stays, so the plain sum overestimates the transmission and the extinction coefficients saturate.

The pedestal is estimated for every image where no light of an LED is expected, in the far field of the strips:

  1. Two bands run parallel to the strip, one on each side, over +-R along the strip (R = search_area_radius) and from the inner to the outer far-field distance across it. Pixels closer than the inner distance to any LED of any strip are left out, so neighbouring strips do not enter the bands.

  2. The level of a band is the mean of its pixels of the color channel.

  3. The levels of each side are smoothed along the strip, with a running median and a running mean over the LEDs. The pedestal varies slowly along the strip, so this removes the noise of the estimate without changing it: the pedestal corrected sum is as noisy as the plain sum.

  4. The pedestal of an LED is the lower of the levels of its two sides. An object in one of the bands, another light source or the brighter side of a gradient can only raise a level, so the lower side is the safe one.

  5. N times this pedestal is subtracted from the sum of the search area, with N the number of pixels of the color channel in the search area.

Why the far field and not the neighbourhood of the LED: around an LED there is light of the LED itself, the wings of its image and, behind a window or in hot gases, a halo that can reach a hundred pixels and more. A pedestal measured there subtracts transmitted light and raises the extinction coefficients, the more the denser the smoke. The far field holds none of it. Stray light that peaks at the strip is brighter at the LED than in the far field, so the far-field level is a lower bound of the pedestal: the corrected sum lies between the plain sum and the light of the LED in the search area. The bound holds for the dark level and for scattered light. Ambient light on a background that is brighter beside the strip than behind the LED would raise the far-field level above the pedestal of the search area.

What the correction does not do: where the image of an LED widens beyond its search area, the search area loses light of the LED. That error has the opposite sign and is no pedestal; the correction leaves it as it is.

class ledsa.data_extraction.farfield_background.FarfieldGeometry(pixels: List[Tuple[Tuple[ndarray, ndarray], Tuple[ndarray, ndarray]] | None], strip_of_led: ndarray, position_on_strip: ndarray)[source]

Bases: NamedTuple

The far-field bands of the LEDs of an image and the place of every LED on its strip.

pixels: List[Tuple[Tuple[ndarray, ndarray], Tuple[ndarray, ndarray]] | None]

Alias for field number 0

position_on_strip: ndarray

Alias for field number 2

strip_of_led: ndarray

Alias for field number 1

ledsa.data_extraction.farfield_background.build_farfield_geometry(search_areas: ndarray, led_arrays: Sequence[Sequence[int]], img_shape: Tuple[int, int], search_area_radius: int, distances: Tuple[float, float]) → FarfieldGeometry[source]

Determine the pixels of the two far-field bands of each LED and the place of each LED on its strip.

With t the coordinate along the strip and s the coordinate across it, both relative to the center of the search area, a pixel belongs to the band of one side if -R <= t < R and inner <= s < outer, and to that of the other side if -outer <= s < -inner. For a vertical strip these are the rows r-R..r+R-1 and the columns c+inner..c+outer-1 and c-outer..c-inner-1. Pixels outside the image and pixels closer than the inner distance to the center of any search area are left out.

Parameters:

  • search_areas (np.ndarray) – Array with rows (LED id, pixel row, pixel column), the row index equals the LED id.

  • led_arrays (Sequence[Sequence[int]]) – LED ids of each LED strip, as found in step 2.

  • img_shape (tuple[int, int]) – Shape of the image.

  • search_area_radius (int) – Radius R of the search areas.

  • distances (tuple[float, float]) – Inner and outer distance of the bands across the strip in pixels.

Returns:

The band pixels and the place on its strip of every LED.

Return type:

FarfieldGeometry

ledsa.data_extraction.farfield_background.estimate_strip_directions(search_areas: ndarray, led_arrays: Sequence[Sequence[int]], num_neighbours: int = 4) → ndarray[source]

Estimate the local direction of the LED strip at each LED from its nearest neighbours in the same LED array.

The direction is the principal axis of the positions of the LED and its num_neighbours nearest neighbours of the same array. It is oriented such that the row component is positive (or, for horizontal strips, the column component), so the two sides of a strip are the same for all of its LEDs.

Parameters:

  • search_areas (np.ndarray) – Array with rows (LED id, pixel row, pixel column), the row index equals the LED id.

  • led_arrays (Sequence[Sequence[int]]) – LED ids of each LED array.

  • num_neighbours (int) – Number of neighbours used to estimate the direction.

Returns:

Array of shape (number of LEDs, 2) with the unit vector (row, column) along the strip, NaN for LEDs that are not part of an array with at least two LEDs.

Return type:

np.ndarray

ledsa.data_extraction.farfield_background.farfield_pedestals(band_levels: ndarray, strip_of_led: ndarray, position_on_strip: ndarray) → ndarray[source]

The pedestal per channel pixel of every LED: the levels of the two sides, smoothed along the strip, and of them the lower one.

Parameters:

  • band_levels (np.ndarray) – Level of the band on each of the two sides of every LED, shape (number of LEDs, 2), NaN for a band without a pixel of the channel.

  • strip_of_led (np.ndarray) – Index of the strip of every LED, -1 for an LED on no strip.

  • position_on_strip (np.ndarray) – Coordinate of every LED along its strip.

Returns:

Pedestal per channel pixel of every LED, NaN for an LED without a level on either side.

Return type:

np.ndarray

ledsa.data_extraction.farfield_background.get_farfield_geometry(search_areas: ndarray, led_arrays: Sequence[Sequence[int]], img_shape: Tuple[int, int], search_area_radius: int, distances: Tuple[float, float]) → FarfieldGeometry[source]

Cached build_farfield_geometry. The geometry only depends on the LED positions and is therefore computed once per process and reused for all images.

Returns:

See build_farfield_geometry.

ledsa.data_extraction.farfield_background.mean_of_channel_pixels(values: ndarray, channel_mask: ndarray, num_pixels: Sequence[int]) → ndarray[source]

Mean over the pixels of the color channel in every band, from the pixels of all bands.

Parameters:

  • values (np.ndarray) – Channel values of the pixels of all bands, the pixels of one band after those of the previous one.

  • channel_mask (np.ndarray) – Whether each of these pixels belongs to the channel.

  • num_pixels (Sequence[int]) – Number of pixels of every band.

Returns:

Mean per channel pixel of every band, NaN for a band without a pixel of the channel.

Return type:

np.ndarray

ledsa.data_extraction.farfield_background.parse_farfield_distances(value: str | None, search_area_radius: int) → Tuple[float, float] | None[source]

Read the far-field distances from their config value.

Parameters:

  • value (str or None) – Value of farfield_distances: the inner and the outer distance of the bands across the strip in pixels, or None.

  • search_area_radius (int) – Radius R of the search areas.

Returns:

The distances (inner, outer), None if the pedestal is not estimated.

Return type:

tuple[float, float] or None

Raises:

ValueError – If the value is neither None nor two distances R <= inner < outer.

ledsa.data_extraction.farfield_background.smooth_along_strip(levels: ndarray, median_size: int = 15, mean_size: int = 5) → ndarray[source]

Smooth the band levels of one side of a strip along the strip: a running median, which drops the outliers of single LEDs, followed by a running mean. An LED without a level takes the one of its neighbours. At the ends of the strip the windows are cut short.

Parameters:

  • levels (np.ndarray) – Level of every LED of the strip, in their order along the strip, NaN for an LED without one.

  • median_size (int) – Number of LEDs of the running median.

  • mean_size (int) – Number of LEDs of the running mean.

Returns:

The smoothed levels, NaN where no LED within reach has a level.

Return type:

np.ndarray