Skip to content

[BUG] - Info messages appearing as warnings in Jupyter notebooksΒ #109

Open

Description

@JHancox

Describe the bug
When running cucim - accelerated dataloading in Jupyter, a lot of pink warning style output can be generated that looks like an error.

Steps/Code to reproduce bug
Run the following code in a notebook with a suitably large image:

from cucim import CuImage
from dask.distributed import as_completed
from dask.distributed import Client, LocalCluster
import os
import numpy as np

cluster = LocalCluster(dashboard_address= 8789, processes=True)
client = Client(cluster)

# iterate over a set of regions from which to threshold
def process_chunk(params):
    start_loc_list = params[0]
    inp_file = params[1]
    patch_size = params[2]
    slide = CuImage(inp_file)
    res = []
    for start_loc in start_loc_list:
        region = np.array(slide.read_region(start_loc, [patch_size, patch_size], 0))
        if region.flatten().var() > 100:
            res.append(start_loc)
        
    return res

# As the results are processed, put them into a list
def compile_results(futures):
    patches = []

    for future in as_completed(futures):
        res1 = future.result()
        if res1:
            for patch in res1:
                patches.append(patch)
                
    return patches

input_file = "patient_100_node_0.tif"
wsi = CuImage(input_file)
sizes=wsi.metadata["cucim"]["resolutions"]
w = sizes["level_dimensions"][0][0]
h = sizes["level_dimensions"][0][1]
patch_size = 256
num_processes = os.cpu_count()

# compute the coordinates of the image patches
start_loc_data = [(sx, sy)
    for sy in range(0, h, patch_size)
        for sx in range(0, w, patch_size)]

# calculate the number of patches per process/thread
chunk_size = len(start_loc_data) // num_processes

# create list of patches to process
start_loc_list_iter = [(start_loc_data[i:i+chunk_size],input_file,patch_size)  for i in range(0, len(start_loc_data), chunk_size)]

# Threshold each patch asynchronously and return a future
future_result1 = list(client.map(process_chunk, start_loc_list_iter))
patches = compile_results(future_result1)`

Expected behavior
Ideally no such output should be produced unless the user specifies that they want info/warning output to be shown.

Environment details (please complete the following information):
Jupyter notebook running NGC PyTorch container 21.07
cucim v21.8.2

Additional context
Here is some example output when loading an image with 10 processes:

[Plugin: cucim.kit.cuslide] Loading...
[Plugin: cucim.kit.cuslide] Loading the dynamic library from: /opt/conda/lib/python3.8/site-packages/cucim/clara/cucim.kit.cuslide@21.08.02.so
[Plugin: cucim.kit.cuslide] loaded successfully. Version: 0
Initializing plugin: cucim.kit.cuslide (interfaces: [cucim::io::IImageFormat v0.1]) (impl: cucim.kit.cuslide)
[Plugin: cucim.kit.cuslide] Loading...
[Plugin: cucim.kit.cuslide] Loading the dynamic library from: /opt/conda/lib/python3.8/site-packages/cucim/clara/cucim.kit.cuslide@21.08.02.so
[Plugin: cucim.kit.cuslide] loaded successfully. Version: 0
Initializing plugin: cucim.kit.cuslide (interfaces: [cucim::io::IImageFormat v0.1]) (impl: cucim.kit.cuslide)
[Plugin: cucim.kit.cuslide] Loading...
[Plugin: cucim.kit.cuslide] Loading the dynamic library from: /opt/conda/lib/python3.8/site-packages/cucim/clara/cucim.kit.cuslide@21.08.02.so
[Plugin: cucim.kit.cuslide] Loading...
[Plugin: cucim.kit.cuslide] Loading the dynamic library from: /opt/conda/lib/python3.8/site-packages/cucim/clara/cucim.kit.cuslide@21.08.02.so
[Plugin: cucim.kit.cuslide] loaded successfully. Version: 0
Initializing plugin: cucim.kit.cuslide (interfaces: [cucim::io::IImageFormat v0.1]) (impl: cucim.kit.cuslide)
[Plugin: cucim.kit.cuslide] loaded successfully. Version: 0
Initializing plugin: cucim.kit.cuslide (interfaces: [cucim::io::IImageFormat v0.1]) (impl: cucim.kit.cuslide)
[Plugin: cucim.kit.cuslide] Loading...
[Plugin: cucim.kit.cuslide] Loading the dynamic library from: /opt/conda/lib/python3.8/site-packages/cucim/clara/cucim.kit.cuslide@21.08.02.so
[Plugin: cucim.kit.cuslide] Loading...
[Plugin: cucim.kit.cuslide] Loading the dynamic library from: /opt/conda/lib/python3.8/site-packages/cucim/clara/cucim.kit.cuslide@21.08.02.so
[Plugin: cucim.kit.cuslide] Loading...
[Plugin: cucim.kit.cuslide] Loading the dynamic library from: /opt/conda/lib/python3.8/site-packages/cucim/clara/cucim.kit.cuslide@21.08.02.so
[Plugin: cucim.kit.cuslide] loaded successfully. Version: 0
Initializing plugin: cucim.kit.cuslide (interfaces: [cucim::io::IImageFormat v0.1]) (impl: cucim.kit.cuslide)
[Plugin: cucim.kit.cuslide] loaded successfully. Version: 0
Initializing plugin: cucim.kit.cuslide (interfaces: [cucim::io::IImageFormat v0.1]) (impl: cucim.kit.cuslide)
[Plugin: cucim.kit.cuslide] loaded successfully. Version: 0
Initializing plugin: cucim.kit.cuslide (interfaces: [cucim::io::IImageFormat v0.1]) (impl: cucim.kit.cuslide)
[Plugin: cucim.kit.cuslide] Loading...
[Plugin: cucim.kit.cuslide] Loading the dynamic library from: /opt/conda/lib/python3.8/site-packages/cucim/clara/cucim.kit.cuslide@21.08.02.so
[Plugin: cucim.kit.cuslide] loaded successfully. Version: 0
Initializing plugin: cucim.kit.cuslide (interfaces: [cucim::io::IImageFormat v0.1]) (impl: cucim.kit.cuslide)
[Plugin: cucim.kit.cuslide] Loading...
[Plugin: cucim.kit.cuslide] Loading the dynamic library from: /opt/conda/lib/python3.8/site-packages/cucim/clara/cucim.kit.cuslide@21.08.02.so
[Plugin: cucim.kit.cuslide] loaded successfully. Version: 0
Initializing plugin: cucim.kit.cuslide (interfaces: [cucim::io::IImageFormat v0.1]) (impl: cucim.kit.cuslide)

Activity

  1. self-assigned this
    on Oct 5, 2021
  2. added this to the v21.12.00 milestone on Nov 10, 2021
  3. modified the milestones: v21.12.00, v22.02.00 on Jan 11, 2022
  4. modified the milestones: v22.02.00, v22.04.01 on Apr 5, 2022
  5. modified the milestones: v22.04.01, v22.06.00 on May 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

  • Status
    No status

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions