pwspy.utility.fileIO.loadAndProcess

pwspy.utility.fileIO.loadAndProcess(fileFrame, processorFunc=None, parallel=None, procArgs=None, initializer=None, initArgs=None)[source]

DEPRECATED! This over-complicated function should be replaced with usage of processParallel. A convenient function to load a series of Data Cubes from a list or dictionary of file paths.

Parameters

  • fileFrame – A dataframe containing a column of PwsCube file paths titled ‘cube’ and other columns to act as specifiers for each cube. If no specifiers are used this can just be a list of file paths.

  • processorFunc – A function that each loaded cell should be passed to. The first argument of processorFunc should be the loaded PwsCube. Additional arguments can be passed to processorFunc using the procArgs variable.

  • parallel – default is False. If True then the loading and processing will be performed in parallel on multiple cores, otherwise it will be done using multithreading on a single core. Setting this to true can result if big speedups if the time to run processorFunc is greater than the time to load an PwsCube from file.

  • procArgs – Optional arguments to pass to processorFunc

  • initializer – A function that is run once at the beginning of each spawned process. Can be used for copying shared memory.

  • initArgs – A tuple of arguments to pass to the initializer function.

Returns

  • An object of the same form as fileFrame except the PwsCube file paths have been replaced by PwsCube Object.

  • If using processorFunc the return values from processorFunc will be returned rather than PwsCube Objects.