pwspy.utility.fileIO.processParallel

pwspy.utility.fileIO.processParallel(fileFrame, processorFunc, initializer=None, initArgs=None, procArgs=None, numProcesses=None)[source]

A convenience function to process the rows of a pandas DataFrame in parallel

Parameters

  • fileFrame (DataFrame) – A dataframe. Each row of the frame will be passed as the first argument to the processorFunc.

  • processorFunc (Callable[[], Any]) – A function that each row number and row of the fileFrame should be passed to as the first and second argument respectively. Additional arguments can be passed to processorFunc using the procArgs variable. The function should return the value which you want included in the return of processParrallel.

  • procArgs (Optional[Tuple]) – Optional arguments to pass to processorFunc

  • initializer (Optional[Callable]) – A function that is run once at the beginning of each spawned process. Can be used for copying shared memory.

  • initArgs (Optional[Tuple]) – A tuple of arguments to pass to the initializer function.

Return type

List containing the results of each execution of processorFunc.