API Reference

Pilot-MapReduce

MapReduce

class pmr.MapReduce(pmrDesc, coordinationUrl)

MapReduce: Class for managing MapReduce Jobs

__init__(pmrDesc, coordinationUrl)

Initializes MapReduce with Pilot computes/Data description and coordination system

startPilot()

Start the pilot compute and data services

stopPilot()

Stops the pilot compute and data services

setNbrReduces(nbrReduces)

Set the number of reduces of the MapReduce Job @param nbrReduces: Takes number of Reduces as integer

setChunk(chunkDesc)

Registers the chunk task description @param chunkDesc: SAGA Job Description of chunk task

setMapper(mapDesc)

Registers the Map task description @param mapDesc: SAGA Job Description of Map task

setReducer(reduceDesc)

Registers the Reduce task description @param reduceDesc: SAGA Job Description of Reduce task

setOutputPath(path)

Sets the output path to store the final results of MapReduce job @param reduceDesc: SAGA Job Description of Reduce task

getPilotComputes()

Get pilot computes

getDetails()

Returns the execution time of MapReduce phases

@return: dictionary with execution timing details of MapReduce phases

chunkOnly(inputDu)

Executes the chunk Job only

@param inputDu: Takes input Data Units as Input argument @return: List of chunk task output Data Units

mapOnly(chunkDus)

Executes the Map Job only

@param mapDus: Takes map chunk/split Data Units as Input @return: List of Map task output Data Units

submitComputeUnit(desc)

Submits SAGA Job description to Pilot

@param desc: SAGA Job description

submitDataUnit(desc)

Submits SAGA Job description to Pilot

@param desc: SAGA Job description

reduceOnly(mapDus)

Executes the Reduce job only

@param mapDus: Takes map task Data Units as Input @return: Output Data Unit

runJob()

Executes the entire MapReduce workflow

Mapper

class pmr.Mapper(args)

mapper: Class for managing Map phase of MapReduce Job

__init__(args)

Initializes Map task with parameters passed by MapReduce framework as command line parameters.

partition(key)

Default partition function. This function could be overwritten by custom partition functions.

@param key: map task emitted key value @return: partition number into which the key,value pair has to be written to

emit(key, value)

Emit the key value based on the partition function

finalize()

Prepare the map output files sort the map output contents

Reducer

class pmr.Reducer(args)

reducer: Class for managing Reduce phase of MapReduce Job

__init__(args)

Initializes Reduce task with parameters passed by MapReduce framework as command line parameters.

emit(key, value)

Emit the key value pair to reduce file

finalize()

Close the reduce file

Hadoop

class pmr.Hadoop(pmrDesc, coordinationUrl)

Hadoop: Class for managing Hadoop Jobs

setUpCluster()

Setup Hadoop Cluster

submitJob(desc)

Submit Hadoop Job description

stopCluster()

Tear down Hadoop cluster

Yarn

class pmr.Yarn(pmrDesc, coordinationUrl)

Hadoop: Class for managing Yarn Jobs

setUpCluster()

Setup Yarn Cluster

submitJob(desc)

Submit Yarn Job description

stopCluster()

Tear down Yarn cluster

Spark

class pmr.Spark(pmrDesc, coordinationUrl)

Spark: Class for managing Spark cluster

setUpCluster()

Setup Spark Cluster

submitJob(desc)

Submit spark Job description

stopCluster()

Tear down spark cluster