NYT::TIntermediateTablesHintSpec< TDerived > Struct Template Reference

Class to provide information on intermediate mapreduce stream protobuf types. More...

#include <operation.h>

Public Member Functions
template<class T>
TDerived &	HintMapOutput ()
	Specify intermediate map output type.
template<class T>
TDerived &	HintReduceCombinerInput ()
	Specify reduce combiner input.
template<class T>
TDerived &	HintReduceCombinerOutput ()
	Specify reduce combiner output.
template<class T>
TDerived &	HintReduceInput ()
	Specify reducer input.
template<class T>
TDerived &	AddMapOutput (const TRichYPath &path)
	Add output of map stage.
const TVector< TStructuredTablePath > &	GetStructuredMapOutputs () const
const TMaybe< TTableStructure > &	GetIntermediateMapOutputDescription () const
const TMaybe< TTableStructure > &	GetIntermediateReduceCombinerInputDescription () const
const TMaybe< TTableStructure > &	GetIntermediateReduceCombinerOutputDescription () const
const TMaybe< TTableStructure > &	GetIntermediateReducerInputDescription () const

Public Attributes
TVector< TRichYPath >	MapOutputs_

Detailed Description

template<class TDerived>
struct NYT::TIntermediateTablesHintSpec< TDerived >

Class to provide information on intermediate mapreduce stream protobuf types.

When using protobuf format it is important to know exact types of proto messages that are used in input/output.

Sometimes such messages cannot be derived from job class i.e. when job class uses NYT::TTableReader<::google::protobuf::Message> or NYT::TTableWriter<::google::protobuf::Message>.

When using such jobs user can provide exact message type using this class.

Note

Only input/output that relate to intermediate tables can be hinted. Input to map and output of reduce is derived from AddInput/AddOutput.

Member Function Documentation

AddMapOutput()

template<class TDerived>

template<class T>

TDerived & NYT::TIntermediateTablesHintSpec< TDerived >::AddMapOutput

(

const TRichYPath &

path

)

Add output of map stage.

Mapper output table #0 is always intermediate table that is going to be reduced later. Rows that mapper write to tables #1, #2, ... are saved in MapOutput tables.

---

The documentation for this struct was generated from the following file:

• yt/cpp/mapreduce/interface/operation.h