NYT::IJob Class Reference

Interface for all user jobs. More...

#include <operation.h>

Inheritance diagram for NYT::IJob:

Public Types
enum	EType {   Mapper , Reducer , ReducerAggregator , RawJob ,   VanillaJob }
	Type of job.

Public Member Functions
virtual void	Save (IOutputStream &stream) const
	Save job state to stream to be restored on cluster nodes.
virtual void	Load (IInputStream &stream)
	Restore job state from a stream.
const TNode &	SecureVault () const
	Get operation secure vault (specified in NYT::TOperationOptions::SecureVault) inside a job.
i64	GetOutputTableCount () const
	Get number of output tables.
virtual void	PrepareOperation (const IOperationPreparationContext &context, TJobOperationPreparer &preparer) const
	Method allowing user to control some properties of input and output tables and formats.

Detailed Description

Interface for all user jobs.

Member Function Documentation

PrepareOperation()

virtual void NYT::IJob::PrepareOperation ( const IOperationPreparationContext & context, TJobOperationPreparer & preparer ) const

virtual

Method allowing user to control some properties of input and output tables and formats.

User can override this method in their job class to:

• specify output table schemas. The most natural way is usually through NYT::TJobOperationPreparer::OutputDescription (especially for protobuf), but you can use NYT::TJobOperationPreparer::OutputSchema directly
• specify output row type (NYT::TJobOperationPreparer::OutputDescription)
• specify input row type (NYT::TJobOperationPreparer::InputDescription)
• specify input column filter and renaming (NYT::TJobOperationPreparer::InputColumnFilter and NYT::TJobOperationPreparer::InputColumnRenaming)
• specify format hints (NYT::TJobOperationPreparer::InputFormatHints, NYT::TJobOperationPreparer::OutputFormatHints and NYT::TJobOperationPreparer::FormatHints)
• maybe something more, cf. the methods of NYT::TJobOperationPreparer.

If one has several similar tables, groups can be used. Groups are delimited by NYT::TJobOperationPreparer::BeginInputGroup / NYT::TJobOperationPreparer::TInputGroup::EndInputGroup and NYT::TJobOperationPreparer::BeginOutputGroup / NYT::TJobOperationPreparer::TOutputGroup::EndOutputGroup. Example:

preparer
  .BeginInputGroup({1,2,4,8})
    .ColumnRenaming({{"a", "b"}, {"c", "d"}})
    .ColumnFilter({"a", "c"})
  .EndInputGroup();

Note

All the output table schemas must be set (possibly as empty nonstrict using NYT::TJobOperationPreparer::NoOutputSchema or NYT::TJobOperationPreparer::TOutputGroup::NoSchema). By default all the output table schemas are marked as empty nonstrict.

---

The documentation for this class was generated from the following file:

• yt/cpp/mapreduce/interface/operation.h