operation.h File Reference

Go to the source code of this file.

Classes
struct	NYT::TUnspecifiedTableStructure
	Tag class marking that the row type for table is not specified. More...
struct	NYT::TProtobufTableStructure
	Tag class marking that table rows have protobuf type. More...
struct	NYT::TStructuredTablePath
	Table path marked with NYT::TTableStructure tag. More...
struct	NYT::TVoidStructuredRowStream
	Tag class marking that row stream is empty. More...
struct	NYT::TTNodeStructuredRowStream
	Tag class marking that row stream consists of NYT::TNode. More...
struct	NYT::TTYaMRRowStructuredRowStream
	Tag class marking that row stream consists of NYT::TYaMRRow. More...
struct	NYT::TProtobufStructuredRowStream
	Tag class marking that row stream consists of protobuf rows of given type. More...
struct	NYT::TJobBinaryDefault
	Tag class marking that current binary should be used in operation. More...
struct	NYT::TJobBinaryLocalPath
	Tag class marking that binary from specified local path should be used in operation. More...
struct	NYT::TJobBinaryCypressPath
	Tag class marking that binary from specified Cypress path should be used in operation. More...
class	NYT::TAutoMergeSpec
	Options for auto merge operation stage. More...
class	NYT::TWithAutoMergeSpec< TDerived >
	Base for operations with auto merge options. More...
class	NYT::TSchedulerResources
	Resources controlled by scheduler and used by running operations. More...
class	NYT::TUserJobInputFormatHintsBase< TDerived >
	Base for input format hints of a user job. More...
class	NYT::TUserJobOutputFormatHintsBase< TDerived >
	Base for output format hints of a user job. More...
class	NYT::TUserJobFormatHintsBase< TDerived >
	Base for format hints of a user job. More...
class	NYT::TUserJobFormatHints
	User job format hints. More...
class	NYT::TRawOperationIoTableSpec< TDerived >
	Spec of input and output tables of a raw operation. More...
struct	NYT::TSimpleRawOperationIoSpec< TDerived >
	Base spec for IO in "simple" raw operations (Map, Reduce etc.). More...
class	NYT::TRawMapReduceOperationIoSpec< TDerived >
	Spec for IO in MapReduce operation. More...
class	NYT::TOperationInputSpecBase
	Base spec of operations with input tables. More...
class	NYT::TOperationOutputSpecBase
	Base spec of operations with output tables. More...
struct	NYT::TOperationIOSpecBase
	Base spec for operations with inputs and outputs. More...
struct	NYT::TOperationIOSpec< TDerived >
	Base spec for operations with inputs and outputs. More...
struct	NYT::TOperationSpecBase< TDerived >
	Base spec for all operations. More...
struct	NYT::TUserOperationSpecBase< TDerived >
	Base spec for all operations with user jobs. More...
struct	NYT::TIntermediateTablesHintSpec< TDerived >
	Class to provide information on intermediate mapreduce stream protobuf types. More...
struct	NYT::TAddLocalFileOptions
struct	NYT::TJobProfilerSpec
	Specifies a job profiler. More...
struct	NYT::TDiskRequest
	Specification of a disk that will be available in job. More...
struct	NYT::TUserJobSpec
	Spec of user job. More...
struct	NYT::TMapOperationSpecBase< TDerived >
	Spec of Map operation. More...
struct	NYT::TMapOperationSpec
	Spec of Map operation. More...
struct	NYT::TRawMapOperationSpec
	Spec of raw Map operation. More...
struct	NYT::TReduceOperationSpecBase< TDerived >
	Spec of Reduce operation. More...
struct	NYT::TReduceOperationSpec
	Spec of Reduce operation. More...
struct	NYT::TRawReduceOperationSpec
	Spec of raw Reduce operation. More...
struct	NYT::TJoinReduceOperationSpecBase< TDerived >
	Spec of JoinReduce operation. More...
struct	NYT::TJoinReduceOperationSpec
	Spec of JoinReduce operation. More...
struct	NYT::TRawJoinReduceOperationSpec
	Spec of raw JoinReduce operation. More...
struct	NYT::TMapReduceOperationSpecBase< TDerived >
	Spec of MapReduce operation. More...
struct	NYT::TMapReduceOperationSpec
	Spec of MapReduce operation. More...
struct	NYT::TRawMapReduceOperationSpec
	Spec of raw MapReduce operation. More...
struct	NYT::TSortOperationSpec
	Spec of Sort operation. More...
struct	NYT::TMergeOperationSpec
	Spec of Merge operation. More...
struct	NYT::TEraseOperationSpec
	Spec of Erase operation. More...
struct	NYT::TRemoteCopyOperationSpec
	Spec of RemoteCopy operation. More...
struct	NYT::TVanillaTask
	Task of Vanilla operation. More...
struct	NYT::TVanillaOperationSpec
	Spec of Vanilla operation. More...
struct	NYT::TOperationOptions
	Options for NYT::IOperationClient::Map and other operation start commands. More...
class	NYT::TRawJobContext
	Context passed to NYT::IRawJob::Do. More...
class	NYT::ISerializableForJob
	Interface for classes that can be Saved/Loaded (to be used with Y_SAVELOAD_JOB). More...
class	NYT::IOperationPreparationContext
	Provider of information about operation inputs/outputs during NYT::IJob::PrepareOperation. More...
class	NYT::TJobOperationPreparer
	Fluent builder class for NYT::IJob::PrepareOperation. More...
class	NYT::TJobOperationPreparer::TInputGroup
	Group of input tables that allows to specify properties on all of them at once. More...
class	NYT::TJobOperationPreparer::TOutputGroup
	Group of output tables that allows to specify properties on all of them at once. More...
class	NYT::IJob
	Interface for all user jobs. More...
class	NYT::IStructuredJob
	Interface for jobs with typed inputs and outputs. More...
class	NYT::IMapperBase
	Base interface for structured (typed) map jobs. More...
class	NYT::IMapper< TR, TW >
	Base interface for structured (typed) map jobs with given reader and writer. More...
class	NYT::IReducerBase
	Base interface for structured (typed) reduce jobs. More...
class	NYT::IReducer< TR, TW >
	Base interface for structured (typed) reduce jobs with given reader and writer. More...
class	NYT::IRawJob
	This method is called before feeding input rows to reducer (before Do method). virtual void Start(TWriter* writer) { Y_UNUSED(writer); }. More...
class	NYT::ICommandJob
	Interface of jobs that run the given bash command. More...
class	NYT::TCommandRawJob
	Raw job executing given bash command. More...
class	NYT::IVanillaJobBase
	Base interface for vanilla jobs. More...
class	NYT::IVanillaJob< void >
	Interface of vanilla job without outputs. More...
class	NYT::TCommandVanillaJob
	Vanilla job executing given bash command. More...
class	NYT::IVanillaJob< TW >
	Interface for vanilla jobs with output tables. More...
struct	NYT::TOperationAttributeFilter
	Class describing which attributes to request in NYT::IClient::GetOperation or NYT::IClient::ListOperations. More...
struct	NYT::TGetOperationOptions
	Options for NYT::IClient::GetOperation call. More...
struct	NYT::TOperationProgress
	Operation progress. More...
struct	NYT::TOperationBriefProgress
	Brief operation progress (numbers of jobs in these states). More...
struct	NYT::TOperationResult
	Operation result. More...
struct	NYT::TOperationEvent
	Operation event (change of state). More...
struct	NYT::TOperationAttributes
	Operation info. More...
struct	NYT::TListOperationsOptions
	Options of NYT::IClient::ListOperations command. More...
struct	NYT::TListOperationsResult
	Response for NYT::IClient::ListOperations command. More...
class	NYT::TTaskName
	Task name (can either well-known or just a string). More...
struct	NYT::TJobAttributeFilter
	A class that specifies which attributes to request when using NYT::IClient::GetJob or NYT::IClient::ListJobs. More...
struct	NYT::TListJobsOptions
	Options for NYT::IClient::ListJobs. More...
struct	NYT::TCoreInfo
	Description of a core dump that happened in the job. More...
struct	NYT::TJobAttributes
	Job attributes. More...
struct	NYT::TListJobsResult
	Response for NYT::IOperation::ListJobs. More...
struct	NYT::TGetJobOptions
	Options for NYT::IClient::GetJob. More...
struct	NYT::TGetJobInputOptions
	Options for NYT::IClient::GetJobInput. More...
struct	NYT::TGetJobFailContextOptions
	Options for NYT::IClient::GetJobFailContext. More...
struct	NYT::TGetJobStderrOptions
	Options for NYT::IClient::GetJobStderr. More...
struct	NYT::TGetFailedJobInfoOptions
	Options for NYT::IOperation::GetFailedJobInfo. More...
struct	NYT::TGetJobTraceOptions
	Options for NYT::IClient::GetJobTrace. More...
struct	NYT::TJobTraceEvent
	Response for NYT::IOperation::GetJobTrace. More...
struct	NYT::IOperation
	Interface representing an operation. More...
struct	NYT::IOperationClient
	Interface of client capable of managing operations. More...

Namespaces
namespace	NYT
	Main namespace of YT client.

Macros
#define	Y_SAVELOAD_JOB(...)
	Declare what fields of currently declared job class to save and restore on cluster node.
#define	Y_SAVELOAD_JOB_DERIVED(Base, ...)
	Same as the macro above, but also calls Base class's SaveLoad methods.
#define	OPERATION_INL_H_

Typedefs
using	NYT::TTableStructure
	Tag class to specify table row type.
using	NYT::TStructuredRowStreamDescription
	Tag class to specify type of rows in an operation row stream.

Enumerations
enum class	NYT::EAutoMergeMode { NYT::Disabled , NYT::Relaxed , NYT::Economy , NYT::Manual }
	Auto merge mode. More...
enum class	NYT::EProfilingBinary { NYT::JobProxy , NYT::UserJob }
	Binary to run job profiler on. More...
enum class	NYT::EProfilerType { NYT::Cpu , NYT::Memory , NYT::PeakMemory }
	Type of job profiler. More...
enum class	NYT::ESchemaInferenceMode : int { FromInput , FromOutput , Auto }
	Schema inference mode. More...
enum	NYT::EMergeMode : int { MM_UNORDERED , MM_ORDERED , MM_SORTED }
	Merge mode.
enum class	NYT::EOperationAttribute : int {   Id , Type , State , AuthenticatedUser ,   StartTime , FinishTime , BriefProgress , BriefSpec ,   Suspended , Result , Progress , Events ,   Spec , FullSpec , UnrecognizedSpec , Alerts }
	Attributes to request for an operation.
enum class	NYT::EOperationBriefState : int { InProgress , Completed , Aborted , NYT::Failed }
	"Coarse-grained" state of an operation. More...
enum class	NYT::EOperationState : int {   None , Starting , Orphaned , WaitingForAgent ,   Initializing , Preparing , Materializing , ReviveInitializing ,   Reviving , RevivingJobs , Pending , Running ,   Completing , Completed , Aborting , Aborted ,   Failing , Failed }
	Operation state.
enum class	NYT::EOperationType : int {   Map , Merge , Erase , Sort ,   Reduce , MapReduce , RemoteCopy , JoinReduce ,   Vanilla }
	Operation type.
enum class	NYT::ECursorDirection { Past , Future }
	Direction of cursor for paging, see NYT::TListOperationsOptions::CursorDirection.
enum class	NYT::EListJobsDataSource : int { Runtime , Archive , Auto , Manual }
	Data source for NYT::IClient::ListJobs command.
enum class	NYT::EJobType : int {   Map , PartitionMap , SortedMerge , OrderedMerge ,   UnorderedMerge , Partition , SimpleSort , FinalSort ,   SortedReduce , PartitionReduce , ReduceCombiner , RemoteCopy ,   IntermediateSort , OrderedMap , JoinReduce , Vanilla ,   SchedulerUnknown , ReplicateChunk , RemoveChunk , RepairChunk ,   SealChunk , ShallowMerge , MergeChunks , AutotomizeChunk ,   ReincarnateChunk }
	Job type.
enum class	NYT::ETaskName : int {   Map , PartitionMap0 , SortedMerge , OrderedMerge ,   UnorderedMerge , Partition0 , Partition1 , Partition2 ,   SimpleSort , FinalSort , SortedReduce , PartitionReduce ,   ReduceCombiner , RemoteCopy , IntermediateSort , OrderedMap ,   JoinReduce }
	Well-known task names.
enum class	NYT::EJobState : int {   None , Waiting , Running , Aborting ,   Completed , Failed , Aborted , Lost }
	Job state.
enum class	NYT::EJobSortField : int {   Type , State , StartTime , FinishTime ,   Address , Duration , Progress , Id ,   TaskName }
	Job sort field. More...
enum class	NYT::EJobSortDirection : int { Ascending , Descending }
	Job sort direction. More...
enum class	NYT::EJobAttribute : int {   Id , Type , State , Address ,   TaskName , StartTime , FinishTime , Progress ,   StderrSize , Error , Result , BriefStatistics ,   InputPaths , CoreInfos }
	Attributes to request for a job.

Functions
bool	NYT::operator== (const TUnspecifiedTableStructure &, const TUnspecifiedTableStructure &)
bool	NYT::operator== (const TProtobufTableStructure &lhs, const TProtobufTableStructure &rhs)
template<typename TRow>
TStructuredTablePath	NYT::Structured (TRichYPath richYPath)
	Create marked table path from row type.
template<typename TRow>
TTableStructure	NYT::StructuredTableDescription ()
	Create tag class from row type.
const TNode &	NYT::GetJobSecureVault ()
	Get operation secure vault (specified in NYT::TOperationOptions::SecureVault) inside a job.
TRawTableReaderPtr	NYT::CreateRawJobReader (int fd=0)
	Create default raw job reader.
THolder< IProxyOutput >	NYT::CreateRawJobWriter (size_t outputTableCount)
	Create default raw job writer.

Detailed Description

Header containing interface to run operations in YT and retrieve information about them.

See also

the doc.

Macro Definition Documentation

Y_SAVELOAD_JOB

#define Y_SAVELOAD_JOB

(

...

)

Value:

    virtual void Save(IOutputStream& stream) const override { Save(&stream); } \
    virtual void Load(IInputStream& stream) override { Load(&stream); } \
    Y_PASS_VA_ARGS(Y_SAVELOAD_DEFINE(__VA_ARGS__))

Declare what fields of currently declared job class to save and restore on cluster node.

Y_SAVELOAD_JOB_DERIVED

#define Y_SAVELOAD_JOB_DERIVED	(		Base,
			... )

Value:

    virtual void Save(IOutputStream& stream) const override { \
        Base::Save(stream); \
        Save(&stream); \
    } \
    virtual void Load(IInputStream& stream) override { \
        Base::Load(stream); \
        Load(&stream); \
    } \
    Y_PASS_VA_ARGS(Y_SAVELOAD_DEFINE(__VA_ARGS__))

Same as the macro above, but also calls Base class's SaveLoad methods.