如何在 Reducer 中访问 Mapper Counter 值?

时间:2023-05-03
本文介绍了如何在 Reducer 中访问 Mapper Counter 值?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着跟版网的小编来一起学习吧!

问题描述

限时送ChatGPT账号..

我想访问 reducer 中的 myCounter.my 值:

I want to acces the myCounter.my value in reducer :

public static class Map extends Mapper<LongWritable, Text, ImmutableBytesWritable, ImmutableBytesWritable>
{
    public static enum myCounter{my};

    @Override
    public void map(LongWritable key, Text value, Context context) 
    {
        context.getCounter(myCounter.my).increment(1);
        context.write( new ImmutableBytesWritable ( ),new ImmutableBytesWritable() );
    }
}


public static class Reduce extends Reducer<ImmutableBytesWritable, ImmutableBytesWritable, Text, Text>
{
    @Override
    public void reduce(ImmutableBytesWritable key,Iterable<ImmutableBytesWritable> result,Context context)
    {

    }
}

从reducer访问映射器的计数器(对于旧API给定)如何使它适用于新的 API?

Accessing a mapper's counter from a reducer(for old API is given ) how to make it work for new API ?

或者

我想知道映射器输出的总数?有没有更好的办法?(我无法访问 Reducer 中的计数器:

I want to know the total number of mapper output ? Is there any better way ? (i am not able to access counter in Reducer:

组名->org.apache.hadoop.mapred.Task$Counter 计数器名->MAP_OUTPUT_RECORDS)

谢谢

推荐答案

您可以通过作业对象访问计数器,使其适用于新 API.

You to make it work for new API by accessing the counters via job object.

Configuration conf = context.getConfiguration();
Cluster cluster = new Cluster(conf);
Job currentJob = cluster.getJob(context.getJobID());
long val=currentJob.getCounters().findCounter(myCounter.my).getValue();

这篇关于如何在 Reducer 中访问 Mapper Counter 值?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持跟版网!

上一篇:hdfs中的文件路径 下一篇:Hadoop Map Reduce 用于 Google 网络图

相关文章