`

jafka学习之LogManager

    博客分类:
  • mq
阅读更多
       今天终于要讲到LogManager了,在讲LogManager之前,我们还是先来看下几个基本概念和jafka的存储结构。
      

 
     下面是一个网友画的图:
       

       从这个里面可以看到,消息队列跟路径下,每个topic的每个分区都是一个目录,里面是一个一个的文件,都以jafka结尾
      那下来,我们看下类的实现。
      
public class LogManager implements PartitionChooser, Closeable {
    // 服务器配置
    final ServerConfig config;
    // 一个定时任务执行器
    private final Scheduler scheduler ;
    // log清理间隔
    final long logCleanupIntervalMs ;
    // 默认log清理间隔
    final long logCleanupDefaultAgeMs ;
    // 实现需要恢复,这个后面再讲
    final boolean needRecovery ;

    private final Logger logger = LoggerFactory.getLogger(LogManager.class);

    ///////////////////////////////////////////////////////////////////////
    final int numPartitions ;
    // 消息队列的跟路径
    final File logDir;
    // log的刷新频率
    final int flushInterval ;
    // log创建的写锁
    private final Object logCreationLock = new Object();

    final Random random = new Random();
    // 一个启动器的闭锁
    final CountDownLatch startupLatch;

    // 一个log的池,key为topic,value是partition和Log的对应
    private final Pool<String, Pool<Integer, Log>> logs = new Pool<String, Pool<Integer, Log>>();

    // log flush的后台任务执行器
    private final Scheduler logFlusherScheduler = new Scheduler(1, "jafka-logflusher-" , false);

    // topic的注册的task
    private final LinkedBlockingQueue<TopicTask> topicRegisterTasks = new LinkedBlockingQueue<TopicTask>();

    // topic注册task是否停止状态
    private volatile boolean stopTopicRegisterTasks = false;

    // log刷新频率的一个map,key为topic,value为刷新频率
    final Map<String, Integer> logFlushIntervalMap;

    // log保持的一个map,key为topic,value为时长
    final Map<String, Long> logRetentionMSMap;

    // log保留的size,这个后面详细解释
    final int logRetentionSize ;

    /////////////////////////////////////////////////////////////////////////
    // 记录broker、topic在zk的注册情况
    private ServerRegister serverRegister;

    // topic和分区个数的一个map
    private final Map<String, Integer> topicPartitionsMap;

    // log的回滚策略
    private RollingStrategy rollingStategy;

    // 最大的消息的size
    private final int maxMessageSize ;
   
    LogManager这类,主要做两件事情,一个是初始化的时候从目录中恢复log,一个是对运行时的相关命令做出响应。
  我们先来看下恢复log这个:
   
 public void load() throws IOException {
     // 滚动策略为fixedsize 的滚动策略
        if (this .rollingStategy == null) {
            this.rollingStategy = new FixedSizeRollingStrategy(config .getLogFileSize());
        }
        // 如果消息队列的根路径不存在,则创建
        if (!logDir .exists()) {
            logger.info( "No log directory found, creating '" + logDir.getAbsolutePath() + "'");
            logDir.mkdirs();
        }
        // 判断是否是目录并且可写
        if (!logDir .isDirectory() || !logDir.canRead()) {
            throw new IllegalArgumentException(logDir .getAbsolutePath() + " is not a readable log directory.");
        }
        // 获取这个里面的所有的子文件
        File[] subDirs = logDir.listFiles();
        if (subDirs != null) {
            for (File dir : subDirs) {
               // 跳过文件,因为在这个目录下,全部是topic-partition的文件夹
                if (!dir.isDirectory()) {
                    logger.warn( "Skipping unexplainable file '" + dir.getAbsolutePath() + "'--should it be there?");
                } else {
                    logger.info( "Loading log from " + dir.getAbsolutePath());
                   
                    // 获取到目录的名称
                    final String topicNameAndPartition = dir.getName();
                    if (-1 == topicNameAndPartition.indexOf('-' )) {
                        throw new IllegalArgumentException("error topic directory: " + dir.getAbsolutePath());
                    }
                   
                    // 将目录名称按照-进行分割,得到一个 kv,然后key就是topic,value是partition
                    final KV<String, Integer> topicPartion = Utils.getTopicPartition(topicNameAndPartition);
                    final String topic = topicPartion.k ;
                    final int partition = topicPartion.v;
                   
                    // 针对这个分区创建一个log对象
                    Log log = new Log(dir, partition, this.rollingStategy , flushInterval , needRecovery , maxMessageSize);

                    // 将新创建的log加入到map中
                    logs.putIfNotExists(topic, new Pool<Integer, Log>());
                    Pool<Integer, Log> parts = logs.get(topic);

                    parts.put(partition, log);
                   
                    // 如果时间的partition比配置中的大,则进行升级
                    int configPartition = getPartition(topic);
                    if (configPartition <= partition) {
                        topicPartitionsMap.put(topic, partition + 1);
                    }
                }
            }
        }

        /* Schedule the cleanup task to delete old logs */
        if (this .scheduler != null) {
            logger.debug( "starting log cleaner every " + logCleanupIntervalMs + " ms" );
            this.scheduler .scheduleWithRate(new Runnable() {

                public void run() {
                    try {
                        cleanupLogs();
                    } catch (IOException e) {
                        logger.error( "cleanup log failed.", e);
                    }
                }

            }, 60 * 1000, logCleanupIntervalMs);
        }
        // 将topic的注册任务注册到 zk里面
        if (config .getEnableZookeeper()) {
            this.serverRegister = new ServerRegister(config, this );
            serverRegister.startup();
            TopicRegisterTask task = new TopicRegisterTask();
            task.setName( "jafka.topicregister");
            task.setDaemon( true);
            task.start();
        }
    }
    其他运行时的我就不做详细解释了,我都加了注释了,请大家对照着注释进行看下。
    
public class LogManager implements PartitionChooser, Closeable {
    // 服务器配置
    final ServerConfig config;
    // 一个定时任务执行器
    private final Scheduler scheduler ;
    // log清理间隔
    final long logCleanupIntervalMs ;
    // 默认log清理间隔
    final long logCleanupDefaultAgeMs ;
    // 实现需要恢复,这个后面再讲
    final boolean needRecovery ;

    private final Logger logger = LoggerFactory.getLogger(LogManager.class);

    ///////////////////////////////////////////////////////////////////////
    final int numPartitions ;
    // 消息队列的跟路径
    final File logDir;
    // log的刷新频率
    final int flushInterval ;
    // log创建的写锁
    private final Object logCreationLock = new Object();

    final Random random = new Random();
    // 一个启动器的闭锁
    final CountDownLatch startupLatch;

    // 一个log的池,key为topic,value是partition和Log的对应
    private final Pool<String, Pool<Integer, Log>> logs = new Pool<String, Pool<Integer, Log>>();

    // log flush的后台任务执行器
    private final Scheduler logFlusherScheduler = new Scheduler(1, "jafka-logflusher-" , false);

    // topic的注册的task
    private final LinkedBlockingQueue<TopicTask> topicRegisterTasks = new LinkedBlockingQueue<TopicTask>();

    // topic注册task是否停止状态
    private volatile boolean stopTopicRegisterTasks = false;

    // log刷新频率的一个map,key为topic,value为刷新频率
    final Map<String, Integer> logFlushIntervalMap;

    // log保持的一个map,key为topic,value为时长
    final Map<String, Long> logRetentionMSMap;

    // log保留的size,这个后面详细解释
    final int logRetentionSize ;

    /////////////////////////////////////////////////////////////////////////
    // 记录broker、topic在zk的注册情况
    private ServerRegister serverRegister;

    // topic和分区个数的一个map
    private final Map<String, Integer> topicPartitionsMap;

    // log的回滚策略
    private RollingStrategy rollingStategy;

    // 最大的消息的size
    private final int maxMessageSize ;

    public LogManager(ServerConfig config, //
                      Scheduler scheduler, //
                      long logCleanupIntervalMs, //
                      long logCleanupDefaultAgeMs, //
                      boolean needRecovery) {
        super();
        this.config = config;
        this.maxMessageSize = config.getMaxMessageSize();
        this.scheduler = scheduler;
        //        this.time = time;
        this.logCleanupIntervalMs = logCleanupIntervalMs;
        this.logCleanupDefaultAgeMs = logCleanupDefaultAgeMs;
        this.needRecovery = needRecovery;
        //
        this.logDir = Utils.getCanonicalFile (new File(config.getLogDir()));
        this.numPartitions = config.getNumPartitions();
        this.flushInterval = config.getFlushInterval();
        this.topicPartitionsMap = config.getTopicPartitionsMap();
        this.startupLatch = config.getEnableZookeeper() ? new CountDownLatch(1) : null;
        this.logFlushIntervalMap = config.getFlushIntervalMap();
        this.logRetentionSize = config.getLogRetentionSize();
        this.logRetentionMSMap = getLogRetentionMSMap(config.getLogRetentionHoursMap());
        //
    }

    public void setRollingStategy(RollingStrategy rollingStategy) {
        this.rollingStategy = rollingStategy;
    }

    public void load() throws IOException {
     // 滚动策略为fixedsize 的滚动策略
        if (this .rollingStategy == null) {
            this.rollingStategy = new FixedSizeRollingStrategy(config .getLogFileSize());
        }
        // 如果消息队列的根路径不存在,则创建
        if (!logDir .exists()) {
            logger.info( "No log directory found, creating '" + logDir.getAbsolutePath() + "'");
            logDir.mkdirs();
        }
        // 判断是否是目录并且可写
        if (!logDir .isDirectory() || !logDir.canRead()) {
            throw new IllegalArgumentException(logDir .getAbsolutePath() + " is not a readable log directory.");
        }
        // 获取这个里面的所有的子文件
        File[] subDirs = logDir.listFiles();
        if (subDirs != null) {
            for (File dir : subDirs) {
               // 跳过文件,因为在这个目录下,全部是topic-partition的文件夹
                if (!dir.isDirectory()) {
                    logger.warn( "Skipping unexplainable file '" + dir.getAbsolutePath() + "'--should it be there?");
                } else {
                    logger.info( "Loading log from " + dir.getAbsolutePath());
                   
                    // 获取到目录的名称
                    final String topicNameAndPartition = dir.getName();
                    if (-1 == topicNameAndPartition.indexOf('-' )) {
                        throw new IllegalArgumentException("error topic directory: " + dir.getAbsolutePath());
                    }
                   
                    // 将目录名称按照-进行分割,得到一个 kv,然后key就是topic,value是partition
                    final KV<String, Integer> topicPartion = Utils.getTopicPartition(topicNameAndPartition);
                    final String topic = topicPartion.k ;
                    final int partition = topicPartion.v;
                   
                    // 针对这个分区创建一个log对象
                    Log log = new Log(dir, partition, this.rollingStategy , flushInterval , needRecovery , maxMessageSize);

                    // 将新创建的log加入到map中
                    logs.putIfNotExists(topic, new Pool<Integer, Log>());
                    Pool<Integer, Log> parts = logs.get(topic);

                    parts.put(partition, log);
                   
                    // 如果时间的partition比配置中的大,则进行升级
                    int configPartition = getPartition(topic);
                    if (configPartition <= partition) {
                        topicPartitionsMap.put(topic, partition + 1);
                    }
                }
            }
        }

        /* Schedule the cleanup task to delete old logs */
        if (this .scheduler != null) {
            logger.debug( "starting log cleaner every " + logCleanupIntervalMs + " ms" );
            this.scheduler .scheduleWithRate(new Runnable() {

                public void run() {
                    try {
                        cleanupLogs();
                    } catch (IOException e) {
                        logger.error( "cleanup log failed.", e);
                    }
                }

            }, 60 * 1000, logCleanupIntervalMs);
        }
        // 将topic的注册任务注册到 zk里面
        if (config .getEnableZookeeper()) {
            this.serverRegister = new ServerRegister(config, this );
            serverRegister.startup();
            TopicRegisterTask task = new TopicRegisterTask();
            task.setName( "jafka.topicregister");
            task.setDaemon( true);
            task.start();
        }
    }

    private void registeredTaskLooply() {
        while (!stopTopicRegisterTasks ) {
            try {
                TopicTask task = topicRegisterTasks.take();
                if (task.type == TaskType.SHUTDOWN) break;
                serverRegister.processTask(task);
            } catch (Exception e) {
                logger.error(e.getMessage(), e);
            }
        }
        logger.debug( "stop topic register task");
    }

    class TopicRegisterTask extends Thread {

        @Override
        public void run() {
            registeredTaskLooply();
        }
    }

    // 将log保持的时间单位从小时换算到毫秒
    private Map<String, Long> getLogRetentionMSMap(Map<String, Integer> logRetentionHourMap) {
        Map<String, Long> ret = new HashMap<String, Long>();
        for (Map.Entry<String, Integer> e : logRetentionHourMap.entrySet()) {
            ret.put(e.getKey(), e.getValue() * 60 * 60 * 1000L);
        }
        return ret;
    }

    public void close() {
     // log的刷新器停止
        logFlusherScheduler.shutdown();
       
        // 将所有的log都关闭
        Iterator<Log> iter = getLogIterator();
        while (iter.hasNext()) {
            Closer. closeQuietly(iter.next(), logger);
        }
       
        // 构建一个topic shutdown的消息
        if (config .getEnableZookeeper()) {
            stopTopicRegisterTasks = true ;
            //wake up again and again
            topicRegisterTasks.add(new TopicTask(TaskType.SHUTDOWN , null));
            topicRegisterTasks.add(new TopicTask(TaskType.SHUTDOWN , null));
            Closer. closeQuietly(serverRegister);
        }
    }

    /**
     * Runs through the log removing segments older than a certain age
     *
     * @throws IOException
     */
    private void cleanupLogs() throws IOException {
        logger.trace( "Beginning log cleanup...");
        int total = 0;
        Iterator<Log> iter = getLogIterator();
        long startMs = System.currentTimeMillis();
       
        // 遍历所有的log执行对过期的log进行清理和过大的log进行清理的操作
        while (iter.hasNext()) {
            Log log = iter.next();
            total += cleanupExpiredSegments(log) + cleanupSegmentsToMaintainSize(log);
        }
        if (total > 0) {
            logger.warn( "Log cleanup completed. " + total + " files deleted in " + (System.currentTimeMillis () - startMs) / 1000 + " seconds" );
        } else {
            logger.trace( "Log cleanup completed. " + total + " files deleted in " + (System.currentTimeMillis() - startMs) / 1000 + " seconds");
        }
    }

    /**
     * Runs through the log removing segments until the size of the log is at least
     * logRetentionSize bytes in size
     *
     * @throws IOException
     */
    private int cleanupSegmentsToMaintainSize(final Log log) throws IOException {
     // 如果log大小永远有效或者还没有到清理的时机就不清理
        if (logRetentionSize < 0 || log.size() < logRetentionSize) return 0;

        // 对log文件执行过滤,过滤的条件是,是否log的大小超过了 retentionsize
        List<LogSegment> toBeDeleted = log.markDeletedWhile(new LogSegmentFilter() {

            long diff = log.size() - logRetentionSize;

            public boolean filter(LogSegment segment) {
                diff -= segment.size();
                return diff >= 0;
            }
        });
       
        // 将需要删除的 logsegment从log中删除
        return deleteSegments(log, toBeDeleted);
    }

    private int cleanupExpiredSegments(Log log) throws IOException {
     // 获取到当前时间
        final long startMs = System.currentTimeMillis ();
       
        // 获取到topic
        String topic = Utils.getTopicPartition(log.dir.getName()). k;
       
        // 获取到log清楚的毫秒数
        Long logCleanupThresholdMS = logRetentionMSMap.get(topic);
        if (logCleanupThresholdMS == null) {
            logCleanupThresholdMS = this.logCleanupDefaultAgeMs ;
        }
        final long expiredThrshold = logCleanupThresholdMS.longValue();
        List<LogSegment> toBeDeleted = log.markDeletedWhile(new LogSegmentFilter() {
            // 如果文件超时,则被过滤为需要删除的文件
            public boolean filter(LogSegment segment) {
                //check file which has not been modified in expiredThrshold millionseconds
                return startMs - segment.getFile().lastModified() > expiredThrshold;
            }
        });
       
        // 执行删除文件操作
        return deleteSegments(log, toBeDeleted);
    }

    /**
     * Attemps to delete all provided segments from a log and returns how many it was able to
     */
    private int deleteSegments(Log log, List<LogSegment> segments) {
        int total = 0;
        // 对于需要删除的LogSegment,关闭messageSet,删除原始文件
        for (LogSegment segment : segments) {
            boolean deleted = false;
            try {
                try {
                    segment.getMessageSet().close();
                } catch (IOException e) {
                    logger.warn(e.getMessage(), e);
                }
                if (!segment.getFile().delete()) {
                    deleted = true;
                } else {
                    total += 1;
                }
            } finally {
                logger.warn(String. format("DELETE_LOG[%s] %s => %s", log.name, segment.getFile().getAbsolutePath(),
                        deleted));
            }
        }
        return total;
    }

    /**
     * Register this broker in ZK for the first time.
     */
    public void startup() {
     // 注册broker,并创建topic的任务
        if (config .getEnableZookeeper()) {
            serverRegister.registerBrokerInZk();
            for (String topic : getAllTopics()) {
                serverRegister.processTask(new TopicTask(TaskType.CREATE , topic));
            }
            startupLatch.countDown();
        }
       
        // 初始化写log的刷新器
        logger.debug( "Starting log flusher every {} ms with the following overrides {}", config.getFlushSchedulerThreadRate(), logFlushIntervalMap);
        logFlusherScheduler.scheduleWithRate(new Runnable() {
            public void run() {
                flushAllLogs( false);
            }
        }, config.getFlushSchedulerThreadRate(), config.getFlushSchedulerThreadRate());
    }

    /**
     * flush all messages to disk
     *
     * @param force flush anyway(ignore flush interval)
     */
    public void flushAllLogs(final boolean force) {
        Iterator<Log> iter = getLogIterator();
        while (iter.hasNext()) {
            Log log = iter.next();
            try {
                boolean needFlush = force;
                // 如果不需要刷新
                if (!needFlush) {
                    long timeSinceLastFlush = System.currentTimeMillis() - log.getLastFlushedTime();
                    Integer logFlushInterval = logFlushIntervalMap.get(log.getTopicName());
                    if (logFlushInterval == null) {
                        logFlushInterval = config.getDefaultFlushIntervalMs();
                    }
                    final String flushLogFormat = "[%s] flush interval %d, last flushed %d, need flush? %s";
                    // 判断刷新时间间隔是否大于刷新频率,如果大于则其实为强制刷新
                    needFlush = timeSinceLastFlush >= logFlushInterval.intValue();
                    logger.trace(String.format(flushLogFormat, log.getTopicName(), logFlushInterval,
                            log.getLastFlushedTime(), needFlush));
                }
                if (needFlush) {
                    log.flush();
                }
            } catch (IOException ioe) {
                logger.error( "Error flushing topic " + log.getTopicName(), ioe);
                logger.error( "Halting due to unrecoverable I/O error while flushing logs: " + ioe.getMessage(), ioe);
                Runtime. getRuntime().halt(1);
            } catch (Exception e) {
                logger.error( "Error flushing topic " + log.getTopicName(), e);
            }
        }
    }

    private Collection<String> getAllTopics() {
        return logs .keySet();
    }

    private Iterator<Log> getLogIterator() {
        return new IteratorTemplate<Log>() {
            // 拿到一个topic里面的partition和Log的对应关系
            final Iterator<Pool<Integer, Log>> iterator = logs.values().iterator();

            Iterator<Log> logIter;

            @Override
            protected Log makeNext() {
                while (true ) {
                    if (logIter != null && logIter.hasNext()) {
                        return logIter .next();
                    }
                    if (!iterator .hasNext()) {
                        return allDone();
                    }
                    logIter = iterator.next().values().iterator();
                }
            }
        };
    }

    private void awaitStartup() {
        if (config .getEnableZookeeper()) {
            try {
                startupLatch.await();
            } catch (InterruptedException e) {
                logger.warn(e.getMessage(), e);
            }
        }
    }

    // 给一个topic和partition获取到partition和log的对应关系
    private Pool<Integer, Log> getLogPool(String topic, int partition) {
        awaitStartup();
        if (topic.length() <= 0) {
            throw new IllegalArgumentException("topic name can't be empty");
        }
        // 这个里面是有一些判断,但是逻辑没啥
        Integer definePartition = this.topicPartitionsMap .get(topic);
        if (definePartition == null) {
            definePartition = numPartitions;
        }
        if (partition < 0 || partition >= definePartition.intValue()) {
            String msg = "Wrong partition [%d] for topic [%s], valid partitions [0,%d)";
            msg = format(msg, partition, topic, definePartition.intValue() - 1);
            logger.warn(msg);
            throw new InvalidPartitionException(msg);
        }
        return logs .get(topic);
    }

    /**
     * Get the log if exists or return null
     *
     * @param topic     topic name
     * @param partition partition index
     * @return a log for the topic or null if not exist
     */
    public ILog getLog(String topic, int partition) {
     // 验证topic是否非法
        TopicNameValidator. validate(topic);
        Pool<Integer, Log> p = getLogPool(topic, partition);
        return p == null ? null : p.get(partition);
    }

    /**
     * Create the log if it does not exist or return back exist log
     *
     * @param topic     the topic name
     * @param partition the partition id
     * @return read or create a log
     * @throws IOException any IOException
     */
    public ILog getOrCreateLog(String topic, int partition) throws IOException {
     // 如果要创建的partition比配置中的partition大,则抛出异常
        final int configPartitionNumber = getPartition(topic);
        if (partition >= configPartitionNumber) {
            throw new IOException("partition is bigger than the number of configuration: " + configPartitionNumber);
        }
        boolean hasNewTopic = false;
        // 获取到这个topic所对应partition的一个pool
        Pool<Integer, Log> parts = getLogPool(topic, partition);
        if (parts == null) {
           // 如果pool为空,则创建一个pool
            Pool<Integer, Log> found = logs.putIfNotExists(topic, new Pool<Integer, Log>());
            if (found == null) {
                hasNewTopic = true;
            }
            parts = logs.get(topic);
        }
        // 根据partiition获取到对应的log
        Log log = parts.get(partition);
        if (log == null ) {
           // 创建log对象
            log = createLog(topic, partition);
            // 将新创建的logput到parts中
            Log found = parts.putIfNotExists(partition, log);
            if (found != null) {
                Closer. closeQuietly(log, logger);
                log = found;
            } else {
                logger.info( format("Created log for [%s-%d], now create other logs if necessary", topic, partition));
                final int configPartitions = getPartition(topic);
                // 对于配置的partition都做一下检查
                for (int i = 0; i < configPartitions; i++) {
                    getOrCreateLog(topic, i);
                }
            }
        }
        // 如果发现有新创建的topic,则构造任务,并注册到 zk中
        if (hasNewTopic && config .getEnableZookeeper()) {
            topicRegisterTasks.add(new TopicTask(TaskType.CREATE , topic));
        }
        return log;
    }

    /**
     * create logs with given partition number
     *
     * @param topic        the topic name
     * @param partitions   partition number
     * @param forceEnlarge enlarge the partition number of log if smaller than runtime
     * @return the partition number of the log after enlarging
     */
    public int createLogs(String topic, final int partitions, final boolean forceEnlarge) {
        // 判断topic名称是否合法
     TopicNameValidator.validate(topic);
        synchronized (logCreationLock ) {
            final int configPartitions = getPartition(topic);
            if (configPartitions >= partitions || !forceEnlarge) {
                return configPartitions;
            }
            topicPartitionsMap.put(topic, partitions);
            if (config .getEnableZookeeper()) {
                if (getLogPool(topic, 0) != null) {//created already
                    topicRegisterTasks.add(new TopicTask(TaskType.ENLARGE , topic));
                } else {
                    topicRegisterTasks.add(new TopicTask(TaskType.CREATE , topic));
                }
            }
            return partitions;
        }
    }

    /**
     * delete topic who is never used
     * <p>
     * This will delete all log files and remove node data from zookeeper
     * </p>
     *
     * @param topic topic name
     * @return number of deleted partitions or - 1 if authentication failed
     */
    public int deleteLogs(String topic, String password) {
     // 验证密码是否合法
        if (!config .getAuthentication().auth(password)) {
            return -1;
        }
        int value = 0;
        synchronized (logCreationLock ) {
           // 获取到这个topic里面的所有的partition以及对应的log
            Pool<Integer, Log> parts = logs.remove(topic);
            if (parts != null) {
               // 对log挨个执行删除操作
                List<Log> deleteLogs = new ArrayList<Log>(parts.values());
                for (Log log : deleteLogs) {
                    log.delete();
                    value++;
                }
            }
            // 如果zk enable,则要发送DELETE的任务
            if (config .getEnableZookeeper()) {
                topicRegisterTasks.add(new TopicTask(TaskType.DELETE , topic));
            }
        }
        return value;
    }

    private Log createLog(String topic, int partition) throws IOException {
        synchronized (logCreationLock ) {
           // 创建对应的文件夹,并且构造一个Log对象
            File d = new File(logDir , topic + "-" + partition);
            d.mkdirs();
            return new Log(d, partition, this.rollingStategy , flushInterval , false, maxMessageSize);
        }
    }

    // 获取log的个数
    private int getPartition(String topic) {
        Integer p = topicPartitionsMap.get(topic);
        return p != null ? p.intValue() : this.numPartitions ;
    }

    /**
     * Pick a random partition from the given topic
     */
    public int choosePartition(String topic) {
        return random .nextInt(getPartition(topic));
    }

    /**
     * read offsets before given time
     *
     * @param offsetRequest the offset request
     * @return offsets before given time
     */
    public List<Long> getOffsets(OffsetRequest offsetRequest) {
     // 根据topic和partition定位到log对象
        ILog log = getLog(offsetRequest. topic, offsetRequest.partition);
        if (log != null ) {
           // 得到log对象的写指针
            return log.getOffsetsBefore(offsetRequest);
        }
        return ILog.EMPTY_OFFSETS ;
    }

    public Map<String, Integer> getTopicPartitionsMap() {
        return topicPartitionsMap ;
    }

}
 
  • 大小: 68.9 KB
  • 大小: 25.1 KB
0
0
分享到:
评论
1 楼 liguanqun811 2016-10-09  
感觉LogManager打开了所有的LogSegment(文件)的句柄,会受到系统的限制吧?

相关推荐

    Laravel开发-logmanager

    `LogManager`是一个针对Laravel 5设计的界面工具,允许用户方便地预览、下载和删除日志文件,极大地提高了开发和维护效率。下面将详细解释Laravel的日志系统以及如何利用`LogManager`进行操作。 ### Laravel日志...

    LogManager_Linux.rar_Linux日志_LogManager_linux 日志_linux 日志_日志

    `LogManager`是一个专门为Linux设计的日志管理工具,它提供了日志分级和自动回滚功能,使得日志记录更加高效且易于维护。下面我们将深入探讨`LogManager`在Linux日志处理中的应用及其核心特性。 首先,日志分级是`...

    aaaaaaaaa

    在Java编程中,`LogManager`这个名字通常与日志管理相关,可能是Java Util Logging(JUL)框架的一部分。日志系统在软件开发中起着至关重要的作用,用于记录应用程序的运行状态、错误信息和调试信息。 `LogManager`...

    日志LogManager

    日志LogManager

    PyPI 官网下载 | flask-logmanager-0.2.9.tar.gz

    《Flask-LogManager:构建高效日志管理系统》 在Python的世界里,Flask是一个轻量级的Web服务框架,以其简洁和灵活的设计赢得了开发者们的喜爱。而在实际开发过程中,日志管理是不可或缺的一部分,它可以帮助我们...

    org.apache.log4j

    Log4j是Apache的一个开放源代码项目,通过使用Log4j,我们可以控制日志信息输送的目的地是控制台、文件、GUI组件、甚至是套接口服务器、NT的事件记录器、UNIX Syslog守护进程等;我们也可以控制每一条日志的输出格式...

    Log4net详细说明使用

    在代码中使用log4net,首先通过`log4net.LogManager.GetLogger("testApp.Logging")`获取一个日志记录器实例,然后调用如`log.Info("message")`的方法来记录信息。在这个例子中,`"testApp.Logging"`是日志器的名称,...

    log4j需要jar

    当我们在项目中遇到“java.lang.NoClassDefFoundError: org/apache/log4j/LogManager”这样的错误时,通常是因为缺少了Log4j的依赖库——log4j.jar。这个错误意味着系统在运行时找不到指定的类,即LogManager类,这...

    fst-1.60.zip

    《深入理解LogManager:探索日志管理的JDK实现与集成》 在IT行业中,日志管理是系统监控、故障排查以及性能优化的关键环节。本文将深入探讨标题为"fst-1.60.zip"的压缩包内容,该包包含了LogManager的相关实现,...

    Tomcat源代码学习研究

    - **Logging**:Tomcat使用LogManager和定制的日志框架,如Juli,提供灵活的日志记录方式。 10. **测试用例与代码分析** - **测试驱动开发**:通过单元测试和集成测试确保代码的正确性。学习源代码时,结合测试...

    log4net 学习用例

    private static readonly ILog log = LogManager.GetLogger(typeof(MyClass)); public void MyMethod() { try { // ... 执行代码 } catch (Exception ex) { log.Error("An error occurred", ex); } } } `...

    logmanager:Magento 日志管理器

    Magento 日志管理器有时您在 Magento 商家的网站上工作,您必须从 Magento 扩展程序中筛选数千行日志才能找到有用的东西。 此扩展程序可让您:基本功能 启用或禁用商店中某些 3rd 方扩展的日志记录查看所有正确安装...

    logManager:模拟器日志管理器

    日志管理器模拟器日志管理器至现在为止,其中包括一些功能: 通过两种方式将数据放入数据库:按用户手动和按脚本自动。 跟踪所有数据,然后显示为表格和数字。关于echarts的一些变化如何改变线条的颜色?...

    C# Nlog 日志记录 学习

    然后,创建并初始化NLog的`LogManager`和`Logger`对象: ```csharp using NLog; // 初始化NLog var config = new NLog.Config.XmlLoggingConfiguration("nlog.config"); NLog.LogManager.Configuration = config; ...

    log4net个人学习示例 vs2010

    在这里,我们使用`LogManager.GetLogger`获取一个日志对象,然后可以调用其各种方法(如Info、Error等)来记录不同级别的信息。 **参考资料** 压缩包中的文档提供了更深入的log4net使用指导: 1. **Log4Net使用...

    fo-dicom-development.zip

    总的来说,"fo-dicom-development.zip"提供的源码不仅展示了DICOM通信的基本操作,还涵盖了与工作列表服务相关的高级特性,对于医疗影像开发者和研究者而言,具有极高的学习价值。深入研究这些源码,有助于我们更好...

    jboss7.txt

    3. **设置引导类路径**:`-Xbootclasspath/p:$JBOSS_HOME/modules/org/jboss/logmanager/main/jboss-logmanager-1.2.2.GA.jar`、`-Xbootclasspath/p:$JBOSS_HOME/modules/org/jboss/logmanager/log4j/main/jboss-...

    log4qt qt5版本

    《基于Qt5的log4qt库改进版》 在软件开发过程中,日志记录是必不可少的一个环节,它能够帮助开发者追踪程序运行状态,定位错误,优化性能。在Qt框架下,log4qt是一个广泛使用的日志记录库,它借鉴了Java的log4j设计...

    c#非常好的日志类,很不错值得下载

    根据给定的信息,本文将详细解释C#中的一个日志类——`LogManager`,它提供了灵活的日志记录功能,并支持自定义文件名、日志类型等特性。 ### 日志类概述 在软件开发过程中,良好的日志记录机制对于调试程序、监控...

    LogManager:崩溃日志捕获上传以及处理

    (1)文件名称:手机名称_手机系统版本_环境_外部版本号_版本升级号_崩溃名称_用户ID.文本文件后缀如:华为_10.2_Debug_2.3.1_10_NSRangeException_1100112.txt(2)格式:纯文本格式(3)说明:版本升级号没有则不...

Global site tag (gtag.js) - Google Analytics