How to push new information to Chukwa
Add a new dataSource (Source Input)
Using Log4J
Chukwa comes with a Log4J Appender. Here the steps that you need to follow in order to use it:
- Create a log4j.properties file that contains the following information:
log4j.rootLogger=INFO, chukwa
log4j.appender.chukwa=org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender
log4j.appender.chukwa.File=${CHUKWA_HOME}/logs/${RECORD_TYPE}.log
log4j.appender.chukwa.DatePattern='.'yyyy-MM-dd
log4j.appender.chukwa.recordType=${RECORD_TYPE}
log4j.appender.chukwa.layout=org.apache.log4j.PatternLayout
log4j.appender.chukwa.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n - Add these parameters to your java command line:
- -DCHUKWA_HOME=${CHUKWA_HOME} -DRECORD_TYPE=<YourRecordType_Here> -Dlog4j.configuration=log4j.properties
- -DRECORD_TYPE=<YourRecordType_Here> is the most important parameter.
- You can only store one record type per file, so if you need to split your logs into different record types just create one appender per data type (%T% see hadoop logs4j configuration file)
- Start your program. Now all your log statements should be written in ${CHUKWA_HOME}/logs/<YourRecordType_Here>.log
Static file like /var/log/messages
- Edit ${CHUKWA_HOME}/conf/initial_adaptors
- Add a line similar to this one:
- add org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLine SysLog 0 /var/log/messages 0
- add org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLine SysLog 0 /var/log/messages 0
This line will automatically register the "CharFileTailingAdaptorUTF8NewLine" adaptor for /var/log/messages
Register a file from another application/language
- Open a socket from your application to the ChukwaLocalAgent
- Write this line to the socket
- Add org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUTF8NewLine <RecordType> <StartOffset> <fileName> <StartOffset>
- Where <RecordType> is the data type that will identify your data
- Where <StartOffset> is the start offset
- Where <fileName> is the local path on your machine
- Close the socket