Date: Tue, 19 Mar 2024 01:38:12 +0000 (UTC) Message-ID: <988111515.52644.1710812292837@cwiki-he-fi.apache.org> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_52643_1199362073.1710812292837" ------=_Part_52643_1199362073.1710812292837 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html
Apache Ranger by design provides configurable audits destination= s. One of destinations is slf4j logging interface, which means audits can b= e streamed into any logging framework that is bound to slf4j. With this fea= ture, Ranger can support streaming audits into popular messaging bus e.g. K= afka to provide realtime data feeds for various monitoring system.
Enabling audits logging is as simple as adding some configurations in lo= gging properties files and adding a few kafka libraries.
Here is an example of configuring Hive logging properties file to enable= sending audit events to Kafka messaging bus.
# # kafka Appender # ### for Ranger 0.4.0 log4j.logger.com.xasecure.audit.provider.Log4jAuditProvider=3DINFO,KAFKA_HI= VE_AUDIT log4j.appender.KAFKA_HIVE_AUDIT=3Dkafka.producer.KafkaLog4jAppender log4j.appender.KAFKA_HIVE_AUDIT.BrokerList=3Dsandbox.hortonworks.com:6667 log4j.appender.KAFKA_HIVE_AUDIT.Topic=3Dhive_audit_log log4j.appender.KAFKA_HIVE_AUDIT.layout=3Dorg.apache.log4j.PatternLayout log4j.appender.KAFKA_HIVE_AUDIT.layout.ConversionPattern=3D%d{ISO8601} %-5p= [%t]: %c{2} (%F:%M(%L)) - %m%n log4j.appender.KAFKA_HIVE_AUDIT.ProducerType=3Dasync
For Kafka 0.8.1.x, the following jar files should be put into $HIVE_DIR/= lib
kafka_2.10-0.= 8.1.2.2.4.2-2.jar scala-library-2.10.4.jar metrics-core-2.2.0.jar
In consumer side, some deserializer should be there to parse normalized = audit records
public class = AuditLogJsonDeserializer extends JsonDeserializer<HiveAuditLogDataModel&= gt; { =09private static Logger LOG =3D LoggerFactory.getLogger(AuditLogJsonDeseri= alizer.class); =09@Override =09public HiveAuditLogDataModel deserialize(JsonParser jp, DeserializationC= ontext ctxt)=20 =09=09=09throws IOException, JsonProcessingException { =09=09HiveAuditLogDataModel model =3D new HiveAuditLogDataModel(); =09=09JsonNode node =3D jp.getCodec().readTree(jp); =09=09String resource =3D node.get("resource").asText(); =09=09// split resource to database, table, and column =09=09String[] tmp =3D resource.split("/"); =09=09if(tmp.length >=3D 1){ =09=09=09model.db =3D tmp[0]; =09=09=09if(tmp.length >=3D 3){ =09=09=09=09model.table =3D tmp[1]; =09=09=09=09model.column =3D tmp[2]; =09=09=09} =09=09} =09=09model.action =3D node.get("action").asText(); =09=09model.clientIP =3D node.get("cliIP").asText(); =09=09SimpleDateFormat formatter =3D new SimpleDateFormat("yyyyMMdd-HH:mm:s= s.SSS-Z"); =09=09try{ =09=09=09model.timestamp =3D formatter.parse(node.get("evtTime").asText()).= getTime(); =09=09}catch(Exception ex){ =09=09=09LOG.error("fail converting evtTime in hive audit log", ex); =09=09} =09=09model.user =3D node.get("reqUser").asText(); =09=09return model; =09} }