f5 ltm sends syslog messages with local/ causing indexing issues with SPlunk
We are trying to point all our f5's to Splunk for syslog, but the default remote servers option is sending local/ in the syslog messages causing indexing issues with Splunk (it's reading them as local messages instead of from remote hosts). When I try to use a modified include statement with a template, the syslog messages not only remove the local/, they also remove the hostname and duplicate the severity level.
With default syslog logging I get:
remote-servers {
Splunk {
host 10.10.60.111
local-ip none
remote-port 514
}
Msg: Dec 10 08:14:48 local/bpeca03-f501 debug snmpd[3542]: error on subcontainer 'ia_addr' insert (-1)\012
08:14:48.438475 00:01:d7:d5:ec:41 > 00:23:5e:56:76:bf, ethertype IPv4 (0x0800), length 138: (tos 0x0, ttl 64, id 34987, offset 0, flags [DF], proto: UDP (17), length: 124) 10.12.254.253.37261 > 10.10.60.111.syslog: SYSLOG, length: 96
Facility daemon (3), Severity debug (7)
Using the include statement:
include "template t_remotetmpl { template(\"<$PRI> $DATE $HOST $PRIORITY $MSG\\n\"); template_escape(no);};filter f_remote_loghost { level(debug..emerg);};destination d_remote_loghost { udp(\"10.10.60.111\" port(514) template(t_remotetmpl));};log { source(s_syslog_pipe); filter(f_remote_loghost); destination(d_remote_loghost);};"
The local/ is removed, but the $HOST variable isn't read properly (just shows up as local), and the $PRIORITY variable is duplicated in the syslog message:
Msg: Dec 10 08:14:48 local debug debug snmpd[3542]: error on subcontainer 'ia_addr' insert (-1)\012
08:15:18.439271 00:01:d7:d5:ec:41 > 00:23:5e:56:76:bf, ethertype IPv4 (0x0800), length 144: (tos 0x0, ttl 64, id 34987, offset 0, flags [DF], proto: UDP (17), length: 130) 10.12.254.253.43748 > 10.10.60.111.syslog: SYSLOG, length: 102
Facility daemon (3), Severity debug (7)
Has anyone see n this issue before? Any help would be appreciated. Thanks!
CH