Telemetry streaming to Elasticsearch

Question

Hi allI am following a couple of threads since I want to send ASM logging to Elasticsearch&nbsp; like this one from&nbsp;Greg&nbsp;What I understand is that I need to send an AS3 declaration and a TS declaration.But there are a couple of things not entirely clear to me.1. Can I remove the iRule, Service_TCP, Pool, Log_Destination, Log_Publisher and Traffic_Log_profile declarations from the AS3 declaration json?&nbsp;In the example the telemetry_asm_security_log_profile does not seem to depend on these?2. In the AS declaration json an IP address is specified 255.255.255.254 (perhaps just an example since it is a subnet mask) and also in the TS declaration where it is 172.16.60.194.How are the IP in the servers section of the AS3 declaration related to the one in the consumer part in the TS declaration?3. In&nbsp;telemetry_asm_security_log_profile the field remoteStorage is set to splunk.According to the reference guide:&nbsp;Reference Guide security-log-profile-application-object&nbsp;the allowed values are“remote”, “splunk”, “arcsight”, “bigiq”.&nbsp;I would opt for just remote. Is that the correct choice?Regards Hans

leslie_hubertus · Answer

Not sure if&nbsp;JRahm&nbsp;may be addressing any of this in his upcoming Embracing AS3: Foundations&nbsp;series?

jrahm · Answer

I won't be focusing on DO or TS in the series, and I haven't done anything with TS personally, but might be able to get some answers for you&nbsp;HGS-97-61.&nbsp;Let me ask around.

alex_madjeski · Answer

I am trying to answer the same questions.&nbsp; Were you able to get an answer?

ben_novak · Answer

Hi Hans.&nbsp; There is a bunch going on with a solution like this.&nbsp; All of the declaration pieces are required.&nbsp; At a high level the AS3 declaration defines the necessary objects to forward ASM logs to the local TS process, which collect data and send to Elastic/Splunk.&nbsp; The ASM logging profile can be configured to send logs to HSL destination so they are not written to the local file system (which can impact performance).&nbsp; The HSL destination points to the "telemetry_local" TCP virtual server, which uses the iRule to point to the TS listener service.&nbsp; I may be a little out of order on this explanation, but it should be close enough.

Do not remove anything.&nbsp; All pieces are required to get the traffic to Splunk as a single payload
I'm pretty sure the AS3 virtualAddress of 255.255.255.254 is an internal address listening on anything.&nbsp; This should minimize IP conflicts with any other possible user configurations on the system.&nbsp; It may be related specifically to TS since the listener doesn't define it, just the port 6514.&nbsp; The TS declaration destination of 172.16.60.23 is the Splunk system to send all the data after TS has collected it all.
You can have ASM logs sent directly to Splunk using the HSL configuration.&nbsp; Many customers do that for dedicated security dashboards.&nbsp; However, this whole solution is bundled as a package.&nbsp; To have all the data come in through TS.&nbsp; Probably so the preconfigured Splunk dashboards know how to find and parse all the data.&nbsp; Remember that TS will also include a ton of other system and application stats, not just the ASM logs.

&nbsp;

hgs-97-62 · Answer

Hello AlexYes I was able with F5 support.The best thing is to work with tmshDescribed here: https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/event-listener.html#requestlog&nbsp;Start with the irule and work your way down.After that you only need to create the listener like below{&nbsp; &nbsp; "class": "Telemetry",&nbsp; &nbsp; "controls": {&nbsp; &nbsp; &nbsp; &nbsp; "class": "Controls",&nbsp; &nbsp; &nbsp; &nbsp; "logLevel": "debug"&nbsp; &nbsp; },&nbsp; &nbsp; "My_System": {&nbsp; &nbsp; &nbsp; &nbsp; "class": "Telemetry_System",&nbsp; &nbsp; &nbsp; &nbsp; "systemPoller": {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "interval": "60"&nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; },&nbsp; &nbsp; "My_Listener": {&nbsp; &nbsp; &nbsp; &nbsp; "class": "Telemetry_Listener",&nbsp; &nbsp; &nbsp; &nbsp; "port": 6514,&nbsp; &nbsp; &nbsp; &nbsp; "trace": true&nbsp; &nbsp; },&nbsp; &nbsp; "My_Consumer": {&nbsp; &nbsp; &nbsp; &nbsp; "class": "Telemetry_Consumer",&nbsp; &nbsp; &nbsp; &nbsp; "type": "Generic_HTTP",&nbsp; &nbsp; &nbsp; &nbsp; "trace": false,&nbsp; &nbsp; &nbsp; &nbsp; "host": "10.0.1.111",&nbsp; &nbsp; &nbsp; &nbsp; "protocol": "http",&nbsp; &nbsp; &nbsp; &nbsp; "port": 9570,&nbsp; &nbsp; &nbsp; &nbsp; "path": "/",&nbsp; &nbsp; &nbsp; &nbsp; "method": "POST",&nbsp; &nbsp; &nbsp; &nbsp; "headers": [&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "name": "content-type",&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "value": "application/json"&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; &nbsp; ],&nbsp; &nbsp; &nbsp; &nbsp; "outputMode": "processed"&nbsp; &nbsp; }}&nbsp;&nbsp;&nbsp;

Forum Discussion

Telemetry streaming to Elasticsearch

5 Replies

Recent Discussions

rSeries and tenant upgrades

redirect not working

cat and grep command for rst messages

iRule resulting in too many redirects

When F5OS r2800 appliance reboots, interfaces configured at tenant level for VLAN are lost

Related Content

Setting up F5 Telemetry Streaming with Splunk Cloud

Automation Toolchain - Telemetry Streaming - Grafana StatsD Graphite

02 - Visualization of F5 BIG-IP metrics on Grafana using Prometheus and Telemetry Streaming service

Deploying BIG-IP Telemetry Streaming with Azure Sentinel as its consumer.

01 - Visualization of F5 BIG-IP metrics on Grafana using Prometheus and Telemetry Streaming service