Summary
The VES can be supported with the help of Kafka broker with the collectd in OPNFV Barometer project which is aim to collect telemetrics from the NFVI.
Barometer Project
The purpose of this project is providing metrics can be used to decide quality of NFVI. For this, the followings are reported
- NIC statistics
- Resources such as CPU, Memory, load, cache, themals, fan speeds, voltages and machine check exceptions.
This means the output of this project will be used in the host itself as well as inside of VM.
The original project name was “Software Fastpath Service Quality Metrics”.
Collectd
Barometer project uses collectd open source project(https://wiki.opnfv.org/display/fastpath/Collectd+101) It was already provide a lot of statistics can be collected from the system(usually unix systme)
Input and output plugins
To support diverse of data and interface, collectd has input and output plugin. The input plugins fed the data to the collectd and output plugins export the data to other system.
The supported plugins are listed in here.
Reference
Barometer and Collectd
Barometer develops several plugins for collectd.
- dpdkstat plugin: A read plugin that retrieves stats from the DPDK extended NIC stats API.
- dpdkevents plugin: A read plugin that retrieves DPDK link status and DPDK forwarding cores liveliness status (DPDK Keep Alive).
- gnocchi plugin: A write plugin that pushes the retrieved stats to Gnocchi. It’s capable of pushing any stats read through collectd to Gnocchi, not just the DPDK stats.
- aodh plugin: A notification plugin that pushes events to Aodh, and creates/updates alarms appropriately.
- hugepages plugin: A read plugin that retrieves the number of available and free hugepages on a platform as well as what is available in terms of hugepages per socket.
- Open vSwitch events Plugin: A read plugin that retrieves events from OVS.
- Open vSwitch stats Plugin: A read plugin that retrieves flow and interface stats from OVS.
- mcelog plugin: A read plugin that uses mcelog client protocol to check for memory Machine Check Exceptions and sends the stats for reported exceptions.
- PMU plugin: A read plugin that provides performance counters data on Intel CPUs using Linux perf interface.
- RDT plugin: A read plugin that provides the last level cache utilization and memory bandwidth utilization.
- virt: A read plugin that uses virtualization API libvirt to gather statistics about virtualized guests on a system directly from the hypervisor, without a need to install collectd instance on the guest.
- SNMP Agent: A write plugin that will act as a AgentX subagent that receives and handles queries from SNMP master agent and returns the data collected by read plugins. The SNMP Agent plugin handles requests only for OIDs specified in configuration file. To handle SNMP queries the plugin gets data from collectd and translates requested values from collectd’s internal format to SNMP format. Supports SNMP: get, getnext and walk requests.
VES Plugin for collectd
https://wiki.opnfv.org/display/fastpath/VES+plugin+updates
1st design - Implemented as Collectd Output plugin
- The VES collectd plugin as output plugin of collectd
- VES plugin get the data from the collectd and export to VES collector in VES format
- VES schema change requires VES collectd plugin updated
- Collectd requries MIT license while OPNFV prefer Apache license
2nd design - Use Kafka broker
VES Application(not collectd plugin anymore) get the data from the Kafka broker and convert the data based on the YAML configuration(collectd stats to VES events)