Writing Kibana 4 Plugins – Visualizations using Data
You need to read Part 2 - Simple Visualization before you read this tutorial.
In the previous part of this tutorial series (which is a required must-read for this part) you’ve learned to create a simple visualization, that doesn’t need access to data from Elasticsearch. In this part we will write another plugin, that uses aggregations (like most of the visualizations) to access data from Elasticsearch.
We will create a very simple tag cloud plugin, that shows the bucket name as a label and the result of the metrics aggregation determines the fontsize of the tag. If you are feeling unfamiliar with bucket and metrics aggregations have a look at my Kibana 4 Visualize tutorial.
The sources of this tutorial can be found on GitHub:
We will use the name tr-k4p-tagcloud throughout this tutorial. You have to replace it with your appropriate unique plugin name.
The tagcloud visualization
First of all you should of course think what data you want to visualize in what way. This means, you should now what kind of bucket and metrics aggregation will result in what visualization.
We keep our visualization very simple and allow exactly one bucket and one metric aggregation. The bucket aggregation will determine which tags will be shown (each bucket will result in one tag). The metrics aggregation will result in the font size of the tag of each bucket, i.e. the higher the metric aggregations result the larger the bucket tag will be shown.
It is important for you to think about how many metrics or bucket aggregations you want to visualize and if you can nest aggregations, etc. You will need to define this later in your plugin.
Define and register you visualization
As in the previous part, the first steps are to create the index.js
, package.json
and register a simple visualization provider. We will just show highlight the visualization provider (which can be found in the public/tagcloud.js
file):
function TagcloudProvider(Private) {
var TemplateVisType = Private(require('ui/template_vis_type/TemplateVisType'));
return new TemplateVisType({
name: 'trTagcloud', // The internal id of the visualization (must be unique)
title: 'Tagcloud', // The title of the visualization, shown to the user
description: 'Tagcloud visualization', // The description of this vis
icon: 'fa-cloud', // The font awesome icon of this visualization
template: require('plugins/tr-k4p-tagcloud/tagcloud.html')
});
}
require('ui/registry/vis_types').register(TagcloudProvider);
There are two changes to the previous tutorial part:
- We leave out the
define()
block (AMD module definition) around the plugin. As mentioned in the previous part, this is not needed anymore due to bundeling with webpack. That also means, you don’t return the provider function anymore inside the module. - We leave out the
requiresSearch: false
in the definition of the visualization. Our tagcloud will use data from Elasticsearch, so you should be able to link it to a search like all other visualizations that consume data. SincerequiresSearch: true
is the default value we can just leave the key.
You will need to create the public/tagcloud.html file, but can leave it empty for now. You can find the first step in tag 0.1.0 in the GitHub project.
Define your schemas
For a visualization that uses data aggregation, you need to specify exactly what aggregations your visualization needs or is allowed to have. These so called schemas will be added to the visualization description. Let’s modify our description as follows:
function TagcloudProvider(Private) {
var TemplateVisType = /* ... */;
// Include the Schemas class, which will be used to define schemas
var Schemas = Private(require('ui/Vis/Schemas'));
return new TemplateVisType({
/* every attribute shown above */,
schemas: new Schemas([
{
group: 'metrics',
name: 'tagsize',
title: 'Tagsize',
min: 1,
max: 1,
aggFilter: ['count', 'avg', 'sum', 'min', 'max', 'cardinality', 'std_dev']
},
{
group: 'buckets',
name: 'tags',
title: 'Tags',
min: 1,
max: 1,
aggFilter: '!geohash_grid'
}
])
});
}
To define your schemas, you create a new Schemas
object, which will take an
array of objects in its contructor. Each object describes one aggregation you
accept for your visualization. Each aggregation object have the following keys:
- group - either “metrics” or “buckets”. Will define, which kind of aggregation you want to describe in this object.
- name - the name (id) of this aggregation. You can use this later to get a reference to the different aggregations again.
- title - the title shown to the user, when he adds the aggregation. Should describe how that aggregation will be visualized (e.g. in that case the bucket aggregation will create tags, the metrics aggregation will influence the tag size)
- min/max - the number of minimum and maximum aggregations of that type, a user can add. E.g. the vertical bar chart has a bucket aggregation for “Split Bars”. It is not limited (i.e. no max value specified) since it can split the bar as many times as the user wishes. In our case we only allow 1 aggregation of each type, due to the way our visualization works.
- aggFilter - a filter on which aggregations should be allowed. It is an array of either aggregation types (see below), that are allowed in this place (as shown in our metrics aggregation) or an array of aggregation types forbidden (each must be prefixed with a bang). In the later case all other aggregations are allowed. If the array has only one element you can also specify it as a string (as shown in the bucket aggregation).
There are some more keys you can specify in the aggregation, which we won’t cover in this tutorial.
The types, that you can specify for metrics aggregations’ aggFilter
are the
following: avg, cardinality, count, max, median, min, percentile_ranks,
percentiles, std_dev, sum
The types, that you can specify for bucket aggregations’ aggFilter are the following: date_histogram, date_range, filters, geohash_grid, histogram, ip_range, range, significant_terms, terms
If you load this plugin inside Kibana you should now be able to chose aggregations as you described, when creating a new visualization of your type.
You can find the sources of this step in tag 0.2.0 on GitHub.
Writing the controller
To add logic to our visualization we will again need an Angular controller. In contrast to the previous part we will extract it into its own file (since it will be a bit larger). Therefor just add the following line to the head of your tagcloud.js
file:
require('plugins/tr-k4p-tagcloud/tagcloudController')
Now you can create a new file public/tagcloudController.js
with an empty controller:
var module = require('ui/modules').get('tr-k4p-tagcloud');
module.controller('TagcloudController', function($scope) {
// Your logic will go here
});
To load the controller modify your template (public/tagcloud.html
) as follows:
<div ng-controller="TagcloudController"></div>
Now we have an empty controller, that will be loaded for our visualization.
Accessing data
There are two variables inherited into your angular scope, that you will need.
One is the vis variable, already seen in the last part of the series, which
holds information about your visualization and the settings the user chose. The
other variable is named esResponse
and holds the Elasticsearch response for
your visualization. Kibana will automatically query Elasticsearch with the
aggregations set by the user and taking into account currently set queries and
filters.
To visualize our data we need to match the response data with the user
configuration for our widget. To access the result of the aggregations we can
look into $scope.esResponse.aggregations
. To find aggregations in that object
we need their ids. To find the ids for a specific aggregation we can use several
methods of $scope.vis.aggs
to find the id.
In our case we first want to extract all the buckets (i.e. the tag names) from the response. You can access the id of the configured tags aggregation as follows:
$scope.vis.aggs.bySchemaName['tags'][0].id
The bySchemaName object contains a mapping of the names (that you have specified
in your schema) to the aggregation configuration. Accessing the tags key will
give us an array of all the aggregation configurations the user has entered for
tags. Since we have set min and max to 1, we know, that there is only one object
and can retrieve its id. We can use that id to lookup the results in
esResponse
.
You generally want to create a watch on the esResponse
variable and update your
data when that variable changes. So let’s put everything together to show a
simple list of tags:
$scope.$watch('esResponse', function(resp) {
if (!resp) {
$scope.tags = null;
return;
}
// Retrieve the id of the configured tags aggregation
var tagsAggId = $scope.vis.aggs.bySchemaName['tags'][0].id;
// Get the buckets of that aggregation
var buckets = resp.aggregations[tagsAggId].buckets;
// Transform all buckets into tag objects
$scope.tags = buckets.map(function(bucket) {
return {
label: bucket.key
};
});
});
That way $scope.tags will have an array ob objects (one for each bucket) with the bucket key as a label. You can now change your tagcloud.html accordingly:
<div ng-controller="TagcloudController">
<span ng-repeat="tag in tags">{{tag.label}}</span>
</div>
You can find the full version (with some CSS added) in 0.3.0 in the repository.
Accessing the metrics aggregation is a little bit easier, but follow similar steps. First we need to get a reference to the metrics aggregation we want to read out for the bucket:
var metricsAgg = $scope.vis.aggs.bySchemaName['tagsize'][0];
Notice, that we don’t read out the id, but the whole aggregation object in that case. Again we can just access, the first element in the array, since we only allowed one metrics aggregation to be configured. We can now complete mapping the buckets with the value from the metrics aggregation:
$scope.tags = buckets.map(function(bucket) {
return {
label: bucket.key,
value: metricsAgg.getValue(bucket)
};
});
We can use the getValue
method on an metrics aggregation object and pass in
the bucket to return the value of this metrics aggregation for the specified
bucket. After this we have a list of tags with label and a value. The last thing
we need to do now is to calculate a font size for each tag. To do this we will
calculate the minimum an maximum value of all tags while collecting the tag
array and use a minimum and maximum font size to calculate afterwards the font.
Since this is not anyhow Kibana specific, but some regular AngularJS JavaScript,
we won’t show it here, but you can look into the tagcloudController.js and
tagcloud.html file in version
0.4.0 on GitHub.
Accessing data sum-up
The tag cloud is a pretty easy example of how to access data. It only has one bucket aggregation and one metrics aggregation. No multiple aggregations, no nested aggregations, etc. In a more complex visualization you might have all that. So let’s try to generalize the process a little bit:
- You can get access to the configured visualization objects via
$scope.vis.aggs
and the different sub methods:bySchemaName
(the names you configured in your schema),bySchemaGroup
(metrics or buckets),byTypeName
(e.g. count, terms, etc.) - To access data inside a bucket you can use the
getValue
method on the aggregation object.
In general there are some more methods, that might become useful in more complex
visualizations (e.g. you could also get the key of a bucket, by using the
aggregation object’s getKey
method). To find out these methods, you will need to
wait for some official plugin development documentation, grep the Kibana source
code or (what often works best) setting yourself breakpoints during development
and just inspect the objects in your browsers dev tools.
Adding filters on click
The last feature we would like to add to our tag cloud is filtering. If the user clicks on a tag, a filter for that value should be automatically added to the dashboard.
First step to create a filter is to retrieve the filter manager service. We will use the Private service (which is responsible for instantiating angular services from required modules, as explained in the previous tutorial part) to instantiate the filter service. The required modifications at the controller:
module.controller('TagcloudController', function($scope, Private) {
var filterManager = Private(require('ui/filter_manager'));
// ...
});
The filter manager has one method called add which we can utilize to create a new filter. First let’s change the HTML to call a method if the user clicks a filter:
<span ng-click="filter(tag)" ng-repeat="tag in tags" ...>
Now we can implement the filter method:
$scope.filter = function(tag) {
// Add a new filter via the filter manager
filterManager.add(
// The field to filter for, we can get it from the config
$scope.vis.aggs.bySchemaName['tags'][0].params.field,
// The value to filter for, we will read out the bucket key from the tag
tag.label,
// Whether the filter is negated. If you want to create a negated filter pass '-' here
null,
// The index pattern for the filter
$scope.vis.indexPattern.title
);
};
You can find this last step of the plugin in version 0.5.0 on GitHub.
What’s next?
Now you are able to write your first visualization, that consumes data from
Elasticsearch. If you write your visualization always keep in mind, what schemas
you really can visualize and make sure you only allow schemas, that you actually
also check for in your code. So if you allow multiple bucket aggregations, make
sure to visualize every bucket, you will get from
$scope.vis.aggs.bySchemaName['foobar']
and not just the first one, as we did.