Shape data

TQL comes with numerous transformation operators that change the shape of events. TQL has functions that work on values within a single event.

This guides showcases the operators and functions that you need for day-to-day data shaping tasks.

Use operators to reshape data flows

Operators work on streams of events and—unlike functions—can keep state between multiple events. Here is a visual overview of transformations that you can perform over a stream of events:

Let’s dive right in.

Filter events with where

Use where to filter events in the input with an expression:

from {x: 1, y: "foo"},
     {x: 2, y: "bar"},
     {x: 3, y: "baz"}
where x != 2 and y.starts_with("b")

{x: 3, y: "baz"}

Slice events with head, tail, and slice

Use the head and tail operators to get the first or last N records of the input.

Get the first event:

from {x: 1, y: "foo"},
     {x: 2, y: "bar"},
     {x: 3, y: "baz"}
head 1

{x: 1, y: "foo"}

Get the last two events:

from {x: 1, y: "foo"},
     {x: 2, y: "bar"},
     {x: 3, y: "baz"}
tail 2

{x: 2, y: "bar"}
{x: 3, y: "baz"}

tail is blocking

The tail operator must wait for its entire input, whereas head N terminates immediately after the first N records have arrived. Use head for the majority of use cases and tail only when you have to.

The slice operator generalizes head and tail by allowing for more flexible slicing. For example, to return every other event starting from the third:

from {x: 1, y: "foo"},
     {x: 2, y: "bar"},
     {x: 3, y: "baz"},
     {x: 4, y: "qux"},
     {x: 5, y: "corge"},
     {x: 6, y: "grault"}
slice begin=3, stride=2

{x: 4, y: "qux"}
{x: 6, y: "grault"}

Pick fields with select and drop

Use the select operator to pick fields:

from {x: 1, y: "foo"},
     {x: 2, y: "bar"},
     {x: 3, y: "baz"}
select x

{x: 1}
{x: 2}
{x: 3}

The drop operator is the dual to select and removes the specified fields:

from {x: 1, y: "foo"},
     {x: 2, y: "bar"},
     {x: 3, y: "baz"}
drop x

{y: "foo"}
{y: "bar"}
{y: "baz"}

Sample schemas with taste

The taste operator provides a sample of the first N events of every unique schemas. For example, to get 3 unique samples:

from {x: 1, y: "foo"},
     {x: 2, y: "bar"},
     {x: 1},
     {x: 2},
     {y: "foo"}
taste 1

{x: 1, y: "foo"}
{x: 1}
{y: "foo"}

Invert event ordering with reverse

Use reverse to invert the order of a stream of events:

from {x: 1},
     {x: 2},
     {x: 3}
reverse

{x: 3}
{x: 2}
{x: 1}

Add fields with set assignment

Use the set operator to add new fields to the output.

from {x: 1},
     {x: 2}
set y = x + 1

{x: 1, y: 2}
{x: 2, y: 3}

You can assign a field name name and project at the same time with an assignment in select:

from {x: 1, y: "foo"},
     {x: 2, y: "bar"}
select y=x

{y: 1}
{y: 2}

Relocate fields with move

Use the move operator to combine set and drop. This pipeline

from {old: 42}
move new = old

produces the same output as this one:

from {old: 42}
set new = old
drop old

{new: 42}

Moving multiple fields is also possible. The pipeline

from {foo: 1, bar: 2}
move foo=bar, qux=foo

is equivalent to

from {foo: 1, bar: 2}
set foo=bar, qux=foo
drop bar, foo

and produces:

{
  foo: 2,
  qux: 1,
}

However, it is not equivalent to the following pipeline containing single assignments:

from {foo: 1, bar: 2}
set foo=bar
drop bar
set qux=foo
drop foo

which yields:

{
  qux: 2,
}

Aggreate events with summarize

Use summarize to group and aggregate data.

from {x: 0, y: 0, z: 1},
     {x: 1, y: 1, z: 2},
     {x: 1, y: 1, z: 3}
summarize y, x=sum(x)

{y: 0, x: 0}
{y: 1, x: 2}

A variety of aggregation functions make it possible to combine grouped data.

Reorder events with sort

Use sort to arrange the output records according to the order of a specific field.

from {x: 2, y: "bar"},
     {x: 3, y: "baz"},
     {x: 1, y: "foo"}
sort -x

{x: 3, y: "baz"}
{x: 2, y: "bar"}
{x: 1, y: "foo"}

Prepending the field with - reverses the sort order.

Break up lists of records with unroll

Working with lists of can be cumbersome. To break up a list into a sequence of events, use the unroll operator:

from {
  xs: [{a: 1}, {a: 2}],
  y: "foo",
}
unroll xs

{
  xs: {
    a: 1,
  },
  y: "foo",
}
{
  xs: {
    a: 2,
  },
  y: "foo",
}

Note that unroll produces as many events as there are elements in the unrolled list. All other fields are simply copied over.

Use functions to change single events

Functions work on single values, including the top-level event that you can reference with the this keyword.

Combine records with merge

Use the merge function to combine two records:

from {
  foo: {
    bar: 1,
    baz: 2,
  },
  qux: {
    fred: 3,
    george: 4,
    bar: 5,
  }
}
this = merge(foo, qux)

{
  bar: 5,
  baz: 2,
  fred: 3,
  george: 4
}

Note that the field bar, which is present in both record foo and qux, has the value 5 in the output because fields from the second argument to merge overwrite fields from the first if they have the have the same name.

Instead of writing

this = merge(foo, qux)

you can also write

this = {...foo, ...qux}

as an idiomatic shorthand. The spread expression ... expands records in the exact same was as merge.

Combine lists with concatenate

Use concatenate to add one list add the end of another:

from {
  xs: [1,2,3],
  ys: [4,5,6],
}
select result = concatenate(xs, ys)'

{
  result: [
    1,
    2,
    3,
    4,
    5,
    6,
  ],
}

The spread expression ... also works for lists. You can rewrite the above example

select result = concatenate(xs, ys)'

more idiomatically as

select result = [...xs, ...ys]'

and get the same result.

Add values to lists with append and prepend

To add a single value to a list, use append to add it to the back and prepend to add it to the front:

from {
  xs: [2],
}
xs = append(xs, 3)
xs = prepend(xs, 1)

{
  result: [
    1,
    2,
    3,
  ],
}