Search Syntax

This guide explains how to use search clause and filter your telemetry data including logs, traces, and metrics.

Table of Contents

• Quick Start
• Basic Concepts
• Query Structure
• Operators
• Field Context
• Data Types
• Functions
• Full-Text Search
• Advanced Examples
• Common Pitfalls

Quick Start

The most basic queries look like this:

service.name = 'payment-service'

You can combine multiple conditions:

service.name = 'payment-service' AND http.status_code >= 400

For full-text search, simply type what you're looking for:

'error connecting to database'

Note: Make sure the phrase you want to search is enclosed in single quotes.

Basic Concepts

Fields and Values

A query consists of fields (what you're searching in) and values (what you're searching for):

• Field: The property name (e.g., service.name, http.status_code)
• Value: What you're comparing against (e.g., 'payment-service', 200)

Query Structure

Conditions follow this general pattern:

field operator value

Multiple conditions can be combined with boolean operators:

condition1 AND condition2 OR condition3

Operators

Comparison Operators

String Matching Operators

Pattern matching wildcards:

• % matches any sequence of characters
• _ matches a single character

Range and Set Operators

Advanced Operators

Operators and Field Existence

This section describes how operators handle missing fields. Not all records contain every field. Understanding how operators behave with missing fields is crucial for accurate queries.

Positive Operators (Automatically Check Field Exists)

These operators only match logs where the field exists:

Example:

response_time_ms > 500

Matches: Logs with response_time_ms field AND value > 500
Skips: Logs without response_time_ms field

Negative Operators (DO NOT Check Field Exists)

These operators match all logs except those with the specified value (including logs missing the field):

Example:

service.name != 'redis'

Matches:

• Logs where service.name = 'api', 'auth', etc.
• Logs that don't have a service.name field at all

Skips: Only logs where service.name = 'redis'

Common Scenarios

Scenario 1: Find All Non-Redis Services

Ambiguous:

service.name != 'redis'

This includes logs without any service name!

Explicit intent:

service.name EXISTS AND service.name != 'redis'

This only includes logs that have a service name (and it's not 'redis')

Scenario 2: Find Logs Without Errors

Ambiguous:

severity_text != 'ERROR'

This includes logs that don't even have an severity_text field!

Clear intent - exclude ERROR level only:

severity_text EXISTS AND severity_text != 'ERROR'

Clear intent - find logs explicitly marked as non-error:

severity_text IN ('INFO', 'WARN', 'DEBUG')

Scenario 3: Find Failed Requests

Good:

status_code >= 400

Only matches logs that have a status_code field with error values

Be careful with:

status_code NOT BETWEEN 200 AND 399

This would include logs without any status_code field!

Boolean Operators

• AND: Both conditions must be true
• OR: At least one condition must be true
• NOT: Negates the condition

Operator Precedence (highest to lowest):

1. Parentheses ()
2. NOT
3. AND
4. OR

Full-Text Search

Note: full-text search is only supported for logs currently.

Quoted Text

Use quotes for exact phrase matching:

'failed to connect to database'

Unquoted Text

Without quotes, each word is searched separately:

error database connection

This searches for logs containing 'error' AND 'database' AND 'connection' (not necessarily together).

Regex

You can use any valid re2 syntax as the full text search.

Examples

'^\[SnapshotGenerator id=\d+\] Creating new KRaft snapshot file snapshot \d+-\d+ because .+ \d+ bytes\.$'

Matches the following log records with body

[SnapshotGenerator id=1] Creating new KRaft snapshot file snapshot 00000000000001109202-0000000001 because we have replayed at least 2800 bytes.
[SnapshotGenerator id=1] Creating new KRaft snapshot file snapshot 00000000000001109163-0000000001 because we have replayed at least 2800 bytes.
[SnapshotGenerator id=1] Creating new KRaft snapshot file snapshot 00000000000001109124-0000000001 because we have replayed at least 2800 bytes.
[SnapshotGenerator id=1] Creating new KRaft snapshot file snapshot 00000000000001108539-0000000001 because we have replayed at least 2800 bytes.
[SnapshotGenerator id=1] Creating new KRaft snapshot file snapshot 00000000000001108305-0000000001 because we have replayed at least 2800 bytes.

Matches logs records that contain email in log body

'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'

Combining Full-Text with Field Searches

'payment failed' AND service.name = 'payment-service'

Functions

Use functions to query array fields (currently supported for json body searches):

HAS Function

Checks if an array contains a specific value:

has(body.user_ids, 123)
has(body.tags, 'production')

HASANY Function

Checks if an array contains any of the specified values:

hasAny(body.regions, ['us-east', 'us-west'])
hasAny(body.status_codes, [500, 502, 503])

HASALL Function

Checks if an array contains all specified values:

hasAll(body.features, ['auth', 'payment', 'notification'])

Important: When to Use Quotes

Always use quotes when:

1. Searching for exact phrases
2. Your search term contains special characters or operators
3. You want to search for operator keywords as text

Examples where quotes are necessary:

# Wrong - AND will be treated as boolean operator
searching for AND operator

# Correct - searches for the phrase including 'AND'
'searching for AND operator'

Regular Expression (REGEXP)

The REGEXP operator uses RE2 syntax for pattern matching. RE2 is a fast, safe regular expression library that guarantees linear time execution. Common patterns:

• . - Any single character
• * - Zero or more of the preceding element
• + - One or more of the preceding element
• ? - Zero or one of the preceding element
• [abc] - Any character in the set
• [^abc] - Any character not in the set
• \d - Any digit
• \w - Any word character (letter, digit, underscore)
• \s - Any whitespace character

Examples:

# Match email addresses
email REGEXP '^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'

# Match IP addresses
ip_address REGEXP '^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$'

# Match URLs starting with https
url REGEXP '^https://'

# Match phone numbers (various formats)
phone REGEXP '^\+?1?\d{3}[-.]?\d{3}[-.]?\d{4}$'

Note: RE2 does not support lookahead/lookbehind assertions or backreferences.

Field Context

Fields can have explicit contexts to specify where to look for the data. This is crucial because telemetry data comes from multiple sources, and the same field name can exist in different contexts with different meanings or data types.

Why Field Context Matters

Consider a field like status_code - different teams use the same field names for different purposes, or when a field has different data types in different contexts (e.g., status_code might be a string in one context but a number in another).

Without explicit context, searching for status_code will search across ALL data types, which may:

• Return unexpected results
• Impact query performance
• Make debugging harder when the same key has different data types

How Context Resolution Works

1. With explicit context: resource.service.name = 'payment' searches ONLY in resource attributes
2. Without explicit data type: status_code = 200 searches across ALL data types where this field exists

Resource Context

Attributes about the source of telemetry:

resource.service.name = 'api-gateway'
resource.k8s.namespace.name = 'production'

Scope Context

Instrumentation library attributes:

scope.name = 'io.opentelemetry.redis'
scope.version = '1.0.0'

Signal-Specific Contexts

For Spans:

Let's take an example of name. This usually refers to the span name. However, if there is also an span attribute with same field name name, then any search without explicit context searches on both span name and attribute. To explicitly search on span name, you can refine the search as span.name = 'GET /users' or for attribute search as attribute.name = 'GET /users'

For Logs:

The following example show how to explicitly provide context

log.severity_text = 'ERROR'  (same as severity_text = 'ERROR')
log.body CONTAINS 'failed to connect' (same as body CONTAINS 'failed to connect')

Attribute Context

Attributes of the Log/Metric/Span item.

attribute.http.status_code = 200
attribute.user_name != 'anon'

Best Practices for Field Context

1. Always use explicit context when you know there is a conflicting key - This ensures accurate results (and better performance)
2. Be aware of data type conflicts - The same field name might have different types. Without explicit data type, the search attempts to match across all data types

Data Types

When field values could be interpreted as different types, you can explicitly specify data types using the :type syntax:

user.age:float64 > 18
price:float64 >= 99.99
is_active:bool = true

Supported data types:

• string
• float64
• bool
• []string, []int64, []float64, []bool (arrays)

Advanced Examples

Complex Boolean Logic

Use parentheses to control evaluation order:

(service.name = 'auth' OR service.name = 'user') AND status_code >= 400

Nested Conditions

region = 'us-east' AND (
  (status = 'error' AND retry_count > 3) OR
  (status = 'timeout' AND response_time_ms > 5000)
)

Duration-Based Queries with Other Conditions

log.severity_text = 'ERROR' AND 
resource.service.name IN ('payment', 'checkout') AND
http.request.duration > 1000

Searching Across Multiple Fields

(span.http.method = 'POST' OR span.http.method = 'PUT') AND
span.http.status_code BETWEEN 200 AND 299 AND
span.http.url LIKE '%/api/v2/%'

Common Pitfalls

Avoid these common mistakes when writing queries. Each example shows the incorrect approach followed by the correct syntax.

1. Forgetting Quotes for String Values

Not recommended: status = active
Correct: status = 'active'

2. Missing Wildcards in LIKE

Wrong: message LIKE 'error' (exact match only)
Correct: message LIKE '%error%' (contains 'error')

3. Incorrect Array Syntax

Wrong: region IN 'us-east', 'us-west'
Correct: region IN ('us-east', 'us-west')

4. Ambiguous Precedence

Wrong: a = 1 OR b = 2 AND c = 3
Correct: a = 1 OR (b = 2 AND c = 3)

5. Case Sensitivity

Remember that LIKE is case-sensitive. Use ILIKE for case-insensitive matching:

# Only matches 'Error' exactly
message LIKE '%Error%'

# Matches 'error', 'ERROR', 'Error', etc.
message ILIKE '%error%'

6. Special Characters in Search

If searching for special characters or operators as text, always use quotes:

# Searching for the literal text 'NOT'
message CONTAINS 'NOT'

# Searching for logs with text [debug]

'[debug]'

# Searching for text with operators
'response != 200'

Troubleshooting

This section helps you resolve common errors when writing search queries.

Syntax Errors

'found syntax errors while parsing the filter expression'

This error occurs when the query parser cannot understand your query syntax. Common causes:

1. Missing quotes around special characters

# Wrong - brackets cause parsing error
message = [error]

# Correct - use quotes for special characters
message = '[error]'

2. Unmatched parentheses or quotes

# Wrong - missing closing parenthesis
(service = 'api' AND status = 'error'

# Correct
(service = 'api' AND status = 'error')

3. Invalid operator usage

# Wrong - invalid operator combination
field NOT = 'value'

# Correct - use proper NOT syntax
field != 'value'
# OR
NOT field = 'value'

Expression Parsing Errors

'full text search is not supported'

This error appears when using full-text search syntax on signals that don't support it (currently only logs support full-text search).

# Wrong - using full-text search on traces/metrics
'database connection failed'  # when querying traces

# Correct - use field-based search for non-log signals
message CONTAINS 'database connection failed'

'key <fieldname> not found'

The specified field doesn't exist in your telemetry data. Common causes:

1. Typo in field name

# Wrong
sevice.name = 'api'  # typo: sevice instead of service

# Correct
service.name = 'api'

3. Incorrect JSON body field syntax

# Wrong - missing body prefix for JSON search
status = 'error'

# Correct - use body prefix for JSON fields
body.status = 'error'

'key <fieldname> is ambiguous'

This indicates the same field name exists in multiple contexts (i.e resource attribute and attribute of span/log). Please use explicit context by prefixing resource. or attribute. to the key.

# Warning - ambiguous field
status_code = 200
# Could match: span.status_code, attribute.status_code, etc.

# Better - be explicit
attribute.http.status_code = 200

'missing key for body json search'

When searching JSON body fields, you must specify the key after body.

# Wrong - incomplete body search
body. = 'error'

# Correct - specify the JSON key
body.error_message = 'timeout'

Function-Related Errors

'unknown function <name>'

Only specific functions are supported. Check for typos:

# Wrong - typo in function name
hasany(body.tags, ['prod', 'staging'])

# Correct - proper case
hasAny(body.tags, ['prod', 'staging'])

Supported functions:

• has() - check if array contains a value
• hasAny() - check if array contains any of the values
• hasAll() - check if array contains all values

'function expects key and value parameters'

Array functions require at least two parameters:

# Wrong - missing parameters
has(body.tags)

# Correct - provide field and value
has(body.tags, 'production')

'function supports only body JSON search'

Currently, array functions only work with JSON body fields:

# Wrong - using function on non-body field
has(attribute.tags, 'prod')

# Correct - use with body fields
has(body.tags, 'prod')

Value Type Errors

'unsupported value type'

Ensure values are one of the supported types:

# Supported value types:
'text value'      # String (quoted)
123               # Number
true              # Boolean
['a', 'b', 'c']   # Array (for IN clauses and functions)

'failed to parse number'

Number parsing failed. Check for invalid formats:

# Wrong
field = 123.456.789  # Invalid number format

# Correct
field = 123.456

Common Patterns and Solutions

1. Searching for operator keywords as text

# Wrong - AND interpreted as operator
searching for AND operator

# Correct - use quotes
'searching for AND operator'

2. Complex conditions with wrong precedence

# Wrong - unclear precedence
a = 1 OR b = 2 AND c = 3

# Correct - use parentheses
a = 1 OR (b = 2 AND c = 3)

3. Variable not found When using dashboard variables:

# If variable $service_name isn't defined
service = $service_name  # Error

# Define the variable or use a literal value
service = 'api-gateway'

Getting Help

If you encounter an error not listed here:

1. Check the search query syntax
2. Verify field names exist in your data
3. Try simplifying the query to isolate the issue
4. Check for typos and proper quotation
5. Ensure you're using the correct operators for your data type

If the issue still persists, reach out to support.

Best Practices

1. Start Simple: Begin with basic field comparisons and add complexity as needed
2. Use Field Context: Be explicit about context (resource., span., etc.) to avoid ambiguity
3. Test Incrementally: Build complex queries step by step, testing each addition
4. Leverage Autocomplete: Use the UI's autocomplete feature to discover available fields
5. Quote String Values: Always quote string values to avoid parsing ambiguities
6. Parenthesize Complex Logic: Use parentheses liberally to make intent clear

Last updated: July 31, 2025