Advanced Reporting with the NeoLoad CLI
The NeoLoad Command Line Interface (CLI) can produce custom reports based on test result data streamed to NeoLoad Web.
TL;DR
# one time only
pip install neoload
neoload login $NLW_TOKEN
# select a test to report on
neoload test-results use [test_result_name_or_guid]
# export data
neoload report --template=builtin:transactions-csv > transactions.csv
Custom Reporting Templates
You can also provide custom Jinja templates to produce HTML or other text-based output formats. Common use cases are:
- A custom single-file HTML report that can be attached to email or chat messages
- Raw data extracts formatted for import into other tools and platforms
- Multiple outputs for different teams or analysis processes
Examples:
neoload report --template tests/resources/jinja/sample-custom-report.html.j2 > ~/report.html
# or
neoload report --out-file ~/temp.json
neoload report --json-in ~/temp.json --template /path/to/a/jinja/devs.j2 > ~/teamA.html
neoload report --json-in ~/temp.json --template /path/to/a/jinja/ops.j2 > ~/teamB.html
neoload report --json-in ~/temp.json --template /path/to/a/jinja/dbas.j2 > ~/teamC.html
Some simple examples of custom HTML reports can be found here:
Additional Capabilities
Beyond basic exports, the CLI ‘report’ command provides additional capabilities, such as:
- Filtering data by timespan (e.g. only export ‘steady state’ section of data)
- Filtering data by element name, useful when producing comparisons on:
- specific transactions (e.g. login, checkout, submit claim/order)
- specific requests such as from a particular service or known issue area
- Combining filters
- Exporting JSON and applying multiple templates
- Building test-over-test trend comparisons
Building Comparisons and Test-over-test Trends
The NeoLoad CLI also helps you create multi-test results JSON aggregates. Consider:
neoload report --type=trends --filter="results=-4" \
--template tests/resources/jinja/sample-trends-report.html.j2
In this case, the NeoLoad CLI will consider the currently used test-result as the basis for the filter value of -4 which means “find results with the same project and scenario as the current result, and pull the last four (if that many exist) in chronological order.”
The above example is using Sample Jinja Templates from the NeoLoad CLI source repo. This is a good basis if you want to construct your own templates, and we encourage you to clone the NeoLoad CLI repo to get a copy of these templates. Once you do, you will probably want to make a copy of the jinja folder and customize that.
Comparing Specific Results
You can also use specific test-result GUIDs as results, pipe-delimited:
neoload report --type=trends --filter="results=9ae019b2-27b4-4598-8db1-8369e277ff4b|d27bc9cd-9e3c-4f1e-8167-2f1e7554f5ec"
This is often useful when comparing the current test run with other known baselines. After you run a test via the CLI, that latest test run is automatically selected as the current test-result your CLI is referring to. Therefore, if comparing to a baseline, you only need to provide the baseline test-result GUID in the filter above.
Trending Based on Specific Results Criteria
The NeoLoad CLI test-results list (‘ls’) command produces JSON which you can process via standard tools like jq. You can filter CLI results based on your own specific filtering criteria and then turn them in to a delimited list for use by the CLI report filter results specifier:
PROJECT=rest_api
SCENARIO=fullTest
# compile a CSV list of GUIDs based on specific project, scenario, execution status
GUIDS=$(neoload test-results --filter="project=$PROJECT|status=TERMINATED|scenario=$SCENARIO" ls \
| jq '.[:5][]|.id' -r | tr '\n' '|')
neoload report --type=trends --filter="results=$GUIDS" \
--template tests/resources/jinja/sample-trends-report.html.j2
Comments
Count