Skip to main content

Test execution via the command line and --noUI support

Requirements

The following components are required for using the new CLI syntax:

  • CODESYS V3.5 SP19 Patch 3 or newer

  • CODESYS Test Manager 5.2.0.0 or newer

Support of the --noUI CLI option requires CODESYS V3.5 SP22 or newer instead. In addition, the --noUI option is supported only with the new --runtest CLI syntax.

Tip

In the Test Manager view in CODESYS, the Copy Command Line to Clipboard command is available in the context menu of test scripts. This command copies the CLI call for the selected test script in the new CLI syntax to the clipboard.

Tip

Arguments which contain spaces, hyphens, or slashes should be enclosed in double quotation marks: "<argument>"

Tip

Credentials might be logged to stdout in plain text.

New CLI syntax --runtest

CLI option

Description

Optional

Values

Note

--runtest

Entry point for every CLI test execution.

No

-

Without this command, all subsequent options are ignored.

--testrepositorylocation=<argument>

The absolute path of the top-level directory of the test repository.

Example: --testrepositorylocation="D:\Test Manager\TestRepository"

No

The path must point to a valid test repository.

A test repository which is not yet known in %LOCALAPPDATA%\CODESYS Test Manager\TestManager.xml is registered before the test execution starts. This ensures smooth test execution in a CI/CD context.

--testscript=<argument>

The qualified name of the test script within the test repository.

The folder and script are separated by a period.

Example: --testscript=FolderA.FolderC.Testscript

No

The syntax for the path follows the notation in the IDE, not the file path.

--testreportfolder=<argument>

The absolute path to the directory where the test reports are saved.

Example: --testreportfolder="D:\Test Manager\MyTestReports"

No

The path must be absolute and valid.

Directories in the path which do not exist yet are created.

--testscriptversion=<argument>

Version of the test script.

Yes

An invalid or nonexistent version causes an error during test execution.

If no version is specified, then the newest available version of the test script is executed.

--tester=<argument>

The name which is listed in the test report.

Yes

Default: Name of the user of the current OS login session

--testreportformat=<argument>

The format in which the test report(s) are saved.

Example: --testreportformat=html,xml

Yes

Possible values:

  • None

  • Html

  • Xml

Default: None

Multiple options can be selected. They must be separated by commas without spaces.

If no value, an invalid value, or only None is specified, then no test report is saved.

With the combination of --testreportformat=None and --testnoexitaftertest, the test report is displayed in CODESYS and can be saved from the UI.

--testprogress=<argument>

The channel for the output of the test progress.

Yes

Possible values:

  • None

  • stdout

  • stderr

  • window

Default UI: stdout,window

Default noUI: stdout

The default for a test execution with UI includes the stdout option. Common CI/CD systems work with the output to stdout, among other things to detect whether a job in a CI/CD pipeline is hanging.

--testnovarrecording

A reduced test report is generated. Variable values are not listed.

Yes

-

--testnoexitaftertest

CODESYS is not closed after the test execution.

Yes

-

--testdebugmode=<argument>

The test execution is started in debug mode.

Yes

Possible values:

  • None

  • HoldOnStart

  • HoldOnError

  • HoldOnBreakpoint

  • All

Default: None

Multiple options can be selected. They must be separated by commas without spaces.

A combination with any of the following options results in an error:

  • --noUI

  • --testcalibration

--testcalibration

The test execution is started in calibration mode.

Yes

-

A combination with any of the following options results in an error:

  • --noUI

  • --testdebugmode=<argument>

--noUI

General CLI option of CODESYS

CODESYS is started without a graphical user interface. However, this operating mode still requires a Windows desktop session. Depending on the CODESYS functionality which is used in the script, modal dialogs may still appear.

Yes 

Compatibility of the legacy CLI syntax --executecommand="TestManager ExecuteScript"

The legacy syntax is still supported. However, the introduction of the new CLI command has resulted in some changes.

Important

It is recommended that you switch to the new syntax.

The --noUI CLI option is still not supported for the legacy syntax, regardless of the CODESYS version.

New and old syntax cannot be mixed.

Legacy CLI option

Change as of CODESYS Test Manager version 5.2.0.0

--repository-location=<argument>

-

--repository-name=<argument>

Omitted

As of version 5.2.0.0 of the CODESYS Test Manager, test repositories are no longer identified by their name.

--script=<argument>

-

--version=<argument>

-

--tester=<argument>

-

--save-xml=<argument>

-

--save-html=<argument>

-

--store

Omitted

As of version 5.2.0.0 of the CODESYS Test Manager, the internal report repository is no longer supported.

--report-name=<argument>

Omitted

As of version 5.2.0.0 of the CODESYS Test Manager, the internal report repository is no longer supported.

--overwrite-files

Omitted

As of version 5.2.0.0 of the CODESYS Test Manager, old test reports are overwritten by default.

--progress=<argument>

The option to specify a log file (absolute path) is no longer supported.

In the CI/CD context, working with stdout has proven effective, so this option no longer provides any added value.

--no-var-recording

-

Exit codes

The following exit codes replace the previous exit codes because the previous exit codes no longer correspond to the implementation of CODESYS Test Manager version 5.2.0.0.

Important

When you start the CODESYS executable directly from the command line, the console returns control immediately by default. The console does not wait until the CODESYS process has ended. This is because CODESYS is a desktop application and not a console application. To make the console wait for the CODESYS process to end, use the Windows command start /wait <command line>. Alternatively, you could also call the command line from a batch file. Use one of these methods to reliably read the exit code.

Exit code

Meaning for the new CLI command

0

No error has occurred.

2

During execution in --noUI and --strict mode, a modal dialog was intercepted and the process was aborted (available as of CODESYS version 3.5 SP23)

9

The --testscript argument has not been specified.

14

The XML test report cannot be saved.

15

The HTML test report cannot be saved.

16

An error occurred while processing the specified --testprogress options.

17

The execution did not return a test result.

19

No valid license has been found for the CODESYS Test Manager. No test report has been created.

20

The combination of the selected --testdebugmode and --testcalibration options is invalid.

21

The combination of --noUI with the selected --testdebugmode or --testcalibration option is invalid.

22

The selected --testprogress option is not permitted in combination with the selected --testdebugmode or --testcalibration option.

23

The test run has been completed with failed test steps.