How to run a program from Python: how to run a system command or another Python script (complete guide)

Table of Contents

1. The subprocess module in Python

2. How to pass arguments to a program in subprocess.run

3. Getting the result of command execution (standard output, stdout))

4. Working with the standard error (stderr)

5. Automatic command splitting into a list of arguments

6. How to run a command with wildcards

7. Calling a command without breaking it into a list of arguments. Running a command in the shell. The shell=True option

8. How to make subprocess.run not wait for the command to complete. Putting a running process into the background

9. How to run another Python script from a Python script

10. How to terminate a running application by timeout

11. subprocess.Popen is an advanced version of subprocess.run

12. Errors when using subprocess.run

12.1 NameError: name 'subprocess' is not defined. Did you forget to import 'subprocess'?

12.2 FileNotFoundError: [Errno 2] No such file or directory

12.3 subprocess.TimeoutExpired: Command … timed out

The subprocess module in Python

The subprocess module allows you to create new processes. With subprocess, you can, for example, execute any Linux or Windows commands from a script. You can also run another Python script using Python.

With subprocess, you can:

• run an operating system program or another Python script
• pass any number of arguments to the program being run
• connect to standard input/output/error streams and get the return code
• get the output of the running command
• check that the command was executed without errors

The subprocess module is included in the standard Python packages and does not require installation.

The subprocess.run function is the main way to work with the subprocess module.

The simplest way to use the function is to run it like this:

import subprocess

result = subprocess.run('ls')

The result of the command is output to standard output – this can be changed, below is how to do it.

The result variable now contains a special CompletedProcess object. From this object, you can get information about the process execution, such as the return code:

print(result.returncode)

Code 0 means that the program was executed successfully.

How to pass arguments to a program in subprocess.run

If you try to specify any arguments to the program being launched, you will encounter an error. For example, the following is an INCORRECT way of passing arguments:

import subprocess

result = subprocess.run('ls -ls')

When executing this script, the error “FileNotFoundError: [Errno 2] No such file or directory” will occur.

If you need to call a command with arguments, you need to pass it this way (as a list):

import subprocess

result = subprocess.run(['ls', '-ls'])

Getting the result of command execution (standard output, stdout))

By default, the run function returns the result of command execution to the standard output.

If you want to change this default behavior, namely, save the command output to a variable instead of immediately displaying it on the screen, you need to add the stdout argument and set it to subprocess.PIPE.

Since in this case a byte stream will be returned rather than a string, you need to decode it. This can be done with a separate command, but it is more convenient to specify the encoding='utf-8' option to automatically decode the bytes into a string:

import subprocess

result = subprocess.run(['ls', '-ls'], stdout=subprocess.PIPE, encoding='utf-8')
print(result.stdout)

If you did not specify the encoding='utf-8' option, then to decode the received bytes, you need to run decode:

import subprocess

result = subprocess.run(['ls', '-ls'], stdout=subprocess.PIPE)
print(result.stdout.decode('utf-8'))

Working with the standard error (stderr)

If the command was executed with an error or did not work correctly, the command output will go to the standard error stream.

You can get this output in the same way as the standard output stream:

import subprocess

result = subprocess.run(['ping', '-c', '3', '-n', 'a'], stderr=subprocess.PIPE, encoding='utf-8')

print(result.stdout)
print(result.stderr)
print(result.returncode)

The following will be output:

None
ping: a: Name or service not known

2

Result.stdout (what the program should output) now contains only an empty string (None); result.stderr contains the standard error output stream; and result.returncode contains the error code

Automatic command splitting into a list of arguments

I think manually splitting a ready-made command line into arguments is not the most interesting activity. Besides, if you are used to the convenient command launch in other programming languages, when you can specify the command and its arguments in one line, then manually splitting the command into arguments for subprocess.run seems like unnecessary work. Moreover, there are some tricky cases where it is not always obvious how to split the command line into arguments, for example:

/bin/vikings -input eggs.txt -output "spam spam.txt" -cmd "echo '$MONEY'"

You can use automatic splitting of the command with arguments into a list that needs to be passed to subprocess.run. Example:

import shlex, subprocess

command_line = 'ls -ls'
args = shlex.split(command_line)
result = subprocess.run(args)

In this script, the full command, along with arguments, is stored in the variable command_line. Then shlex.split splits the command into arguments and stores the resulting list in the variable args. This list is then passed to subprocess.run to run the command.

How to run a command with wildcards

When trying to run a command using wildcard expressions, such as *:

import subprocess

result = subprocess.run(['ls', '-ls', '*php'])

An error will occur:

ls: cannot access '*php': No such file or directory

To call commands that use wildcard expressions, you need to add the shell=True argument and call the command like this:

import subprocess

result = subprocess.run('ls -ls *php', shell=True)

Calling a command without breaking it into a list of arguments. Running a command in the shell. The shell=True option

You may have noticed that the previous command did not need to be broken into arguments – it was passed entirely to subprocess.run. This was made possible by the shell=True option, which means that the specified command should be executed in the shell.

That is, you can use the shell=True option and you no longer have to split the command into a list of arguments:

import subprocess

result = subprocess.run('ls -ls', shell=True)

See also: https://docs.python.org/3/library/subprocess.html#security-considerations

How to make subprocess.run not wait for the command to complete. Putting a running process into the background

Another feature of the run() function is that it waits for the command to complete. If you try, for example, to run the ping command, this aspect will be noticeable:

import subprocess

result = subprocess.run(['ping', '-c', '3', '-n', '8.8.8.8'])

You can put a running process into the background. To do this, you need to:

1) use the shell=True argument

2) write the command line without breaking it into arguments

3) add the “&” symbol at the end of the command after a space

For example:

import subprocess

result = subprocess.run('ping -c 3 -n 8.8.8.8 &', shell=True)
print('It has been run!')

In this case, the message "It has been run!" will be displayed before the ping command is run.

How to run another Python script from a Python script

In general, running a Python program from another Python program is similar to the process of running an OS program.

Example:

import subprocess

result = subprocess.run(['python', 't37.py'])

In this case, the Python script located in the t37.py file will be launched — this file is located in the same directory as the program that launches it, otherwise you need to specify the full path, for example:

import subprocess

result = subprocess.run(['python', '/home/mial/test/p/t37.py'])

And another example with the shell=True option:

import subprocess

result = subprocess.run('python t37.py', shell=True)

If you have met the conditions that allow you to run a script directly in the shell without explicitly specifying the interpreter (for this, the file must be executable, and the shebang must be specified at the beginning of the file), then you can launch a Python script from another Python program as follows:

import subprocess

result = subprocess.run('/home/mial/test/p/t37.py')

And another option with the shell=True option:

import subprocess

result = subprocess.run('/home/mial/test/p/t37.py', shell=True)

How to terminate a running application by timeout

Using the timeout option, you can specify the execution time of the running program, after which the application will be terminated forcibly and an error will be displayed. The time is specified in seconds. Example:

import subprocess

result = subprocess.run('ping -c 3 -n 8.8.8.8', shell=True, timeout=1)

You can trap error messages with the try…except construct:

import subprocess

try:
	result = subprocess.run('ping -c 1 -n 8.8.8.7', shell=True, timeout=1)
except (subprocess.TimeoutExpired):
	print("Time is over")

subprocess.Popen is an advanced version of subprocess.run

You can also use subprocess.Popen, where the creation and management of the underlying process in this module is handled by the Popen class. It provides more flexibility so that developers can handle less common cases not covered by the subprocess.run functions.

For example, with subprocess.Popen you can run programs as a different user.

See also: https://docs.python.org/3/library/subprocess.html#subprocess.Popen

Errors when using subprocess.run

NameError: name 'subprocess' is not defined. Did you forget to import 'subprocess'?

Full error text:

Traceback (most recent call last):
  File "/home/mial/test/p/t38.py", line 2, in <module>
    result = subprocess.run('ls -ls')
             ^^^^^^^^^^
NameError: name 'subprocess' is not defined. Did you forget to import 'subprocess'?

This error means that you have not imported the subprocess module. That is, at the beginning of your script you need to add the line:

import subprocess

FileNotFoundError: [Errno 2] No such file or directory

Full error text:

Traceback (most recent call last):
  File "/home/mial/test/p/t38.py", line 2, in <module>
    result = subprocess.run('ls -ls')
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.12/subprocess.py", line 1955, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'ls -ls'

This error can occur if you pass a single command line instead of a list of arguments. As an alternative to the list of arguments, you can use the "shell=True" option.

subprocess.TimeoutExpired: Command … timed out

Full error text:

Traceback (most recent call last):
  File "/home/mial/test/p/t38.py", line 2, in <module>
    result = subprocess.run('ping -c 3 -n 8.8.8.8', shell=True, timeout=1)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 2141, in _communicate
    self.wait(timeout=self._remaining_time(endtime))
  File "/usr/lib/python3.12/subprocess.py", line 1264, in wait
    return self._wait(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 2045, in _wait
    raise TimeoutExpired(self.args, timeout)
subprocess.TimeoutExpired: Command 'ping -c 3 -n 8.8.8.8' timed out after 0.9999772470036987 seconds

This error occurs when the timeout expires, that is, the time allocated for the execution of the program being launched. If you need to use a timeout, but you do not want your script to terminate due to this error, use the try…except construct.