Troubleshooting#

General comments#

When a problem occurs, the first thing to do is to check the traces left by the software. In particular, the shell from where you launched sphinx may show valuable pieces of information. To increase the verbosity, just launch sphinx with the option --log-level dbg.

Another rich source of logs is ulogcat. It exposes logs generated by the simulated firmwares and by firmwared. To launch it with colors and timestamps:

$ sudo ulogcat -C -vlong

You can also activate some extra ulog logs from firmwared by doing this:

$ echo 'FIRMWARED_VERBOSE_HOOK_SCRIPTS = "y"' | sudo tee /etc/firmwared.conf
$ sudo systemctl restart firmwared

If none of the suggested solutions below works out, an issue may be worth being reported. Beforehand, make sure that:

  • you are using the latest release of Parrot Sphinx

  • you are respecting the minimal system requirements

  • in the description of the issue, please provide:

    • your Linux distribution

    • the kernel version

    • GPU card model and its driver version

    • the simulated firmware version

Suddenly, the UE4 application systematically fails at startup#

X Error of failed request: BadValue (integer parameter out of range for operation)
Major opcode of failed request: 155 (GLX)
Minor opcode of failed request: 3 (X_GLXCreateContext)
Value in failed request: 0x0
Serial number of failed request: 106
Current serial number in output stream: 107
terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument
Signal 6 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554
CommonUnixCrashHandler: Signal=6
Segmentation fault (core dumped)

This kind of error is generally due to an automatic GPU driver update. The solution just consists in rebooting your workstation.

The UE4 application crashes with a Vulkan memory error#

If you see something similar to the log below in the shell output of the UE4 application, it looks like the GPU RAM was not large enough to support the need.

LogCore: Fatal error: File:<...>/UnrealEngine/Engine/Source/Runtime/VulkanRHI/Private/VulkanMemory.cpp [Line: 313]
                      Out of Device Memory, Requested=48000.00Kb MemTypeIndex=7

In that case, here are some points to be checked:

  • One or more external applications may be running in background and demanding much GPU RAM. To check, you can run:

    $ nvidia-smi

  • the HMI window size is too large and/or you are using a 4K screen. In that case, reduce as much as possible the size of the main window. Another solution is to switch to another less-consuming UE4 application, which means to a simpler world.

I have weird visuals in the UE4 application#

If you see something similar to the image below when running the UE4 application, it looks like Unreal doesn’t recognize your graphics card.

_images/bad_visuals.jpg

In that case, here are some points to be checked:

  • Your pc is not using the correct graphics card. To check, you can run:

prime-select query

This command should return nvidia.

If it is not the case, you need to run this command:

prime-select nvidia

  • The secure boot is enabled, and you need to disable it. To do so, you need to restart your pc, open the bios and disable the secure boot.

I cannot take photos/videos from Freeflight 7#

If you are using a real drone as a Wi-Fi adapter, you might have forgotten the set the option --model anafi_ai when calling the configuration script sphinx-drone-as-Wi-Fidongle.

The simulation is too slow#

As this topic is a complex one, a section has been created HERE.

sphinx exits with the error ‘Connection with firmwared timed out’#

The systemd service firmwared is probably not working. To confirm this assumption, enter the following command:

$ fdc ping
PONG

You should get a PONG in response if firmwared is alive. Otherwise, check the systemd status:

$ systemctl status firmwared.service

Some questions that could help:

  • Have you restarted your workstation after installing Parrot Sphinx?

  • Does the current user belong to the group firmwared?

    $ cat /etc/group |grep $(whoami)

  • Have you disabled the UEFI secure boot?

I have no Internet access while a simulation is running#

If you start Parrot Sphinx with the option wifi_iface=xxxx, the Wi-Fi interface used by your workstation to access the Internet can be stolen and used by the simulated firmware.

If the workstation does not have any other network interface (like Ethernet) to connect to the Internet, running a simulation will result in losing connection with the Internet.

If you do not need to connect to a remote controller via Wi-Fi, a solution is to set the option wifi_iface to an empty string.

Unable to connect the drone via the virtual ethernet interface#

While the simulation is running, ifconfig should show an interface ‘fd_vethx’ up:

$ ifconfig

fd_veth0  Link encap:Ethernet  HWaddr 66:a8:ad:06:b2:6b
          inet addr:10.202.0.254  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::64a8:adff:fe06:b26b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:422 (422.0 B)  TX bytes:534 (534.0 B)

If this is not the case, the network-manager coming with your Linux distribution may interfere with sphinx. A solution consists in disabling the network-manager while you are using Parrot Sphinx.

Sphinx has crashed leaving behind a running firmware#

Despite our best effort, in rare circumstances, Parrot Sphinx may crash leaving behind some zombie firmwares. In such scenario, those running firmwares may use system resources preventing you from launching any new simulation. When it’s time to clean up everything use the following command:

$ fdc drop_all instances

This command will kill any running firmware instance and unmount all firmware instances.

During the simulation the drone acts erratically, gets uncontrollable and/or falls.#

The reasons for that issue are likely to be numerous. We are going to focus on the most relevant ones.

  • The real time execution is disturbed by another process. It is highly recommended to avoid running any CPU/GPU consuming task while sphinx is on.

  • On some laptop, the energy saving mode can lead to some disturbances. In that case, check if a kernel module called “intel_powerclamp” is present. If yes, remove it:

$ sudo rmmod intel_powerclamp

To inhibit it once for good:

$ echo "blacklist intel_powerclamp" > /etc/modprobe.d/disable-powerclamp.conf

My Wi-Fi interface got its name changed after exiting the simulation#

To workaround it and set the Wi-Fi interface name once for good, proceed as follows:

  • Create a new udev rule file:

    • /etc/udev/rules.d/76-netnames.rules for Debian 9+

    • /etc/udev/rules.d/70-persistent-net.rules for other distributions

  • Add and adapt the following instruction:

    SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="<WiFi_card_MAC>", NAME="wlan0"

    Replace <WiFi_card_MAC> by the Wi-Fi MAC address. You can find it easily with the ip addr command.

    Important: Wi-Fi MAC address needs to be written in case-sensitive way, meaning the same way as provided by ip addr

  • Deactivate/reactivate the Wi-Fi card so that the changes are taken into account by udev.

That way, the Wi-Fi interface Wi-Fi always has the name wlan0 from now on.

ulog does not work#

“ulog” log system relies on a non-signed kernel module, named “ulogger”. When starting, Sphinx makes sure the module is loaded. The loading may fails if your workstation has the “UEFI Secure boot” enabled.

In that case, the only solution is to disable the secure boot in the UEFI menu.