As you may have noticed in the last days and weeks, there is currently a security hole in the Citrix products Citrix ADC and Netscaler Gateway in different versions. In this article I describe how you can protect your system at least according to the current state of the security hole. I also show you how you can detect if your system is already compromised. Furthermore you will find in this article information when the planned updates will be released by Citrix.
In this case the vulnerability is known since December 17th 2019, and since January 10th 2020 POC codes and instructions have been published on several platforms to exploit the vulnerability. In these instructions you can even find examples of various malicious code.
So, it´s high time to close the door!
CVE-2019-19781 – what is it?
CVE-2019-19781 is a vulnerability. This means that a vulnerability has been found on the affected system. CVE-2019-19781 vulnerability allows an attacker to execute arbitrary code without authentication. This makes it possible to load further malicious code onto the system or to have it executed.
Which platforms are currently affected by the vulnerability?
The vulnerability affects all supported product versions platforms:
• Citrix ADC and Citrix Gateway version 13.0 all supported builds
• Citrix ADC and NetScaler Gateway version 12.1 all supported builds
• Citrix ADC and NetScaler Gateway version 12.0 all supported builds
• Citrix ADC and NetScaler Gateway version 11.1 all supported builds
• Citrix NetScaler ADC and NetScaler Gateway version 10.5 all supported builds
Why is the security breach so dangerous?
Currently Citrix has not yet released an official firmware patch that finally fixes the problem. Currently there is only one workaround. Another problem is that there are always new vulnerabilities being found which require the workaround to be extended. In addition, it is possible that the affected system was successfully protected against the vulnerability from the outside with the workaround, but there were already attackers on the system before that and, for example, have placed a so-called backdoor. This would make it possible to continue to exploit the vulnerability and execute malicious code on the system despite a successful patch.
When are the security updates from Citrix planned?
Currently Citrix is planning to fix all known vulnerabilities before the end of January 2020. More detailed information could be found in the table below.
| Version | Refresh Build | Expected Release Date |
| 10.5 | 10.5.70.x | 31st January 2020 |
| 11.1 | 11.1.63.x | 20th January 2020 |
| 12.0 | 12.0.63.x | 20th January 2020 |
| 12.1 | 12.1.55.x | 27th January 2020 |
| 13.0 | 13.0.47.x | 27th January 2020 |
What is the current workaround?
As a workaround, Citrix has published the steps for the respective product configuration under the article CTX267679. Here you can find the article under the following link:
https://support.citrix.com/article/CTX267679
For the individual systems you have to do the following steps. I have taken over the points from the Citrix article, and there is already another responder policy in place for a reported vulnerability. The additional code required for the second vulnerability can be implemented together with the first policy. It is only important to me at this point to let you know why my code differs from the Citrix version.
You can either import the code via the Web GUI using the provided CLI or directly via Putty.
This is the code that should also be implemented:
add responder policy ctx267027_2 "HTTP.REQ.HEADER(\"NSC_USER\").CONTAINS(\"/../\") || HTTP.REQ.HEADER(\"NSC_NONCE\").CONTAINS(\".pl\") " respondwith403 bind responder global ctx267027_2 2 END -type REQ_OVERRIDE
Now that this has been clarified, we can go into the individual configurations.
Standalone System
Run the following commands from the command line interface of the appliance to create a responder action and policy:
enable ns feature responder add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\"" add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\"))" respondwith403 bind responder global ctx267027 1 END -type REQ_OVERRIDE add responder policy ctx267027_2 "HTTP.REQ.HEADER(\"NSC_USER\").CONTAINS(\"/../\") || HTTP.REQ.HEADER(\"NSC_NONCE\").CONTAINS(\".pl\") " respondwith403 bind responder global ctx267027_2 2 END -type REQ_OVERRIDE save config
Ensure that the changes apply to the management interfaces as well. From the command line interface, please run the following commands and make a reboot.
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot
Citrix ADC HA Pair
On the primary HA node:
enable ns feature responder add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\"" add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\"))" respondwith403 bind responder global ctx267027 1 END -type REQ_OVERRIDE add responder policy ctx267027_2 "HTTP.REQ.HEADER(\"NSC_USER\").CONTAINS(\"/../\") || HTTP.REQ.HEADER(\"NSC_NONCE\").CONTAINS(\".pl\") " respondwith403 bind responder global ctx267027_2 2 END -type REQ_OVERRIDE save config shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot
When the primary machine is shut down, the HA cluster is swung onto the secondary machine.
Let’s now continue with the secondary machine:
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot
Cluster
On CLIP:
enable ns feature responder add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\"" add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\"))" respondwith403 bind responder global ctx267027 1 END -type REQ_OVERRIDE add responder policy ctx267027_2 "HTTP.REQ.HEADER(\"NSC_USER\").CONTAINS(\"/../\") || HTTP.REQ.HEADER(\"NSC_NONCE\").CONTAINS(\".pl\") " respondwith403 bind responder global ctx267027_2 2 END -type REQ_OVERRIDE save config shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot
Each cluster node:
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot
If you use Admin partition:
switch ns partition default enable ns feature responder add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\"" add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\"))" respondwith403 bind responder global ctx267027 1 END -type REQ_OVERRIDE add responder policy ctx267027_2 "HTTP.REQ.HEADER(\"NSC_USER\").CONTAINS(\"/../\") || HTTP.REQ.HEADER(\"NSC_NONCE\").CONTAINS(\".pl\") " respondwith403 bind responder global ctx267027_2 2 END -type REQ_OVERRIDE save config shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot
If you have followed these steps, at least the currently known vulnerability CVE-2019-19781 is closed.
How can I undo this?
In case something does not work, you can undo the changes with the following adjustments:
unbind responder global ctx267027 unbind responder global ctx267027_2 rm responder policy ctx267027 rm responder policy ctx267027_2 rm responder action respondwith403 save config
Remove nsapi command from rc.netscaler. (Below command will search rc.netscaler file for the below pattern and remove the line that was originally added)
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=1 shell "sed -i '' '/skip_systemaccess_policyeval=0/d' /nsconfig/rc.netscaler" reboot
The reboot, in each of the scenarios above, is not necessary to apply the policy but it is a precautionary step to ensure that if there are any open sessions,obtained via the vulnerability prior to policy application, are cleared.
Please also note that there may be a conflict with the priority of the policy if you are already using responder policies.
How can the vulnerability be audited externally CVE-2019-19781?
You can check the vulnerability externally using Python, for example.
Install Python on your computer.
Here you can find the download link: https://www.python.org/downloads/windows/
Instructions for the installation of Python: https://docs.python.org/3/using/windows.html
You can then use the following POC Code to check whether the vulnerability is closed:https://github.com/trustedsec/cve-2019-19781
If the vulnerability was successfully closed, the tool’s feedback should look like this.
The best workaround does not help if the attacker was already inside!
Here are some links to find out if the vulnerability has been exploited on your system and if your system is compromised.
Under this link you will find for example very good hints to check if there are hints for a compromise on your system:
https://www.trustedsec.com/blog/netscaler-remote-code-execution-forensics/
Good Site for more Informations about the CVE-2019-19781:
https://www.fireeye.com/blog/products-and-services/2020/01/rough-patch-promise-it-will-be-200-ok.html
Link to the National Vulnerability Database:
https://nvd.nist.gov/vuln/detail/CVE-2019-19781
Link to “Bundesamt für Sicherheit in der Informatik” of Germany (BSI):
https://www.bsi.bund.de/SharedDocs/Warnmeldungen/DE/CB/2020/01/warnmeldung_cb-k19-1093_update_2.html
Here you can find an interactive map of the vulnerable Systems:
https://docs.google.com/spreadsheets/d/1sJ8-cVyG4vFYq6-MGdapM1eAhUkoXIKlsmuzKdfD9Ys/edit#gid=1779255426
What measures can they take if they find that a break-in has already been successful?
- Disconnect network connections of the Netscaler
- Rif it is a vpx appliance, you can restore a backup before 12/17/2019, but you cannot be 100% sure that the vulnerability was not exploited before. When restoring the machine,disable NAT externally in advance to avoid the next attack
- The safest thing to do is to reinstall the appliance (same MAC, otherwise the license has to be recreated) and reload the ns.conf there. BUT: Please check the config in advance, because it is possible that it has been manipulated and there is a backdoor there
- Revoke and recreate the certificates
- Then you do the workaraound
- Changes the passwords
- Since it is possible that the private keys were stolen, users should change their passwords
- Reopen the network to the outside and keep an eye on the response policy hits
If you want to be informed about security vulnerabilities in the future, you can register via https://support.citrix.com/user/alerts