One of our ultimate goals and a main responsibility as an independent security testing institute is the commitment of our expert knowledge in the field of IT security (and in this case the security of the Internet of Things) to increase the overall security level for everyone. For that reason we perform free product tests on a regular basis and share our findings with manufactures, customers and the public.

Now that we have published quite a number of test results here on our blog and explained the reason “Why”, it is time to shed light on the “What” and “How” we test – What are the main aspects we concentrate on and how do we evaluate the security level of these aspects.

General Focus

The general focus when testing a new device lies on three main aspects: Communication, Application Security and Data Privacy. For these three aspects we have designed several tests to evaluate security to a certain extent. The level of detail of these tests depends on whether they are part of the Quick-Check, which is more or less a hands-on security estimation or the full certification procedure, which represents a full security evaluation. The test criteria are developed and defined by our test lab, although they naturally fall into line with the definitions of other sources concerning IoT security, like e.g. the OWASP with their definitions of IoT security principles (https://www.owasp.org/index.php/Principles_of_IoT_Security). Our test criteria differ in some points (as we are for example not able of testing for problems in large scale scenarios) but apart from these our test focus pretty much covers everything the OWASP defines as important. The specific tests and test scenarios to evaluate security are completely designed and performed by our test lab and use self-developed tools as well as third-party software.

OWASP Principles of IoT Security – Except for a few points our test criteria and scenarios address all of these. (https://www.owasp.org/index.php/Principles_of_IoT_Security)

For the part of evaluating the security of the Communication we mainly concentrate on observing and analysing all out- and in-going communication initiated by the tested hardware and/or the corresponding mobile application. Especially the use and quality of encryption is revised and we check for known weaknesses and vulnerabilities, often unintentionally incorporated when implementing encrypted communication.

The term Application Security includes all tests regarding the products mobile application, which is most of the time an Android iOS app. We check this app for standard security features like code obfuscation, we inspect the application permissions and for greatest part review security relevant code segments in search of vulnerabilities and possibilities for manipulation.

The last main aspect of Data Privacy includes all points regarding the collection and processing of sensitive user data by the tested product. For this purpose we review and evaluate the manufacturer’s data privacy policies. The technical tests for this aspect heavily overlap with the tests for communication and application security as we also try to estimate the amount of collected and sent personal data by analysing the communication and application code in search for potentially unnecessary data collection.

Quick-Check

The purpose of the Quick-Check is a fast evaluation of the general security of a given product. The tests are designed to be performed within 1-2 days and cover the most relevant aspects of online-communication, application security and data privacy. The result is a representative estimation of the overall security of the given product. The depth of the performed tests for a Quick-Check is due to the given time relatively shallow but is sufficient to identify the obvious problems a product is suffering from.  The Quick-Check of course delivers no guarantee for an absolute absence of weaknesses. Having said that, the Quick-Check result of a full 3-star rating still attests the absence of obvious, easy to exploit weaknesses.

The test criteria for the Quick-Check are divided into the three main aspects Communication, Application Security and Data Privacy. Each of the three aspects contains a list of check-points which have to be (at least partially) fulfilled to earn a rating star for the certain aspect, resulting in up to 3 stars when fulfilling enough check-points for all three aspects.

The quick-check three star logo

For the three main aspects we test the following points:

Local-/Online-Communication

  • Local access adequately secured with authentication and encryption
  • Remote Access adequately secured with authentication and encryption
  • Communication between device and server adequately secured with authentication and encryption
  • No critical weaknesses and vulnerabilities found
That’s how it should be: Man-in-the-middle detected and successfully prevented.

Application Security

  • Reverse-Engineering of App effectively hindered (or at least not supported)
  • Secured local storage of user data
  • Adequate, robust and consistent Authentication
  • Effective SSL implementation and consistent use of it
  • No critical weaknesses and vulnerabilities found
With the right tools, it is even possible to take a look into secured connections.

Data Privacy

  • No unnecessary collection of personal data
  • No unsecured transmission of personal data
  • Data privacy statement available, complete, comprehensible and up-to-date

All of the mentioned points consist in most cases of several sub-points but the ultimate goal of each test is stated in the listed points.

The tools we use for the testing of the aspect Communication are mainly reference software like Wireshark for observing and analysing the ongoing connections but also self-developed and other third-party tools to “spy” onto these connections and search for possible weakspots and data leaks helpful for reverse engineering or manipulation of the communication.

The analysis of the application security is performed with quite a selection of tools – standard tools like apktool for the decoding of the application .apk, a JAVA decompiler for decompiling the JAVA code and eventually an IDE of choice to examine the code representation. In between there are many more self-developed tools as well as some more reference software like Android Device Monitor and a couple of other situational tools for viewing, analysing or adjusting certain application aspects.

The data privacy part consists of the search and analysis of the data privacy statement, which is mainly manual and the search for personal data collected and/or sent, which heavily overlaps with the tests from the application security aspect as this includes communication analysis with e.g. Wireshark and some code analysis when certain application segments arise suspicion.

Enhanced Certification Procedure

The certification procedure basically covers exactly the same points as the Quick-Check but the level of detail of the corresponding tests is greatly increased. Whereas the Quick-Check is limited to about 1-2 days and focused on finding obvious weaknesses and vulnerabilities, a full certification procedure takes about 1-2 weeks, depending on complexity of the product and the problems we are able to identify. The goal of this enhanced test procedure is basically to dig deeper and identify the more subtle and therefore more dangerous weaknesses the Quick-Check is not able to reveal.

One of our next articles will deliver a coverage of our full certification procedure, including details on refined test criteria, enhanced testing depth and meaning of the Certified Security status attested by our certificate.

The AV-TEST certifcate logo for extended testing