PPPP API: what I know

While analyzing security of iMiniCam app, I learned some part of UDP protocol used to communicate with the camera. Here is quick bunch of facts.

UDP layer

Port 32100 (Proto_Hello, to external server) and port 32108 (others, to camera).

Type-length layer

Following structure represents first header that is always found in datagram.

struct header {
  uint8_t magic;
  enum proto_type type;
  uint16_t length;
}

In this structure magic is always set to 0xf1, type is one of the below values, packed on single byte and length count data that follows this packet, so if it is equal to zero, only those four bytes are transmitted.

Packet types

These are all the types present in binary, I was reverse engineering. Names are derived from function names that crafts their content, so i.e. for Proto_DevLgn that is equal to 0x10 there exists function PPPP_Proto_Write_DevLgn, PPPP_Proto_Read_DevLgn and PPPP_Proto_Send_DevLgn.

enum proto_type {
  Proto_Hello = 0;
  Proto_HelloAck = 1;
  Proto_HelloTo = 2;
  Proto_HelloToAck = 3;
  Proto_QueryDID = 8;
  Proto_QueryDIDAck = 9;
  Proto_DevLgn = 0x10;
  Proto_DevLgnAck = 0x11;
  Proto_DevLgn_CRC = 0x12;
  Proto_DevLgn1_CRC = 0x13;
  Proto_P2PReq = 0x20;
  Proto_P2PReqAck = 0x21;
  Proto_LanSerch = 0x30;
  Proto_PunchTo = 0x40;
  Proto_PunchPkt = 0x41;
  Proto_P2PRdy = 0x42;
  Proto_RSLgn = 0x60;
  Proto_RSLgnAck = 0x61;
  Proto_RSLgn1 = 0x62;
  Proto_RSLgn1Ack = 0x63;
  Proto_ListReq1 = 0x67;
  Proto_ListReq = 0x68;
  Proto_ListReqAck = 0x69;
  Proto_RlyHello = 0x70;
  Proto_RlyHelloAck = 0x71;
  Proto_RlyPort = 0x72;
  Proto_RlyPortAck = 0x73;
  Proto_ByteCount = 0x78;
  Proto_RlyReq = 0x80;
  Proto_RlyReqAck = 0x81;
  Proto_RlyTo = 0x82;
  Proto_RlyPkt = 0x83;
  Proto_RlyRdy = 0x84;
  Proto_SDevRun = 0x90;
  Proto_SDevLgn = 0x91;
  Proto_SDevLgn_CRC = 0x92;
  Proto_DRWAck = 0xd1;
  Proto_PSR = 0xd8;
  Proto_Alive = 0xe0;
  Proto_AliveAck = 0xe1;
  Proto_Close = 0xf0;
  Proto_MGMDumpLoginDID = 0xf4;
  Proto_MGMDumpLoginDIDDetail = 0xf5;
  Proto_MGMDumpLoginDID1 = 0xf6;
  Proto_MGMLogControl = 0xf7;
  Proto_MGMRemoteManagement = 0xf8;
}

Proto_Hello

This packet’s content is empty (length=0). Sent to external server. Response is Proto_HelloAck.

Proto_HelloAck

Length is 0x10. Content is according to following structure:

struct HelloAck {
  struct sockaddr_in wan_addr;
}

Final word

I hope this very quick introduction will help someone and will lead to documenting the protocol, in enough part, that it will be possible to develop open source client application able to communicate with the cameras using it.

Security analysis of spy camera sold by Chinese suppliers: iMiniCam app

Lately, I bought some random, noname spy camera from usual Chinese sources. Just after unboxing, one detail seemed a bit suspicious to me: together with the camera there comes a link to Android app, that is meant to control the camera. What is worse, the link leads to some Chinese app shop, obviously with only one language available (notice the version – 1.0.33, as it is not the only one in the wild):

http://m.app.so.com/detail/index?pname=com.hbwy.fan.iminicams&id=3646142

As I’ve already seen apps that i.e. calls home, despite having no need to open a single socket and then sending data like IMEI, IMSI, MAC addresses and so on, red lamp turned on and I said to myself: stop. Let’s reverse engineer the thing first!

OSINT

Before I begun my reverse engineering, I attempted to ask uncle Google (and his friends πŸ™‚ ) to gain some knowledge. Usual virustotal gives me this:

Results of virustotal scan of original link (click for details)

Not bad for such a simple application. Searching other shops for the same name leaded to exactly the same application (shared objects included matches), but older version (1.0.25) and in their case virustotal stopped complaining. For me there are two possible reason for that:

  1. Google Play app is not supported by vendor anymore
  2. There is special edition for Chinese, with few surprises

In favor of the second option is the fact that Java code of only the China edition is heavily bloated and usual decompilers cannot reverse the code properly, so somebody tried to make reverse engineer’s life harder. In apk downloaded from downloadapk.net and apkplz.com, everything seem to be fine.

Below are the links, I’ve used in comparison:

  1. Google Play: https://play.google.com/store/apps/details?id=com.hbwy.fan.iminicams (1.0.25)
  2. downloadapk.net: https://downloadapk.net/down_IPLiveCam.html (1.0.25)
  3. apkplz.com: https://apkplz.com/android-apps/iplivecam-apk-download (1.0.25)

Just for completeness, below is the name and webpage of suspected vendor of the stuff, I found accidentally: Albert Technology Co., Ltd. Shenzhenshihong Bo, and their webpage is here: http://www.honest-tech.com/en/

Let’s reverse then

First thing,I’ve noticed after decompilation of APK’s Java classes is:

  public static String a(String paramString)
  {
    if (paramString == null)
      return "";
    if (paramString.length() == 0)
      return "";
    paramString = new BigInteger(paramString.getBytes());
    return new BigInteger("0933910847463829232312312").xor(paramString).toString(16);
  }

  public static String b(String paramString)
  {
    if (paramString == null)
      return "";
    if (paramString.length() == 0)
      return "";
    BigInteger localBigInteger = new BigInteger("0933910847463829232312312");
    try
    {
      paramString = new String(new BigInteger(paramString, 16).xor(localBigInteger).toByteArray());
      return paramString;
    }
    catch (Exception paramString)
    {
    }
    return "";
  }

So, it’s gonna be interesting… As it turns out this one seem to be unused. But, after going a bit further and searching for all hardcoded strings I found this:

cfans/ufo/sdk/d/a.java:37:    PPPP_APIs.PPPP_Initialize("EBGDEIBIKEJOGCJNEBGBFDEOHJNDHANKGEFCBNCJAPJILDLIDEABCLOKGMLNJBKDALMNLKDOONMGAFCJJKNHJN".getBytes());

PPPP is shared object linked to the app via NDK, so I started reversing this libPPPP_API.so. Not so long time later, I was able to see decoded string by running the library and connecting gdb to it:

(gdb) p (char*)$sp+0xc
$16 = 0xbefff78c "121.40.184.33,120.25.151.105,120.25.157.75,"

I’ve check the addresses in whois database and all of them leads to Hangzhou, Zhejiang, China.

Next to decoding of these addresses, there is UDP socket opening routine and at least two ports are used: 32100 and 32108, where first one is used to connect to these services (all three of them) and port 32108 seem to be the port of camera.

What is my IP address?

So far, I identified at least one request that for sure goes to the hardcoded server, so in fact it is used to call home. This is the so called hello request. It sends nothing at all, as can be seen below:

00000000  f1 00 00 00                                       |....|
00000004

[len] [magic] [typ]

And in response it gets packet like that:

00000000  f1 01 00 10 00 02 7b 3d  xx xx xx xx 00 00 00 00  |......{=........|
00000010  00 00 00 00                                       |....|
00000014

[len] [magic] [sockaddr_in] [typ]

Content of this packet is then cast to sockaddr_in packet and experimentally, I was able to tell that it contains WAN address of requesting side (this is the censored part). I haven’t noticed the original sockaddr_in, that was used to send this hello message in memory, so chances are this is the only purpose of that server.

However, there are a lot of other datagrams crafted from same function as this one (including so called P2P requests), so I cannot guarantee I haven’t missed anything. In my opinion, further dynamic analysis is required. And I want to do one either way.

Other indicators?

Beside these suspicious addresses, I noticed some ioctl usage, but this seem to be quite legit. It just checks configuration of network interfaces (the worst that can happen out of this is sending somewhere MAC address of all interfaces available, that is both WiFi and cellular).

I focused my analysis on libPPPP_API, but beside that, there exists also UFO SDK and Adpcm codec. While Adpcm can be considered safe, as it does not import anything weird, UFO SDK is suspicious, just like PPPP API. Maybe little less, as it does not seem to obfuscate any strings. Nevertheless, it opens some sockets,probably only for RTP, but still.

Conclusion

The most important recommendation is to never trust any source of APKs beside official shops, most notably Google Play, as in it there exist some administration staff, that can react in case of obvious malware (how often and in which cases they do is another story). As can be seen in this case, either app vendor, or a shop is attaching some stuff, which is recognized as malware. In case of Google Play variant (and other Play-cloning shops), things are much better and no alert is raised by VirusTotal.

Second point is how to treat the fact of calling home, especially in situation where app vendor tries to hide the fact by obfuscating the string that contains the addresses. In my opinion, it is enough to not let that app do anything outside some closed network (with no means of sending packet to the internet), at least to be able to check the behavior in practice. And this is probably what I am going to try next. However, I can imagine that I will be using the app, despite that, but after banning these three servers. I can also understand that most people will ignore that fact (or even more – will be unaware of that), as in today’s world there is such a huge amount of services that are gathering our data, that it is impossible to block them all.

To sum up, if you want to use it, download from Google Play and keep in mind, some data might leak somewhere on he Chinese coast.

PS: I am not planning to publish complete API, which I am in theory able to reverse engineer. However, expect some quick summary of what I already know about the protocl πŸ™‚

How to convert Android resource ID back into its name

Few weeks ago I made an attempt to reverse engineer some obscure Android APK. It was available only through some Chinese shop, obviously described in only one language there. Unfortunately, it turned out that every tool designed for reverse engineering APK files outputted source with mysterious resource IDs, as plain integers, which is not the most convenient way to read them. Therefore I started looking for any way to find some meaningful name from these ids. At the end of my development effort I found out, there is one file that usually might be used for that purpose – res/values/public.xml, as produced by apktool (if I remember correctly). However, according to its name it contains only public resources, so some of them are missing there (in my case at least some drawable type resources were missing). Therefore, I am publishing my program to do things even more reliably.

arscutils

This program requires my library created together, but which is separate project – libarsc. It is available, as usually through Github and also as a package to be downloaded from PyPI. Just type:

pip install libarsc

with proper privileges.

This is meant to be utility package, but for now it contains only one such tool: rid2name. Its purpose is to convert resource ID into name in format matching the one, programmers use in their Android apps. Therefore with its help it should be possible to make reversed program looks more similar to compiler input on the developer side. To use it, just feed it with resources.arsc file as first parameter, resource id as second one and optionally one of: fqdn, xmlid or json as third one. As a result you should get resource name as used in Java source, XML files or JSON meant for further processing. Example runs are:

$ python3 rid2name.py ../com.g_zhang.iMiniCam_39/original/resources.arsc 0x7f070000 xmlid
@com.g_zhang.iMiniCam:string/app_name
$ python3 rid2name.py ../com.g_zhang.iMiniCam_39/original/resources.arsc 0x7f070000 fqdn
com.g_zhang.iMiniCam.R.string.app_name
$ python3 rid2name.py ../com.g_zhang.iMiniCam_39/original/resources.arsc 0x7f070000 json
{"package": "com.g_zhang.iMiniCam", "type": "string", "key": "app_name"}
$ python3 rid2name.py ../com.g_zhang.iMiniCam_39/original/resources.arsc 0x7f070000
com.g_zhang.iMiniCam.R.string.app_name

There is also quite convenient interface inside Python source, so the file should be includable into bigger projects.

I have to give one warning now: my implementation of ARSC format is not complete, so some things might not work as expected, but from my tests of libarsc, out of 12 ARSC files, extracted from random APK files, I found on my phone, 3 of them failed (returned different MD5) to rebuild into exactly same binary (did not checked exactly what happened there).

libarsc

This is library that was used underneath arscutils. It is able to parse most of the ARSC file, with special treatment of naming part, that allowed creation of rid2name. It is still missing some important parts and if there will be need from my side to extracting some more things, I will implement the rest of the specification. I am also open to any pull requests to my Github repo.

Future

As you might noticed in usage listing, there is a topic of reverse engineering app, which name was shown there. In case I found something interesting inside, there will be another article, where I will try to share my findings.

Edit: my mistake, this is not my target app, just the package name was similar.

Setting up new v3 Hidden Service with ultimate security: Part 4: Installing client certificates to Firefox for Android

This post is a part of Tor v3 tutorial. Other parts are:

  1. Hidden Service setup
  2. PKI and TLS
  3. Client Authentication
  4. Installing client certificates to Firefox for Android

As we now have Hidden Service, requiring clients to authenticate themselves with proper certificate, it would be great to be able to use Android device to access the service. As I shown before, on desktop Firefox it was quite trivial. Unfortunately, things are different on Android. Mobile Firefox does not have any interface for adding any certificates. Furthermore, unlike Chrome, it does not use default Android certificate vault, providing it own instead. On the other hand, under the hood it is more or less the same Firefox, so the support itself is present. Therefore, we need to hack into Firefox internal databases and add the certificate there. In this part, I will show, how to do that.

Caution: similarly to desktop browser, you should not add any random certificates to your main browser. It is even worse idea to do the same with Orfox, as it might allow attackers to reveal your identity. Newer Androids have ability to create user accounts, furthermore Firefox has profiles features, just like on desktop, but harder to use. If you want to do, what is described here, separating this configuration from any other is first thing to do.

Installing CA certificate

Before we do that with user certificate, let’s start with CA. It is way easier, as Firefox has convenient feature allowing to install certificates by browsing them. All we need to provide is a valid MIME type – application/x-x509-ca-cert. So, all we need is some webserver, which we will configure to treat files with extension .crt to be treated as mentioned type. Just after opening certificate file, Firefox should ask if you are sure about adding the certificate and allow you to choose for what purpose it will be used. It also allows to view the certificate to make sure, it is the one we intended to add.

At first, check the certificate
Then use it only for website identification

In theory there is very similar MIME for user certs – application/x-x509-user-cert, but for some reason, what Firefox says after opening this type of file is:

“Couldn’t install because the certificate file couldn’t be read”

And the same effect is, no matter if the file is password protected or not.

Installing client certificate

  1. Go to /data/data/org.mozilla.firefox/files/mozilla on Android device (root required)
  2. Locate default Firefox profile. If there is only one directory in format [bloat].profile, this is it. If not, file profiles.ini should contain only one profile with Default=1. This is what we are looking for
  3. Download files cert9.db and key4.db to Linux machine
  4. Use pk12util to insert certificate into database:
$ pk12util -i [filename].p12 -d.
Enter password for PKCS12 file:
pk12util: no nickname for cert in PKCS12 file.
pk12util: using nickname: [email] - r4pt0r Test Systems
pk12util: PKCS12 IMPORT SUCCESSFUL
  1. Upload files back to Android. Make sure Firefox is not running
  2. Test it by opening your hidden service with Firefox. You should see messages similar to these:
Request for identification
Certificate details
Finally, working cgit via tor!

Setting up new v3 Hidden Service with ultimate security: Part 3: Client Authentication

This post is a part of Tor v3 tutorial. Other parts are:

  1. Hidden Service setup
  2. PKI and TLS
  3. Client Authentication
  4. Installing client certificates to Firefox for Android

As we now have working Public Key Infrastructure, we are ready to use it for more than encrypting traffic (which is already encrypted by Tor). We can very easily turn on client verification on our server. This will prevent anybody not having valid certificate issued by us from visiting our hidden webpage – just in case hiding domain name in hidden services version 3 leaks the name somehow (which should not happen anymore in v3). In this part we will issue client certificate (the procedure is almost identical to server certificate), then configure httpd to require client identification and finally configure Firefox to try sending the certificate. Let’s go!

Issuing user certificate

In my case tmp directory emulated client machine and ca is my Cerificate Authority, which issues certificates. We start by creating request on client side, then sign it on CA side.

$ mkdir tmp
$ cd tmp
$ openssl genrsa -out v3l0c1r4pt0r@gmail.com.key.pem 4096
Generating RSA private key, 4096 bit long modulus
........++
..............................................++
e is 65537 (0x010001)
$ openssl req -config ../ca/intermediate/openssl.cnf -key v3l0c1r4pt0r@gmail.com.key.pem -new -sha256 -out v3l0c1r4pt0r@gmail.com.csr.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:PL
State or Province Name [England]:lodzkie
Locality Name []:
Organization Name [Alice Ltd]:r4pt0r Test Systems
Organizational Unit Name []:
Common Name []:v3l0c1r4pt0r@gmail.com
Email Address []:v3l0c1r4pt0r@gmail.com
$ chmod 400 v3l0c1r4pt0r@gmail.com.*.pem
$ cp v3l0c1r4pt0r@gmail.com.csr.pem ../ca/intermediate/csr/
$ cd ../ca
$ openssl ca -config intermediate/openssl.cnf -extensions usr_cert -days 375 \
> -notext -md sha256 -in intermediate/csr/v3l0c1r4pt0r@gmail.com.csr.pem \
> -out intermediate/certs/v3l0c1r4pt0r@gmail.com.cert.pem
Using configuration from intermediate/openssl.cnf
Enter pass phrase for /home/r4pt0r/Research/cubie/newtor/ca/intermediate/private/intermediate.key.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4097 (0x1001)
        Validity
            Not Before: Feb 27 17:14:40 2018 GMT
            Not After : Mar  9 17:14:40 2019 GMT
        Subject:
            countryName               = PL
            stateOrProvinceName       = lodzkie
            organizationName          = r4pt0r Test Systems
            commonName                = v3l0c1r4pt0r@gmail.com
            emailAddress              = v3l0c1r4pt0r@gmail.com
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Cert Type:
                SSL Client, S/MIME
            Netscape Comment:
                OpenSSL Generated Client Certificate
            X509v3 Subject Key Identifier:
                ED:24:E6:FF:1D:9B:61:AC:29:66:39:59:FB:5D:77:25:F7:A3:55:47
            X509v3 Authority Key Identifier:
                keyid:3D:AC:8E:21:79:5A:AD:7B:7C:92:92:65:B7:19:D0:E8:00:0E:50:70

            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Client Authentication, E-mail Protection
Certificate is to be certified until Mar  9 17:14:40 2019 GMT (375 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
$ cd ../tmp
$ cp ../ca/intermediate/certs/v3l0c1r4pt0r@gmail.com.cert.pem ./
$ openssl pkcs12 -export -inkey v3l0c1r4pt0r@gmail.com.key.pem -in v3l0c1r4pt0r@gmail.com.cert.pem -out v3l0c1r4pt0r@gmail.com.p12
Enter Export Password:
Verifying - Enter Export Password:

Last step was packaging certificate and key into PKCS#12 container. That is for securing key (we can encrypt it with password), and is a form required by Firefox. After creation of .p12 (and verifying it is fine), we can (and SHOULD) delete source files, as they are not protected in any way.

Configuring httpd to require user certificate

To enforce client verification, following lines must be added to virtual host configuration, in our case it might go just after SSL certificate file paths.

    SSLVerifyClient require
    SSLVerifyDepth 2

We have to reload httpd for changes to take effect.

Installing certificate to Firefox

At last, to start using newly generated certificate, we should install it to Firefox. The procedure is similar to the one with CA certificate. We need to open Certificate Manager window. Then, instead of going to Authorities, we go to Your Certificates. Then we click on Import and select .p12 file.

Certificate Manager / Your Certificates

If the file has password, Firefox will ask for it and after successfully reading the content. If everything went well, you should see your certificate on the list. Now we can try connecting to our hidden service. We should see the window like this:

Server asks for client’s identity

Finally, after confirmation, you should see your hidden service content. Congrats!

Setting up new v3 Hidden Service with ultimate security: Part 2: PKI and TLS

This post is a part of Tor v3 tutorial. Other parts are:

  1. Hidden Service setup
  2. PKI and TLS
  3. Client Authentication
  4. Installing client certificates to Firefox for Android

After setting up working Tor hidden service, the next step to ultimate security is having properly implemented Public Key Infrastructure (PKI). For this step, there are a lot of tutorials already existing and there is not much that needs to be added to them. Personally, I was using tutorial available here for the second time now and I find it very well-written. Because I am going to follow this tutorial, I will just post commands that have to be executed.

Before starting, I have to add one important remark. To make our PKI really secure one, it is crucial to have root CA air-gapped, that is device, on which it will be generated should be disconnected permanently from the internet. Good candidate for such a device might be some old laptop or Raspberry Pi Zero, as it lacks Ethernet port and anything reasonable to connect to internet. It is also important to store generated certificate in a safe place and secure it with strong non-dictionary password, which will be saved only in our mind.

If the requirements are fulfilled, we can start the setup. Below are commands to type as well as output from them, for easier determination of whether the commands were successful or not.

Preparations

At first, we need to create following directory structure:

ca
β”œβ”€β”€ [drwxr-xr-x]  certs
β”œβ”€β”€ [drwxr-xr-x]  crl
β”œβ”€β”€ [-rw-r--r--]  index.txt
β”œβ”€β”€ [drwxr-xr-x]  intermediate
β”‚Β Β  β”œβ”€β”€ [drwxr-xr-x]  certs
β”‚Β Β  β”œβ”€β”€ [drwxr-xr-x]  crl
β”‚Β Β  β”œβ”€β”€ [drwxr-xr-x]  csr
β”‚Β Β  β”œβ”€β”€ [-rw-r--r--]  index.txt
β”‚Β Β  β”œβ”€β”€ [drwxr-xr-x]  newcerts
β”‚Β Β  β”œβ”€β”€ [drwx------]  private
β”‚Β Β  └── [-rw-r--r--]  serial
β”œβ”€β”€ [drwxr-xr-x]  newcerts
β”œβ”€β”€ [drwx------]  private
└── [-rw-r--r--]  serial

And file content is (enclosed between pipe symbols: |):

./index.txt: ||
./intermediate/index.txt: ||
./intermediate/serial: |1000
|
./serial: |1000
|

Then, we need to save this file into root/openssl.cnf and this file into root/intermediate/openssl.cnf. Inside them, the only thing that have to be changed is dir property in CA_default section. Use absolute path to your directory.

Root CA

Note: when giving values for certain fields, better give some country, state (I have just checked it’s necessary), ON, most importantly, Common Name and e-mail. Just in case some program will check if they exists.

$ openssl genrsa -aes256 -out private/ca.key.pem 8192
Generating RSA private key, 8192 bit long modulus
.................++
....++
e is 65537 (0x010001)
Enter pass phrase for private/ca.key.pem:
Verifying - Enter pass phrase for private/ca.key.pem:
$ chmod 400 private/ca.key.pem
$ openssl req -config openssl.cnf -key private/ca.key.pem -new -x509 -days 7300 \
> -sha256 -extensions v3_ca -out certs/ca.cert.pem
Enter pass phrase for private/ca.key.pem:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:PL
State or Province Name [England]:lodzkie
Locality Name []:
Organization Name [Alice Ltd]:r4pt0r Test Systems
Organizational Unit Name []:
Common Name []:r4pt0r Root CA
Email Address []:admin@example.com
$ chmod 444 certs/ca.cert.pem
$ openssl x509 -noout -text -in certs/ca.cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            9a:16:72:e8:ac:81:cd:be
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = PL, ST = lodzkie, O = r4pt0r Test Systems, CN = r4pt0r Root CA, emailAddress = admin@example.com
        Validity
            Not Before: Feb 20 17:22:27 2018 GMT
            Not After : Feb 15 17:22:27 2038 GMT
        Subject: C = PL, ST = lodzkie, O = r4pt0r Test Systems, CN = r4pt0r Root CA, emailAddress = admin@example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (8192 bit)
                Modulus:
                    00:dd:8c:8f:5d:be:f4:0f:63:91:9c:73:bf:a8:17:
<quite a lot of data>
                    6d:c1:3f:5c:05
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                29:53:8A:D2:ED:CF:35:C2:BB:A8:12:06:01:74:99:A3:B8:E5:DC:FE
            X509v3 Authority Key Identifier:
                keyid:29:53:8A:D2:ED:CF:35:C2:BB:A8:12:06:01:74:99:A3:B8:E5:DC:FE

            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
    Signature Algorithm: sha256WithRSAEncryption
         a9:6d:9e:d4:bf:1b:55:d8:f0:b5:e9:9d:56:e8:58:04:d6:c3:
<quite a lot of data>
         89:50:26:4f:3e:93:95:06:c7:38:08:c7:16:0e:d2:a2

Intermediate CA

$ openssl genrsa -aes256 -out intermediate/private/intermediate.key.pem 8192
Generating RSA private key, 8192 bit long modulus
.++
........................................................................................................................................................................................................................................................................................++
e is 65537 (0x010001)
Enter pass phrase for intermediate/private/intermediate.key.pem:
Verifying - Enter pass phrase for intermediate/private/intermediate.key.pem:
$ chmod 400 intermediate/private/intermediate.key.pem
$ openssl req -config intermediate/openssl.cnf -new -sha256 \
> -key intermediate/private/intermediate.key.pem -out intermediate/csr/intermediate.csr.pem
Enter pass phrase for intermediate/private/intermediate.key.pem:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:PL
State or Province Name [England]:lodzkie
Locality Name []:
Organization Name [Alice Ltd]:r4pt0r Test Systems
Organizational Unit Name []:
Common Name []:r4pt0r Intermediate CA
Email Address []:admin@example.com
$ openssl ca -config openssl.cnf -extensions v3_intermediate_ca -days 3650 \
> -notext -md sha256 -in intermediate/csr/intermediate.csr.pem -out intermediate/certs/intermediate.cert.pem
Using configuration from openssl.cnf
Enter pass phrase for ca/private/ca.key.pem:
Can't open ca/index.txt.attr for reading, No such file or directory
140341269315520:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:74:fopen('ca/index.txt.attr','r')
140341269315520:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:81:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4096 (0x1000)
        Validity
            Not Before: Feb 20 17:35:09 2018 GMT
            Not After : Feb 18 17:35:09 2028 GMT
        Subject:
            countryName               = PL
            stateOrProvinceName       = lodzkie
            organizationName          = r4pt0r Test Systems
            commonName                = r4pt0r Intermediate CA
            emailAddress              = admin@example.com
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                3D:AC:8E:21:79:5A:AD:7B:7C:92:92:65:B7:19:D0:E8:00:0E:50:70
            X509v3 Authority Key Identifier:
                keyid:29:53:8A:D2:ED:CF:35:C2:BB:A8:12:06:01:74:99:A3:B8:E5:DC:FE

            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
Certificate is to be certified until Feb 18 17:35:09 2028 GMT (3650 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
$ openssl x509 -noout -text -in intermediate/certs/intermediate.cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4096 (0x1000)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = PL, ST = lodzkie, O = r4pt0r Test Systems, CN = r4pt0r Root CA, emailAddress = admin@example.com
        Validity
            Not Before: Feb 20 17:35:09 2018 GMT
            Not After : Feb 18 17:35:09 2028 GMT
        Subject: C = PL, ST = lodzkie, O = r4pt0r Test Systems, CN = r4pt0r Intermediate CA, emailAddress = admin@example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (8192 bit)
                Modulus:
                    00:d4:c9:03:36:4a:dd:3d:ee:ca:bd:c1:d8:fe:51:
<quite a lot of data>
                    5a:ca:74:74:c8:a2:b2:69:0a:0c:c7:f9:d6:8a:58:
                    41:45:73:fc:2b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                3D:AC:8E:21:79:5A:AD:7B:7C:92:92:65:B7:19:D0:E8:00:0E:50:70
            X509v3 Authority Key Identifier:
                keyid:29:53:8A:D2:ED:CF:35:C2:BB:A8:12:06:01:74:99:A3:B8:E5:DC:FE

            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
    Signature Algorithm: sha256WithRSAEncryption
         15:04:2f:85:89:f6:77:82:c4:60:78:f0:4f:ac:39:ad:15:14:
<quite a lot of data>
         7c:71:95:db:16:02:de:01:70:fe:8f:48:94:92:11:1b
$ openssl verify -CAfile certs/ca.cert.pem intermediate/certs/intermediate.cert.pem
intermediate/certs/intermediate.cert.pem: OK
$ cat intermediate/certs/intermediate.cert.pem certs/ca.cert.pem > intermediate/certs/ca-chain.cert.pem
$ chmod 444 intermediate/certs/ca-chain.cert.pem

Server certificate

In the following parts, wherever [domain] appears, it should be changed to hostname of our hidden service.

At first, we need to generate certificate request (CSR) on our server:

$ openssl genrsa -out [domain].onion.key.pem 4096
Generating RSA private key, 4096 bit long modulus
.................++
..............................................................................++
e is 65537 (0x010001)
$ chmod 400 [domain].onion.key.pem
$ openssl req -config ca/intermediate/openssl.cnf \
> -key [domain].onion.key.pem -new -sha256 -out [domain].onion.csr.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:PL
State or Province Name [England]:lodzkie
Locality Name []:
Organization Name [Alice Ltd]:r4pt0r Test Systems
Organizational Unit Name []:
Common Name []:[domain].onion
Email Address []:admin@[domain].onion

Then, we will sign the request with intermediate CA private key, thus issuing the certificate. But first of all, we need to receive the CSR from the server, to intermediate/csr/ directory.

$ openssl ca -config intermediate/openssl.cnf -extensions server_cert -days 375 \
> -notext -md sha256 -in intermediate/csr/[domain].onion.csr.pem -out intermediate/certs/[domain].onion.cert.pem
Using configuration from intermediate/openssl.cnf
Enter pass phrase for ca/intermediate/private/intermediate.key.pem:
Can't open ca/intermediate/index.txt.attr for reading, No such file or directory
139810167087040:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:74:fopen('ca/intermediate/index.txt.attr','r')
139810167087040:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:81:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4096 (0x1000)
        Validity
            Not Before: Feb 20 17:52:13 2018 GMT
            Not After : Mar  2 17:52:13 2019 GMT
        Subject:
            countryName               = PL
            stateOrProvinceName       = lodzkie
            organizationName          = r4pt0r Test Systems
            commonName                = [domain].onion
            emailAddress              = admin@[domain].onion
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Cert Type:
                SSL Server
            Netscape Comment:
                OpenSSL Generated Server Certificate
            X509v3 Subject Key Identifier:
                DD:6E:E8:78:91:B9:F7:F4:0A:06:3F:D2:38:6D:11:4E:3C:D3:BC:E0
            X509v3 Authority Key Identifier:
                keyid:3D:AC:8E:21:79:5A:AD:7B:7C:92:92:65:B7:19:D0:E8:00:0E:50:70
                DirName:/C=PL/ST=lodzkie/O=r4pt0r Test Systems/CN=r4pt0r Root CA/emailAddress=admin@example.com
                serial:10:00

            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
Certificate is to be certified until Mar  2 17:52:13 2019 GMT (375 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
$ openssl x509 -noout -text -in intermediate/certs/[domain].onion.cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4096 (0x1000)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = PL, ST = lodzkie, O = r4pt0r Test Systems, CN = r4pt0r Intermediate CA, emailAddress = admin@example.com
        Validity
            Not Before: Feb 20 17:52:13 2018 GMT
            Not After : Mar  2 17:52:13 2019 GMT
        Subject: C = PL, ST = lodzkie, O = r4pt0r Test Systems, CN = [domain].onion, emailAddress = admin@[domain].onion
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:c5:d3:e2:a0:97:b8:4d:67:22:94:c9:be:17:e3:
<quite a lof of data>
                    49:76:cf
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Cert Type:
                SSL Server
            Netscape Comment:
                OpenSSL Generated Server Certificate
            X509v3 Subject Key Identifier:
                DD:6E:E8:78:91:B9:F7:F4:0A:06:3F:D2:38:6D:11:4E:3C:D3:BC:E0
            X509v3 Authority Key Identifier:
                keyid:3D:AC:8E:21:79:5A:AD:7B:7C:92:92:65:B7:19:D0:E8:00:0E:50:70
                DirName:/C=PL/ST=lodzkie/O=r4pt0r Test Systems/CN=r4pt0r Root CA/emailAddress=admin@example.com
                serial:10:00

            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
    Signature Algorithm: sha256WithRSAEncryption
         b0:92:d9:d5:3b:31:38:f6:b8:51:1f:41:e9:f7:d8:e6:33:67:
<quite a lot of data>
         ee:c4:eb:19:86:69:00:26:8d:04:7b:97:0b:8f:f5:76
$ openssl verify -CAfile intermediate/certs/ca-chain.cert.pem intermediate/certs/[domain].onion.cert.pem
intermediate/certs/[domain].onion.cert.pem: OK

httpd configuration

Finally, we can use generated files to set up HTTPS encryption on webserver. For this, I am using httpd as it is the most common webserver in use. We need following files:

  1. [domain].onion.key.pem – this is private key, that will be used to set up TLS session
  2. [domain].onion.cert.pem – this is certificate that will prove our identity, so web browser will not display any warnings as long as we will have CA certificate installed
  3. ca-chain.cert.pem – this is chain of certificates we created together with intermediate CA, that consists of both CAs – root and intermediate

Below is httpd configuration file, after enabling TLS:

Listen 666

<VirtualHost *:666>
    ServerAdmin admin@re-ws.pl
    DocumentRoot "/home/r4pt0r/tor/hs/public_html"
    ServerName 192.168.253.4
    ErrorLog "[path]/tor/hs/error_log"
    CustomLog "[path]/tor/hs/access_log" common
    ScriptAlias /cgit/ "/usr/lib/cgit/cgit.cgi/"
    Alias /cgit-css "/usr/share/webapps/cgit/"
    SSLEngine on
    SSLCertificateFile "[path]/tor/hs/tls/[domain].onion.cert.pem"
    SSLCertificateKeyFile "[path]/tor/hs/tls/[domain].onion.key.pem"
    SSLCACertificateFile "[path]/tor/hs/tls/ca-chain.cert.pem"
</VirtualHost>

As can be seen above, all necessary files had been moved to tls directory of our hidden service main directory.

Afterwards, one slight change is needed in torrc file:

HiddenServicePort 443 127.0.0.1:666

From now on, we need to use https://[domain].onion to visit our site, as it is now TLS-encrypted and using port 443, which is default for HTTPS. For convenience, we can set up another httpd vhost on different port, that will redirect all HTTP traffic through HTTPS and link it to port 80, so remembering about https in address will not be necessary. But, it is only optional, so I will leave it as an exercise to the reader.

Firefox

From this point it is useful to have Firefox that is not constantly reminding about insecure connection. To prevent this, we should install CA certificate into Firefox. One remark here: as we are going to hack Firefox to trust our certificate, now our whole browsing through that instance of Firefox relies on our CAs private key. So, it is best to not use the same instance for anything else unless you are really sure, the private keys for both root and intermediate are perfectly secure.

To install the certificate, follow the screenshots below:

On preferences page, go to security and scroll all the way down to View Certificates button
On Authorities tab, click Import and select your CA certificate
Confirm that this certificate will be able to identify websites
Finally, we are secure and no exclamation mark appears!

Setting up new v3 Hidden Service with ultimate security: Part 1: Hidden Service setup

This post is a part of Tor v3 tutorial. Other parts are:

  1. Hidden Service setup
  2. PKI and TLS
  3. Client Authentication
  4. Installing client certificates to Firefox for Android

As a student I was lucky to have unlimited private Git repositories on Github, since they introduced that to their first paid plan. Unfortunately, I don’t have access to educational e-mail anymore, so I won’t be able to renew the service. This leads to a need to have that feature migrated to somewhere else. Some time ago, I installed cgit and gitolite on my single board computer (SBC). But, because of Github, there was no need to use that. Now it seems like a good replacement to Github’s Developer plan.

Few weeks ago, there was interesting event – Tor Project introduced new version of their Hidden Services – v3, which changes length of hidden service address in .onion domain and disables “feature” enabling some nodes in the network to index all existing service addresses. This seems like a good moment to give it a try and check, how fast (or rather how slow) will be the solution providing git through Tor on few-year-old SBC. By the way, I will show, how to configure things with maximum security in mind.

Disclaimer: I am not a person with deep knowledge of inner workings of Tor network, so I strongly encourage you to read thing or two, about how to use it safely. This article might contain errors that might reveal your identity, especially when used together with not-self-owned hidden services.

Prerequisites

Let’s start with summary of what we will need to make Tor v3 work:

  • tor in version 0.3.2.9 or higher
  • alternatively Tor Browser 7.5 or higher
  • for Android: Orbot and Orfox (at the moment of writing this, there is no support in current version of Orbot, so custom compilation is required – I am using Termux to provide tor binary)
  • httpd or any other HTTP server, able to provide service with only one vhost on separate TCP port

Because of the way, I am planning to configure hidden service in future, it might be a good idea to set up separate Tor browser at this moment, dedicated to this service, if it is going to be production configuration. If this is just an experiment, this advice could safely be ignored. However it is good to know, how to undo any modifications to the browser that will be done in the next parts.

httpd

What we need to do is to listen on localhost, on some random TCP port. Then we will set up httpd to provide only one virtual host on this custom port. It would be perfect to disable any other vhosts as our hidden service will work also as non-hidden service for local users, so if other service is buggy and allows to connect to other local services (see e.g. DNS rebinding), at least address of our hidden service will be compromised.

I have following configuration:

Listen 666

<VirtualHost *:666>
    ServerAdmin [email]@[domain]
    DocumentRoot "[path]/public_html"
    ServerName [domain].onion
    ErrorLog "[path]/error_log"
    CustomLog "[path]/access_log" common
</VirtualHost>

<Directory "[path]/public_html">
    DirectoryIndex index.html index.php index.txt
    AllowOverride All
    Options FollowSymlinks
    Require all granted
</Directory>

Furthermore, httpd must be able to traverse to public_html directory, so every directory from public_html up to root must have execute privilege for http user and directory itself as well as its content must be available (or better owned) by http.

After that and after starting httpd, it should be possible to visit http://localhost:666 via web browser and see content of public_html directory. If this is true, we can move on to tor configuration.

tor

SocksPort auto

HiddenServiceDir /etc/tor/hsv3
HiddenServiceVersion 3
HiddenServicePort 80 127.0.0.1:666

SafeLogging 0
Log notice stdout
Log notice file /etc/tor/hsv3/hs.log
Log info file /etc/tor/hsv3/hsinfo.log

Now, on the first startup of tor,Β  it should create keys for our new hidden service. We can look into /etc/tor/hsv3/hostname to see the .onion address. It is good idea to set key files and hostname file as readable as only user running tor service. In case of service started by systemd, this will probably be tor by default.

After starting tor service (systemctl start tor in case of systemd), we can check if everything works properly by visiting our hidden service with tor-enabled browser (using tor 0.3.2.9 or higher). That’s it.

Firefox for Android

At the time of writing this article there is still no upgrade for Orbot app, providing GUI interface for tor. Because of that, it might be required to use ordinary Firefox to use tor as a proxy, which is generally bad idea for connecting to any hidden services, because of privacy and anonymity. Fortunately, we can live with revealing our identity to ourselves πŸ™‚ so we can do it only this single time.

What we need to change are following configuration options, available under about:config page:

  • network.proxy.socks to localhost
  • network.proxy.socks_port to 9050
  • network.proxy.socks_remote_dns to true
  • network.proxy.socks_version to 5, if any other (should be default)
  • network.proxy.type to 1 (0 means no proxy, 5 is system proxy)

Conclusion

Now we are ready to use our hidden service, from both desktop and mobile. Still, we use only HTTP protocol, which is not a big problem, as tor already provides encryption. Neverheless our next goal would be to configure HTTPS. And then we will configure client authentication for ultimate security of our hidden service.

LKV373A: state of the reverse engineering

This article is part of series about reverse-engineering LKV373A HDMI extender. Other parts are available at:

Last time, I showed how to do objdump, able to disassemble instructions for not yet supported processor – LKV373A encoder. This time, as promised in part 5, I am just publishing, what I was able to do.

Reverse engineering repo

Repository is located, as usually, on Github, here. The most important file there is printout of encoder firmware, generated with my fork of objdump. Also, there are few scripts, I used to make process more automatic. Especially useful for someone, who might want to reproduce the process or continue my work might be the ELF generator script. It is written in Python and uses my ELF-creation library from part 4. As there is no way to install the library where it should be – /usr/lib/pythonX.X/site-packages, I used MAKEELFPATH environment variable to find makeelf.

Less important scripts, but still sometimes useful, are the ones to generate graph of function cross-references. They are able of generating .dot file, which can be converted to PNG (which is bad way of making it useful – it is 32768 pixels wide) or opened somewhere. And I used Gephi to read the final graph.

binutils development

Also binutils fork was improved a little bit. It is now possible to see symbols on every jump and call instruction. Moreover I did a kind of hack to be able to see references to data, as they mostly consisted of two instructions – first setting upper half of register, then adding lower half. So in that case, objdump is now displaying content of register, which most of the time don’t work, as objdump parses code linearly, without caring about jumps. But, if this opcode pair is used to really reference some address, it is quite reliable, so it is possible to see the address, as well as symbol name, in same way as with calls. Also, there is much more information about opcode types, so completely unknown instructions had been eliminated. Some of them were not used a single time and were renamed to resXX – for reserved.

Does it make sense?

Now, as I know really a lot about the firmware, it is time to try to answer the question in the heading. Well, the goal of reversing the encoder was to find compression and checksum algorithm for SMEDIA/ITE firmware blob. And, as far as I know now, it is really likely that at least compression algorithm is in fact somewhere here. I can say that, because I’ve seen routine for processing the data that seem to be compressed in SMEDIA/SMAZ container. Moreover, SMEDIA itself is also processed here, in original form, as one of the firmware’s responsibilities is firmware upgrade, so it is being copied to some internal ROM. So, if both – compressed and decompressed versions of firmware is here, then decompression also should be somewhere.

Now, this reverse engineering work might take really, really long time to get some results, but on the other hand, I might find, what I am looking for today. So, now the topic will probably seem to be dead for some undefined time.

LKV373A: porting objdump

This article is part of series about reverse-engineering LKV373A HDMI extender. Other parts are available at:

After part number four, we already have ELF file, storing all the data we found in firmware image, described in a way that should make our analysis easier. Moreover, we have ability to define new symbols inside our ELF file. The next step is to add support for our custom architecture into objdump and this is what I want to show in this tutorial.

Finding best architecture to copy

If we want to set up new architecture in objdump code, we need to learn interfaces that need to be implemented. It would be easier if we can use some existing code to do so. After some looking into the binutils’ code I learned that what is of special interest are bfd and opcodes libraries. They contain code dedicated to particular architectures. The first one seem to be related to object file handling (which in our case is ELF), so we should not tinker with it too much. Second one is related to disassembling binary programs, so is what we are looking for.

I did some quick examination of source code related to popular architectures and it seems not to be easy to adjust to our needs. Architecture I found to be best suitable for modification is Microblaze. Its source seem to be quite well-written, clean and short. Also from my research of architecture name for LKV373A (part 2, failed by the way) I also remember it is quite similar to the one present in LKV373A, so it is even better decision to use it.

Compiling objdump for target architecture

At first it is useful to learn how to compile objdump, so it will be able to disassemble program written for our target. Microblaze is not really a mainstream architecture, so there aren’t many programs compiled for it available online after typing 'microblaze program elf' into usual search engine. However, I was able to find 2 of them, so I was able to verify that compilation worked. If you can’t find any, I uploaded these to MEGA, so they can serve as test cases. First one is minimal valid file, the other one is quite huge.

Compilation is very easy. The only thing that needs to be done beside usual ./configure && make && make install is adding target option to configure script. So, the script looks as follows:

./configure --target=microblaze-elf

Of course, install step can safely be skipped as well as compilation of other tools, beside objdump. objdump itself seem to be built using make binutils/objdump. However it can’t be build successfully using that shortcut, so whole binutils package must be configured the way, everything not buildable is excluded from the build.

Setting up own architecture

Next step is to add support for our brand new, custom architecture to binutils’ configuration files and copy microblaze sources, so they will simulate our architecture, until we will write our own implementation. Then it should be possible to test objdump again, against our sample microblaze programs and disassembly should still work.

Even without any modification to binutils’ source or configs, it should be possible to configure it for any random architecture. The only constraint is format of the target string: ARCH-OS-FORMAT, where FORMAT is most likely to be elf. So, if we pass lkv373a-unknown-elf as target, it will work. -unknown part is usually skipped and this will not work. If we need it to work, config.sub must be modified. config.sub is used to convert any string, passed to configure into canonical form, so in our case lkv373a-unknown-elf. If it detects, that it is already in canonical form, it does nothing.

Final configure command will be slightly more complex, as we have to disable some parts, that are not of our interest and requires additional effort to work:

./configure --target=lkv373a-unknown-elf --disable-gas --disable-ld --disable-gdb

Although passing something random as target option works on configure stage, it will obviously fail on make stage. What make is doing at first is configuring all the sublibraries. What is of our interest is bfd and opcodes. And the first one fails. So this is the first problem, we need to get rid of.

bfd/config.bfd

The purpose of this file is to set some environment variables depending on target architecture. If it does not know the architecture, it returns error to caller, which is probably bfd’s configure script, called by make. According to documentation in file header, it sets following variables:

  1. targ_defvec – default vector. This links target to list of objects that will provide support for ELF file built for specific architecture (stored in bfd/configure.ac)
  2. targ_selvecs – list of other selected vectors. Useful e.g. when we need support for both 32- and 64-bit ELFs. Not needed here.
  3. targ64_selvecs – 64-bit related stuff. Used when target can be both 32- and 64-bit, meaningless in our case.
  4. targ_archs – name of the symbol storing bfd_arch_info_type structure. It provides description of architecture to support.
  5. targ_cflags – looks like some hack to add extra CFLAGS to compiler. We don’t care.
  6. targ_underscore – not sure what it is, should have no impact on our goals (possible values are yes or no)

To sum up, what we need to do on this step is to define default vector, we will later add to configure.ac and set name of architecture description structure. The structure itself will be defined later. Finally, I ended up with the following patch:

@@ -173,6 +173,7 @@ hppa*)     targ_archs=bfd_hppa_arch ;;
 i[3-7]86)   targ_archs=bfd_i386_arch ;;
 i370)     targ_archs=bfd_i370_arch ;;
 ia16)     targ_archs=bfd_i386_arch ;;
+lkv373a)  targ_archs=bfd_lkv373a_arch ;;
 lm32)           targ_archs=bfd_lm32_arch ;;
 m6811*|m68hc11*) targ_archs="bfd_m68hc11_arch bfd_m68hc12_arch bfd_m9s12x_arch bfd_m9s12xg_arch" ;;
 m6812*|m68hc12*) targ_archs="bfd_m68hc12_arch bfd_m68hc11_arch bfd_m9s12x_arch bfd_m9s12xg_arch" ;;
@@ -924,6 +925,10 @@ case "${targ}" in
     targ_defvec=iq2000_elf32_vec
     ;;

+  lkv373a*-*)
+    targ_defvec=lkv373a_elf32_vec
+    ;;
+
   lm32-*-elf | lm32-*-rtems*)
     targ_defvec=lm32_elf32_vec
     targ_selvecs=lm32_elf32_fdpic_vec

bfd/configure.ac

Now we need to define vector, we just declared to use for lkv373a architecture.

505     k1om_elf64_fbsd_vec)         tb="$tb elf64-x86-64.lo elfxx-x86.lo elf-ifunc.lo elf-nacl.lo elf64.lo $elf"; target_size=64 ;;
506     lkv373a_elf32_vec)           tb="$tb elf32-lkv373a.lo elf32.lo $elf" ;;
507     l1om_elf64_vec)              tb="$tb elf64-x86-64.lo elfxx-x86.lo elf-ifunc.lo elf-nacl.lo elf64.lo $elf"; target_size=64 ;;

Unfortunately, as we did modifications to .ac script, we now need to rebuild our configure. From my experience, any tinkering with autohell, after solving one problem, creates 5 more. We need to get into bfd directory and reconfigure project:

cd bfd
autoreconf

Now, if it worked for you, you should definitely go, play some lottery πŸ™‚ . For me it said that I need exactly same version of autoconf as used by binutils’ developers. Because autoconf is so great, probably what I will show now is completely useless for anyone, but hacks I needed to do are at first to add:

20 m4_define([_GCC_AUTOCONF_VERSION], [2.69])

to the beginning of configure.ac file. Then bfd/doc/Makefile.am contains removed cygnus option at the beginning, in AUTOMAKE_OPTIONS, so we need to remove it. After that doing automake --add-missing, as autoreconf suggests, and then again autoreconf should solve the problem. But, as I said, this will probably not work for you. I can only wish you good luck.

(if were following the steps, you might have noticed that autoconf complained about not being in version 2.64 and we overridden version from 2.69 to 2.69 and it worked πŸ™‚ , don’t ask me, why, please!)

After this step, compilation should start (and obviously will fail miserably on bfd as it misses few symbols). Now its time to make bfd compilable.

bfd/elf32-lkv373a.c

This file is meant to provide support for custom features of ELF file. As we don’t have any, we can safely do nothing here. Good template of such file is elf32-m88k.c as it does exactly this.

One thing that seem to be important here is EM value of architecture described. EM is an enum used in ELF file to define target architecture, so it might be required to adjust in our new elf32-lkv373a.c file. By the way definition of this value have to be added to include/elf/common.h:

433 /* LKV373A architecture */
434 #define EM_LKV373A              0x373a

It might also be a good idea to add it to elfcpp/elfcpp.h. To make the file compile, it is necessary to add following to bfd/bfd-in2.h as value of bfd_architecture enum:

2398   bfd_arch_lkv373a,    /* LKV373A */

bfd/archures.c

As we declared bfd_lkv373a_arch as symbol with CPU description structure, we now need to add this declaration to archures.c, as this is the file, where it will be used. We have to add:

611 extern const bfd_arch_info_type bfd_l1om_arch;
612 extern const bfd_arch_info_type bfd_lkv373a_arch;
613 extern const bfd_arch_info_type bfd_lm32_arch;

bfd/targets.c

Similar situation is in targets.c file. Here we have to provide declaration of our vector as bfd_target. This will be another structure, which seem to be generated automatically, so we should not care about it.

704 extern const bfd_target l1om_elf64_fbsd_vec;
705 extern const bfd_target lkv373a_elf32_vec;
706 extern const bfd_target lm32_elf32_vec;

bfd/cpu-lkv373a.c

This last file, we need in bfd, provides bfd_arch_info_type structure and… that’s it! Can be easily borrowed from cpu-microblaze.c with only slight modifications. One thing that needs explanation here isΒ section_align_power. As far as I understand it, it is power of two to which the beginning of the section in memory must be aligned. It should be safe to put 0 here, as we are not going to load our ELF into memory.

This should close the bfd part of initialization. As you can see, there was no development at all to be done here. Let’s now go to opcodes library.

opcodes/configure.ac

At first we need to define objects to build for LKV373A architecture in opcodes library. This is quite similar to what we had to do in configure.ac of bfd library.

282         bfd_iq2000_arch)        ta="$ta iq2000-asm.lo iq2000-desc.lo iq2000-dis.lo iq2000-ibld.lo iq2000-opc.lo" using_cgen=yes ;;
283         bfd_lkv373a_arch)       ta="$ta lkv373a-dis.lo" ;;
284         bfd_lm32_arch)          ta="$ta lm32-asm.lo lm32-desc.lo lm32-dis.lo lm32-ibld.lo lm32-opc.lo lm32-opinst.lo" using_cgen=yes ;;

Hopefully, -dis file will be enough to be implemented. I’ve made a copy from microblaze configuration. The same way we will copy whole source file and any related headers in the next step.

Now, similarly to bfd’s configure.ac, we have to reconfigure it. And again, nobody knows what errors we will encounter.

opcodes/disassemble.c

The only thing that have to be done here is to set pointer of disassemble function. For this following snippets should be added:

53 #define ARCH_lkv373a
255 #ifdef ARCH_lkv373a
256     case bfd_arch_lkv373a:
257       disassemble = print_insn_lkv373a;
258       break;
259 #endif

And to disassemble.h:

62 extern int print_insn_lkv373a           (bfd_vma, disassemble_info *);

opcodes/lkv373a-dis.c

This is, where real stuff will happen. As our goal, for now, is not to make implementation of LKV373A architecture, but rather set everything up, so objdump will build, we can copy source file from microblaze-dis.c. It is also required to copy headers, related to MicroBlaze, used by this file, so:

  • opcodes/microblaze-dis.h
  • opcodes/microblaze-opc.h
  • opcodes/microblaze-opcm.h

And change include directives in them to link to lkv373a file, rather than microblaze ones.

Now, optionally we could change names of any symbols referring to name microblaze, but this should not be required, as original microblaze files should not be included in the build. The only change than need to be done is print_insn_microblaze into print_insn_lkv373a, as this is what we added to disassemble.c.

You should now be able to compile working objdump with LKV373A support (of course with wrong implementation, for now). We can now verify that everything works on slightly modified ELF file for MicroBlaze architecture (EM field must point to LKV373A – value must be 0x373a). Well done!

NOTE: all the steps, done till now are available on tutorial-setup tag in repository on Github.

Functions to implement

Now, finally the real fun starts. Bindings between opcodes library and objdump itself, require at leastΒ print_insn_lkv373a to be implemented.

What should happen inside this function is quite simple and can be described in following steps:

  1. Gets bfd_vma and struct disassemble_info (called info below) as parameters
  2. Read raw data containing instructions using info->read_memory_func
  3. Call info->memory_error_func in case of any errors
  4. Use info->fprintf_func to print disassembled instruction into info->stream
  5. Optionally use info->symbol_at_address_func to determine if there is any symbol declared at address decoded from instructions
  6. If symbol exists, call info->print_address_func
  7. Return number of bytes consumed

Following is some documentation, I wrote for easier implementation (mostly translated inline comments), of functions to be called:

  /**
   * \brief Function used to get bytes to disassemble
   *
   * \param memaddr Address of the current instruction
   * \param myaddr Buffer, where the bytes will be stored
   * \param length Number of bytes to read
   * \param dinfo Pointer to info structure
   *
   * \return errno value or 0 for success
   */
  int (*read_memory_func)
    (bfd_vma memaddr, bfd_byte *myaddr, unsigned int length,
     struct disassemble_info *dinfo);
  /**
   * \brief Call if unrecoverable error occurred
   *
   * \param status errno from read_memory_func
   * \param memaddr Address of current instruction
   * \param dinfo Pointer to info structure
   */
  void (*memory_error_func)
    (int status, bfd_vma memaddr, struct disassemble_info *dinfo);
  /**
   * \brief Pointer to fprintf
   *
   * \param stream Pass info->stream here
   * \param char Format string
   * \param ... vargs
   *
   * \return Number of characters printed
   */
  typedef int (*fprintf_ftype) (void *, const char*, ...) ATTRIBUTE_FPTR_PRINTF_2;
  /**
   * \brief Determines if there is a symbol at the given ADDR
   *
   * \param addr Address to check
   * \param dinfo Pointer to info structure
   *
   * \return If there is returns 1, otherwise returns 0
   * \retval 1 If there is any symbol at ADDR
   * \retval 0 If there is no symbol at ADDR
   */
  int (* symbol_at_address_func)
    (bfd_vma addr, struct disassemble_info *dinfo);
  /**
   * \brief Print symbol name at ADDR
   *
   * \param addr Address at which symbol exists
   * \param dinfo Pointer to info structure
   */
  /* Function called to print ADDR.  */
  void (*print_address_func)
    (bfd_vma addr, struct disassemble_info *dinfo);

For easier start of development, this commit can be used as template. You can find effects of implementation according to this description on lkv373a branch of my binutils fork on Github. After this step, you should have working objdump, able to disassemble architecture of your choice.

Alternative way

According to binutils’ documentation, porting to new architectures should be done using different approach. Instead of copying sources from other architectures, developers should write CPU description files (cpu/ directory) and then use CGEN to generate all necessary files. However, I found these files way too complicated comparing to goal, I wanted to achieve, therefore I used the shortcut. In reality, however, this might be a better way, as the final result should be the support for new architecture not only in objdump, but also in e.g. GAS (GNU assembler). If you want to go that way, another useful resource might be description of CPU description language.

Plans for the future

As I am now able to speed up reverse engineering of both instruction set and LKV373A firmware, I am planning to create public repository of my progress and guess operations done by some more opcodes as I already know only few of them. So, I will probably push some more commits to binutils repo as well. I hope this will enable me to gain some more knowledge about LKV373A and allow, me or someone else, to reverse engineer second part of the firmware, which seem to be way more interesting that the one, I was reverse engineering till now.

LKV373A: crafting ELF

This article is part of series about reverse-engineering LKV373A HDMI extender. Other parts are available at:

As we should now be able to follow any jump present in the code, it is now time to make analysis more automatic. My target tool for that purpose will be objdump. However, we still have firmware image as raw dump of memory. To be able to use objdump easily, we need to pack our firmware into some container understandable by objdump. Most obvious choice is ELF (Executable and Linkable Format) and this is what I am going to use.

For the purpose of packing data into ELF, I’ve made Python library that makes it easier. For now, it is able to split firmware image into sections, like .text or .data, so objdump will be able to disassembly only the parts of firmware that are in fact a code. Moreover, it can define symbols inside the binary, so it is possible to store information, where certain functions starts and ends, same for any variables, like strings. As of now, there is no CLI interface for the program. If it turns out that such interface is necessary (like for addition of many symbols), it will be added.

Library code can be downloaded from Github. Currently, any LKV373A-specific modifications to this library is stored on branch lkv373a, to not rubbish main – master branch. Throughout this tutorial, I assume, we are using code on this branch, so there might be some LKV373A-specifics, especially regarding enum types (i.e. processor architecture enum).

At this point, I need to warn, that I am not going to describe internal structure of ELF file, nor any features that might be visible from outside, like sections concept, so if you are not familiar with them, it is good time to learn about them, as it might be very difficult to understand, what I am writing about. There are many good resources explaining them. Ones I was using are: this blog post and this documentation.

Creating new ELF

Example code that creates brand new ELF file is as easy as:

 1 #!/usr/bin/python3
 2 # demo script for creating ELF
 3 import os
 4 from elf import *
 5 
 6 elf = ELF(e_machine=EM.EM_LKV373A)
 7 
 8 fp = os.open('lkv373a.fw.elf',os.O_CREAT|os.O_WRONLY)
 9 os.write(fp, bytes(elf))
10 os.close(fp)

This, at first does all necessary imports, then creates new ELF object in line 6, and, finally, converts it to bytes object and immediately writes to file descriptor. That’s it!

After this, you should get valid, empty ELF file for architecture called lkv373a, which, obviously does not exists and no other program know how to handle, but we are going to change that in future.

While creating ELF object, few things can be defined, in addition to architecture id. They are all described in documentation, I will mention near the end of this tutorial. You are also free to dig in structure of ELF object. There is no encapsulation in it and structure validation is very permissive, so even completely broken ELFs could be produced, if needed.

Adding section

Next step is to add some sections to our ELF file.

fw = os.open('LKV373A_TX_V3.0c_d_20161116_bin.bin',os.O_RDONLY)
fw_blob = os.read(fw, 0xffffffff)

irq_blob = fw_blob[:0x1000]
text_blob = fw_blob[0x7d100:0x7d100+0x0b53c0]
data_blob = fw_blob[0x0b53c0:0x0b53c0+0x102060]
smedia_blob = fw_blob[0x200000:0x200000+0x283105]

irq_id = elf.append_section('.irq',irq_blob,0)
txt_id = elf.append_section('.text',text_blob,0x7d100)
data_id = elf.append_section('.data',data_blob,0x0b53c0)
smedia_id = elf.append_section('.smedia',smedia_blob, 0x200000)

At first, I am extracting them from firmware image and then inserting them to ELF object. append_section is a handy wrapper to low-level modifications that must be done on ELF structure, hidden under what we can see as ELF instance (these low-level structures are, however still available to the user as ELF.Elf member).

Modifying section attributes

Ok, so now we have sections in our ELF file, ready to save to disk. Before that, one thing can yet be done: setting proper attributes. They tell readers, if program is able to write or execute sections of memory, among other features, I am going to ignore here. This might be useful, as some readers might be confused about what is code (text) and what is data. In our case, we have two text sections (.irq and .text), so we are going to set them executable flag (SHF_EXECINSTR). Furthermore, we will set SHF_ALLOC flag for any section that is going to be loaded into memory (so all of them).

This can be done with:

elf.Elf.Shdr_table[irq_id].sh_flags = SHF.SHF_ALLOC | SHF.SHF_EXECINSTR
elf.Elf.Shdr_table[txt_id].sh_flags = SHF.SHF_ALLOC | SHF.SHF_EXECINSTR
elf.Elf.Shdr_table[data_id].sh_flags = SHF.SHF_ALLOC
elf.Elf.Shdr_table[smedia_id].sh_flags = SHF.SHF_ALLOC

Adding segment

Segments are another concept, existing beside sections. They are stored in program header of ELF file and are somehow linked to section data. They allow to define another set of attributes to areas in memory. I don’t think they will be required to define, to perform analysis in objdump, but since at least one such program header, defining segment must exist in ELF file of type executable, there is interface similar to this for sections.

To define new segment, based on .text section, you can issue:

elf.append_segment(txt_id, flags='rx')

This also marks the segment as read and executable, but not writable.

Loading existing ELF

Loading existing ELF can be easily done from file with:

newelf, b = ELF.from_file('lkv373a.fw.elf')

Alternatively, it can also be loaded from bytes object:

fd = os.open('some.elf', os.O_RDONLY)
b = os.read(fd, 0xffff)
os.close(fd)
manualelf, b = Elf32.from_bytes(b)

In latter case, I assumed that os library is already imported into python.

Adding a symbol

This is very useful for making analysis of code. New symbol can be added using calls like:

elf.append_symbol('irq0', irq_id, 0, 0x44, STB.STB_GLOBAL, STT.STT_FUNC)
elf.append_symbol('sprintf', txt_id, 0x9b9f8-0x7d100, 0x78, STB.STB_GLOBAL,
        STT.STT_FUNC)
elf.append_symbol('thread_c_path', data_id, 0xba78a-0x0b53c0, 0xba7bb-0xba78a,
        STB.STB_LOCAL, STT.STT_OBJECT)

First call defines function of length 0x44 in .irq section. To do this, ID of .irq section must be known. Luckily, we want to add symbol at the beginning of the section, so as offset, 0 was provided.

In the second case, we also want to define a function, but now we only know absolute address of the function (0x9b9f8), but what we need to pass is offset in .text section. To achieve this, we need to subtract address of the start of .text section (0x7d100).

In the last example, we define a string as an object of certain address and length. Both address and length are computed by subtracting absolute addresses. This symbol will be marked as local, which is default behavior for append_symbol function.

Library documentation

There are many more things possible to do using makeelf library. What I showed here is mostly, what is possible using high-level wrappers, doing many things under the hood. But as there is also low-level interface, virtually anything is possible.

To make exploring interfaces easier, I’ve made doxygen documentation for most of the library. It can be found on my server, here. Feel free to use the library for anything you want.

Conclusion

The library presented here should allow us make one step further to easy to use reverse engineering environment. It will by the way allow to store new findings in easily-modifiable Python scripts.

What I showed in examples to library interfaces split LKV373A firmware image into 4 sections. At this moment I already know that there are at least 6 sections, where code and data are in two parts (forming ICDCDS layout, where I-irq, C-code and so on). Also there should be some more symbols possible to place at this moment.

If I succeed in porting objdump, or any other tool able to disassemble ELF file, next step would be to publish Python script, utilizing library presented here, that annotates LKV373A firmware. So stay tuned, I hope there will be many further interesting findings throughout this reversing process!