SADVE – tiny program for computing #define values

While tinkering with spy camera, I found one detail that is significantly slowing the process of reverse engineering and debugging the applications, installed on its embedded Linux platform – finding final values of preprocessor directives and sometimes also results of sizeof() operator.

As I am not aware of any existing solution for that problem (I guess there might be some included in one of the more sophisticated IDEs, however I use Vim for development) it is good reason to create one. By the way I used cmake template I published some days ago to bootstrap the project.

Usage

Ease of use was the main goal here, as it is obviously possible to create improvised solution by creating hello-world type of program, including required headers and printing the symbol we want to compute value of.

So, to be able to use SADVE, you just have to clone the repo and use standard cmake installation commands and you’re done:

mkdir -p build
cd build
cmake ..
make
sudo make install

Then you can call it like below:

sadve -d AF_INET sys/socket.h

And you should get 2 as an answer. That’s it. If instead you want to get size of some structure, you can type:

sadve -s sockaddr sys/socket.h

And you should get size of sockaddr structure. Obviously, you can see full usage with sadve --help.

Internals

Internally the program simply automates the process I described in the first paragraph of Usage – it applies what is desired by user to hello-world-like template and compiles. Therefore it might not be the best idea to make it a backend for web service available for general public, at least without a lot of isolation and input sanitization. However for private usage this should be enough. If you are interested in doing such task with cmake, I encourage you to dive into source code on Github.

To speed the process up, I had to store all cmake build files in ~/.cache with no interface for cleaning it up.

UART pinout for noname spy camera

Front side of the main board

As I wrote few months ago, I bought tiny WiFi camera, advertised as a spy camera or nannycam. This week, I decided to work on the topic a bit. However, due to some serious failure, I alarmed on Twitter, I was not able to connect to its WiFi hotspot anymore. Therefore I had to use UART to recover it from backup. Below you can find parameters needed to connect to this cam. At first however I want to present any identification numbers, that might be useful to confirm it is the same device, as it has no real name.

Identification

Device overview (rubber package already stripped from PCB)

As can be seen in picture on the right, the device consists of main PCB, camera with tape cable, battery pack and optional USB cable for charging. To be able to reach UART header, I had to strip the rubber package from main board. Below I was able to see two identification strings:

  1. HB-WIFI-Z6 – this is most likely the name of the board, unfortunately neither Google or even Taobao does not know it
  2. MS-ME198407 – this is very interesting, as it seems to mean some internal name of laptop computer (don’t know who is the vendor)

Furthermore on camera tape there is one more magic string – HY-OV9712-6. After first dash it seems to be oh – not to be confused with zero). How do I know it? Because OV9712 is model name of camera optics made by OmniVision and it more or less matches the parameters of the camera.

Last batch of IDs is, at first processor name and vendor, which is quite unusual at least outside China – T10 made by Ingenic, which appear to produce MIPS cores and dev boards for it. Also I can see in logs the board should be called ISVP, which is not necessarily true – see Google. At last cpuinfo says that system type is mango, which appear to be fairly common in cheap Chinese cameras.

UART pinout

It can be found on the back of the board, near its edge.

Back of the main board
5-pin header
Num. Function
1 ?
2 ?
3 RX
4 TX
5 GND

Software

To connect to UART above, you have to use 115200 bauds in 8N1 mode. During the powerup, you can see it utilizes custom uboot as bootloader. It should be possible to interrupt it in one second timeslot. After that Linux is loaded and it asks for login (you most likely will not see it because of the amount of messages printed). root account is present and does not ask for password. Continue reading “UART pinout for noname spy camera”

Using CMocka for unit testing C code

Writing unit tests along with the source code (or even before the code itself – see TDD) is currently very popular among programmers writing in languages like Java or C#. For C code, however it is a bit different. There are only a few frameworks enabling the possibility to write unit tests. One of them is quite special – it allows to mock functions. And its name is CMocka. Unfortunately there are not many resources that describes the process of setting up cmocka, especially together with cmake to allow programmers add new executables, tests and mocks without unnecessary overhead. But before showing how to do it, let’s go back to basics (if you already know them, you can skip next heading).

What is mocking?

Mocking is a mechanism that allows to substitute object, we do not want to test, with empty implementation, which we can further configure to do whatever we like, like simulate errors. Usually objects are mocked, because we import them from some external library and this is not purpose of unit testing to test these external dependencies. This is how they work in object-oriented languages like the ones mentioned at the beginning. In C, it is only a bit different in that, instead of mocking the object (which does not exist in C), we mock functions. Similarly to mocking object, this allows us to control behavior of external function and i.e. test reaction of our code to errors.

To give very short impression about the possibilities, it opens let’s say we want to test function that accepts connection to a socket. How could we test such a function without touching the accept() function? Probably we would need another program that performs connect(). In such a simple case it requires us to write second program to cover just one case. Then, what if we want to check reaction for failed connection establishment? Manpage tells us, ECONNABORTED is returned in that case. So, how to force our test program to break the connection before second side gets return from accept() function? Do you see how complex it gets?

But, what if we could link our test program (the one that calls function using accept()) to our own accept()? Then at first we could write our fake function so it pretends that real socket had been created and return some positive integer. In second case, we could just set errno to ECONNABORTED and return -1. That’s it!

This, and by the way, much more is possible with CMocka. Let’s see how to configure simplest possible project to use cmocka with cmake.

Hey cmake, do my unit tests!

For purpose of this tutorial, let’s say we have quite simple project. We just started it, so it’s perfect time to enable unit testing for TDD approach. It consists of one console (CLI) application split into two modules. First one – program is just main() function and another function it calls. Second one – module provide just one function that is also called by main(). For configuration, we have two extremely short CMakeLists.txt files:

CMakeLists.txt
cmake_minimum_required (VERSION 3.0) project (cmocka_template) add_subdirectory(src)
src/CMakeLists.txt
add_executable(program program.c module.c)

You can view the code itself on Github. With this configuration it is now possible to make the project with cmake. To enable testing, at first we have to append following script to main CMakeLists.txt:

27 list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake/Modules")
28 
29 # cmocka
30 option(ENABLE_TESTS "Perform unit tests after build" OFF)
31 if (ENABLE_TESTS)
32   find_package(CMocka CONFIG REQUIRED)
33   include(AddCMockaTest)
34   include(AddMockedTest)
35   add_subdirectory(test)
36   enable_testing()
37 endif(ENABLE_TESTS)

In line 27, we tell cmake to look for additional cmake scripts to include in cmake/Modules directory. This would be required by includes in lines 33 and 34.

Then, we define option to enable/disable tests. This will allow users not willing to run unit tests, build the project without having to satisfy dependency to CMocka in line 32.

In that line FindCmocka will be called and will setup few variables. Of our interest would be ${CMOCKA_LIBRARIES}. This points to CMocka library, we have to link with all our test programs.

Includes in lines 33 and 34, provides functions with the analogous names, first one is part of CMocka sources, the other define simple wrapper on the first one, so user do not have to type all the parameters that usually stays the same from test to test.

At last we include another build script from subdirectory and enable testing.

With help of the mentioned add_mocked_test wrapper in simplest case, where we do not use any functions external to the module, all we have to do is call add_mocked_test(module). However, if we do call some external functions, we have to call it like below and provide sources with these functions:

add_mocked_test(program SOURCES ${CMAKE_SOURCE_DIR}/src/module.c)

Alternatively, we can also join many sources into library and pass it like that:

add_mocked_test(module LINK_LIBRARIES mylib)

This finally becomes -lmylib in ld.

Simplest test

As we should have complete build script, we can now write simple test:

test/test_main.c
24 #include <stdarg.h> 25 #include <stddef.h> 26 #include <setjmp.h> 27 #include <cmocka.h> 28 29 #define main __real_main 30 #include "program.c" 31 #undef main 32 33 typedef struct {int a; int b; int expected;} vector_t; 34 35 const vector_t vectors[] = { 36 {0,1,0}, 37 {1,0,0}, 38 {1,1,1}, 39 {2,3,6}, 40 }; 41 42 static void test_internal(void **state) 43 { 44 int actual; 45 int i; 46 47 for (i = 0; i < sizeof(vectors)/sizeof(vector_t); i++) 48 { 49 /* get i-th inputs and expected values as vector */ 50 const vector_t *vector = &vectors[i]; 51 52 /* call function under test */ 53 actual = internal(vector->a, vector->b); 54 55 /* assert result */ 56 assert_int_equal(vector->expected, actual); 57 } 58 } 59 60 int main() 61 { 62 const struct CMUnitTest tests[] = { 63 cmocka_unit_test(test_internal), 64 }; 65 66 return cmocka_run_group_tests(tests, NULL, NULL); 67 }

At first we have to include some headers. Then in lines 29 and 31, preprocessor hack is done. This is because we already have a main() function used to define test suite. So to not have it redeclared, we temporarily change its name to __real_main, so this is how our program.c/main() will be called, when included in line 30. For tests with modules not containing main() function this is superfluous.

In lines 33-40, we define sets of data to feed into our function with expected results. This is useful if we have many such vectors. For single inputs/outputs pair this is unnecessary. In line 50 one vector is extracted from that array.

Then in line 53 function under test is called and in line 56 its value is asserted with assert_int_equal.

Finally in main() function we have to define list of tests in this suite and call cmocka_run_group_tests to do rest of the job for us. Done.

add_mocked_test internals

For better understanding of the process, let’s have a short look into add_mocked_test() function. It’s source is as simple as:

cmake/Modules/AddMockedTest.cmake
32 function(add_mocked_test name) 33 # parse arguments passed to the function 34 set(options ) 35 set(oneValueArgs ) 36 set(multiValueArgs SOURCES COMPILE_OPTIONS LINK_LIBRARIES LINK_OPTIONS) 37 cmake_parse_arguments(ADD_MOCKED_TEST "${options}" "${oneValueArgs}" 38 "${multiValueArgs}" ${ARGN} ) 39 40 # define test 41 add_cmocka_test(test_${name} 42 SOURCES test_${name}.c ${ADD_MOCKED_TEST_SOURCES} 43 COMPILE_OPTIONS ${DEFAULT_C_COMPILE_FLAGS} 44 ${ADD_MOCKED_TEST_COMPILE_OPTIONS} 45 LINK_LIBRARIES ${CMOCKA_LIBRARIES} 46 ${ADD_MOCKED_TEST_LINK_LIBRARIES} 47 LINK_OPTIONS ${ADD_MOCKED_TEST_LINK_OPTIONS}) 48 49 # allow using includes from src/ directory 50 target_include_directories(test_${name} PRIVATE ${CMAKE_SOURCE_DIR}/src) 51 endfunction(add_mocked_test)

At the beginning, arguments are parsed, which is outside of the scope of this article (those interested could read the official documentation). What is important in this part is that SOURCES from the function call becomes ADD_MOCKED_TEST_SOURCES, COMPILE_OPTIONS are ADD_MOCKED_TEST_COMPILE_OPTIONS and so on.

Then in line 41 add_cmocka_test function is called. This functions does some of the job for us. For example, we do not have to worry about defining executable, linking libraries to it and making it a test called by CTest.

Then in line 42 source file list is passed to it, so it can link them into final executable. We can also pass our own compiler and linker flags, so they are routed to it in lines 43-44 and 47.

Last thing it does is to link this test executable with all required libraries and the only requirements for CMocka to work is to pass its shared library using CMOCKA_LIBRARIES variable, which is available thanks to finding CMocka package in CMakeLists.txt.

Beside that, we make C headers in src/ directory visible to our test program in line 50. That’s it.

Enabling mocks

Killer feature of CMocka however is its API for creating mocked functions. To use it, on CMake side all we have to do, beside what we already did is add flags for linker (to be precise -Wl,--wrap=function_name for every function to be mocked). As can be seen in add_mocked_test source, we could use LINK_OPTIONS argument for that purpose. However it is timesaving to have this integrated into the function’s interface. All we have to do is add a loop for creating argument list:

cmake/Modules/AddMockedTest.cmake
40 # create link flags for mocks 41 set(link_flags "") 42 foreach (mock ${ADD_MOCKED_TEST_MOCKS}) 43 set(link_flags "${link_flags} -Wl,--wrap=${mock}") 44 endforeach(mock)

Then add new argument and modify LINK_OPTIONS of add_cmocka_test to pass that list:

53                   LINK_OPTIONS ${link_flags} ${ADD_MOCKED_TEST_LINK_OPTIONS})

Now, it should work. Then on the source side we can create new test for existing test_program test suite:

79 static void test_main(void **state)
80 {
81   int expected = 0;
82   int actual;
83 
84   /* expect parameters to printf call */
85   expect_string(__wrap_printf, format, "%d\n");
86   expect_value(__wrap_printf, param1, 60);
87 
88   /* printf should return 3 */
89   will_return(__wrap_printf, 3);
90 
91   /* call __real_main as this is main() from program.c */
92   actual = __real_main(0, NULL);
93 
94   /* assert that main return success */
95   assert_int_equal(expected, actual);
96 }

And write mocked object:

43 int __wrap_printf (const char *format, ...)
44 {
45   int param1;
46 
47   /* extract result from vargs ('printf("%d\n", result)') */
48   va_list args;
49   va_start(args, format);
50   param1 = va_arg(args, int);
51   va_end(args);
52 
53   /* ensure that parameters match expecteds in expect_*() calls  */
54   check_expected_ptr(format);
55   check_expected(param1);
56 
57   /* get mocked return value from will_return() call */
58   return mock();
59 }

Now let’s take a look at what is happening here. At first in function test_main, in lines 85-86 we are telling CMocka that printf function is expected to be called with parameter format of a certain content and param1 should have value 60. param1 here is first variadic argument for printf function and its name is completely arbitrary. The key to working mock is to use this same name inside of mocked function.

In line 89, we say mocked printf to return 3 (this is number of bytes written to console and will then be ignored by code in __real_main, but for completeness it is set to proper value here).

Finally, we call the function under test and assert its result, the same way as in the example without mocks above.

On the other side is the mocked printf implementation. In languages like Java it is common to have this mock automatically generated based on the rules provided by user. In C with CMocka it is not so easy, we have to write the mock ourselves. However it is not very hard.

As we chosen variadic function for our mock and we want to check if these variadic arguments are as expected, we have to extract them at first, which is done in lines 48-51. As a result we have param1 variable (notice that it is local variable). Remember that this must match exactly what we declared in test body using one of expect_* functions. This name is how function-parameter pair is extracted from internal dictionaries.

Then in lines 54-55 arguments are checked for having expected contents. Note that there is no difference in how we treat positional and variadic parameters in this step.

Finally in line 58, we extract value that we want to be returned from printf (declared in line 89). This may seem superfluous in such a simple case, however if we want use this mocked printf in more than one test, it is really useful feature. Have also in mind that linker does not give us any interface to have two or more mocks for one function. So this way we have chance to write one universal mock.

Final word

This tutorial was made during my work on template for CMocka+CMake projects. So now, as you understand how it is done, you can just go to my Github and clone the template to make it part of your project. The template is licensed under MIT license, so I don’t care about what you do with it, as long as you leave original copyright.

USB to serial converter drivers for Android revisited

Few years ago I compiled kernel drivers of cheap USB-to-serial converter for my previous Android phone. It took few years of using new phone, without single custom-compiled kernel module. Now it is time to change it. By the way, I am going to describe what changed and what hacks have to be made to make the process work on stock ROM, provided by Sony.

kernel is the key

First of all, we need kernel. To be precise, kernel sources. Without that, it is really hard to be successful (I don’t want to tell it is impossible, but really hard, believe me). Because Sony is very liberal in terms of cooperation with community, they provide anything required to tinker with the device (obviously together with caution message about warranty loss, but who cares, right? πŸ™‚ ).

First of all, we need to know, which firmware version the device uses. To be found in Android settings, as compilation number, or something like that. For me, it is 23.5.A.0.575. Then, we have to visit Open Devices downloads section and find our firmware. For me, it was a lot of scrolling, as I have no updates available for quite some time. Inside the package, there should be kernel directory, with complete kernel sources.

Where is my .config?

Next thing we need to know is, which defconfig to use. Full list should be in arch/arm/configs. Now, in case of Sony phones, there is slight problem, as they traditionally use codenames for devices. In case of Xperia Pro, I compiled for before, it was iyokan. For Xperia Z3 Compact, I use now, it is Aries and the only official source of those codenames, I know, is their Github profile. Of course it would be too easy to find some mapping and searching for z3 gives no result. Fortunately, I know my device’s codename.

$ find . -name *aries*
./include/config/mach/sony/aries.h
./arch/arm/configs/shinano_aries_defconfig
./arch/arm/mach-msm/board-sony_aries-gpiomux.c
./arch/arm/mach-msm/board-sony_aries-gpiomux.o
./arch/arm/mach-msm/bms-batterydata-aries.o
./arch/arm/mach-msm/.board-sony_aries-gpiomux.o.cmd
./arch/arm/mach-msm/.bms-batterydata-aries.o.cmd
./arch/arm/mach-msm/bms-batterydata-aries.c
./arch/arm/boot/.msm8974pro-ac-shinano_aries.dtb.cmd
./arch/arm/boot/dts/msm8974pro-ac-shinano_aries_common.dtsi
./arch/arm/boot/dts/dsi-panel-aries.dtsi
./arch/arm/boot/dts/msm8974pro-ac-shinano_aries.dtsi
./arch/arm/boot/dts/msm8974pro-ac-shinano_aries.dts
./arch/arm/boot/msm8974pro-ac-shinano_aries.dtb

As we can see, there is only one config, related to aries: shinano_aries_defconfig (Sony’s Github profile explains that Shinano is platform name). Then, we can safely use this in defconfig phase.

Compilation (and hacking)

Once we have all the sources and kernel configuration, we can start compilation. Or actually, we cannot (probably).

Hacking

Let’s see vermagic of random module already installed on a device:

# modinfo zl10353.ko                            
filename:       zl10353.ko
license:        GPL
author:         Chris Pascoe
description:    Zarlink ZL10353 DVB-T demodulator driver
parm:           debug_regs:Turn on/off frontend register dumps (default:off).
parmtype:       debug_regs:int
parm:           debug:Turn on/off frontend debugging (default:off).
parmtype:       debug:int
depends:
intree:         Y
vermagic:       3.4.0-perf-g43ea728 SMP preempt mod_unload modversions ARMv7

We can see at least two things that will cause troubles:

  1. -perf-g<sha-1>
  2. modversions

In the first case, git commit id is appended to kernel version. Unfortunately, we do not have their repository and after module compilation, we will end up with just 3.4.0. To fix the problem, we have to edit makefile and set EXTRAVERSION to the missing part, so it should look like:

VERSION = 3
PATCHLEVEL = 4
SUBLEVEL = 0
EXTRAVERSION = -perf-g43ea728
NAME = Saber-toothed Squirrel

The second detail, forces us to compile whole kernel. Otherwise, Android kernel will try to check if the module is compatible with current kernel (using CRC checksums) and will fail on missing module_layout symbol CRC.

Bad hacking

In case of very simple drivers, there is a way to omit kernel compilation. However, it is not a safest way to go and serves as permanent --force for modprobe/insmod. I advice to skip this section, unless you are really desperate (and you are not, before trying the proper way).

Go back to some random driver, like the one, we used for vermagic check, pull it to PC and issue:

$ modprobe --dump-modversions zl10353.ko 
0x2067c442      module_layout
0x15692c87      param_ops_int
0xe6b3b90a      arm_delay_ops
0x59e5070d      __do_div64
0x5f754e5a      memset
0x0fc539b8      kmalloc_caches
0x9d669763      memcpy
0x52ac1d50      kmem_cache_alloc_trace
0x27e1a049      printk
0xfbc76af9      i2c_transfer
0x037a0cba      kfree
0xefd6cf06      __aeabi_unwind_cpp_pr0

From my experience, I know that module_layout is most troublesome. So why not add it to modversions of our module? Just run (change the CRC to match modprobe output!):

echo -e '0x2067c442\tmodule_layout\tvmlinux\tEXPORT_SYMBOL_GPL' >> Module.symvers

And you should cheat kernel to trust the symbol, even if in fact it would be different in kernel compiled by you. Then, after insmodding the module, built using the same shortcut as in my previous tutorial, you should possibly see a lot of errors on your dmesg. You can hunt for the symbols, from there and chances are it will work. Haven’t tested personally and I discourage, unless you really know what you are doing. It is wiser choice to wait those few minutes for kernel to compile.

Device specific hacking

During my compilation, I had to do some more hacks, as I had problems with missing headers. This will possibly be only relevant to the specific kernel version and device pair, but just in case, I am writing it down. You can safely skip to compilation and only go back in case of problems with framebuffer for MSM processors.

Following should fix the problem:

ln -s ../../drivers/video/msm/mdss/mdss_mdp_trace.h include/trace/mdss_mdp_trace.h
ln -s ../../drivers/video/msm/mdss/mdss_mdp.h include/trace/mdss_mdp.h

Compiling

Now, it should be fairly easy, though time consuming. Type following commands, one after another:

ARCH=arm CROSS_COMPILE=arm-unknown-eabi- make shinano_aries_defconfig
ARCH=arm CROSS_COMPILE=arm-unknown-eabi- make
ARCH=arm CROSS_COMPILE=arm-unknown-eabi- make modules M=drivers/usb/serial CONFIG_USB_SERIAL=m CONFIG_USB_SERIAL_CP210X=m

If there is no unexpected error during the compilation, you should now be able to insmod your fresh module into the kernel. In case of CP2102 driver, I compiled, there are in fact two drivers: usbserial and cp210x. cp210x depends on usbserial, so usbserial have to be inserted first. Afterwards, if you connect the device, you should see in dmesg, it succeeded, and in case of cp210x, there should be a name of USB tty device (most likely /dev/ttyUSB0, as there should be no USB ttys before).

Just to prove it, that it works, below are photos of of Xperia, running Termux and minicom to connect to Cubieboard2:

Phone sniffing on Linux boot
Phone and Cubieboard2, connected together via UART

PPPP API: what I know

While analyzing security of iMiniCam app, I learned some part of UDP protocol used to communicate with the camera. Here is quick bunch of facts.

UDP layer

Port 32100 (Proto_Hello, to external server) and port 32108 (others, to camera).

Type-length layer

Following structure represents first header that is always found in datagram.

struct header {
  uint8_t magic;
  enum proto_type type;
  uint16_t length;
}

In this structure magic is always set to 0xf1, type is one of the below values, packed on single byte and length count data that follows this packet, so if it is equal to zero, only those four bytes are transmitted.

Packet types

These are all the types present in binary, I was reverse engineering. Names are derived from function names that crafts their content, so i.e. for Proto_DevLgn that is equal to 0x10 there exists function PPPP_Proto_Write_DevLgn, PPPP_Proto_Read_DevLgn and PPPP_Proto_Send_DevLgn.

enum proto_type {
  Proto_Hello = 0;
  Proto_HelloAck = 1;
  Proto_HelloTo = 2;
  Proto_HelloToAck = 3;
  Proto_QueryDID = 8;
  Proto_QueryDIDAck = 9;
  Proto_DevLgn = 0x10;
  Proto_DevLgnAck = 0x11;
  Proto_DevLgn_CRC = 0x12;
  Proto_DevLgn1_CRC = 0x13;
  Proto_P2PReq = 0x20;
  Proto_P2PReqAck = 0x21;
  Proto_LanSerch = 0x30;
  Proto_PunchTo = 0x40;
  Proto_PunchPkt = 0x41;
  Proto_P2PRdy = 0x42;
  Proto_RSLgn = 0x60;
  Proto_RSLgnAck = 0x61;
  Proto_RSLgn1 = 0x62;
  Proto_RSLgn1Ack = 0x63;
  Proto_ListReq1 = 0x67;
  Proto_ListReq = 0x68;
  Proto_ListReqAck = 0x69;
  Proto_RlyHello = 0x70;
  Proto_RlyHelloAck = 0x71;
  Proto_RlyPort = 0x72;
  Proto_RlyPortAck = 0x73;
  Proto_ByteCount = 0x78;
  Proto_RlyReq = 0x80;
  Proto_RlyReqAck = 0x81;
  Proto_RlyTo = 0x82;
  Proto_RlyPkt = 0x83;
  Proto_RlyRdy = 0x84;
  Proto_SDevRun = 0x90;
  Proto_SDevLgn = 0x91;
  Proto_SDevLgn_CRC = 0x92;
  Proto_DRWAck = 0xd1;
  Proto_PSR = 0xd8;
  Proto_Alive = 0xe0;
  Proto_AliveAck = 0xe1;
  Proto_Close = 0xf0;
  Proto_MGMDumpLoginDID = 0xf4;
  Proto_MGMDumpLoginDIDDetail = 0xf5;
  Proto_MGMDumpLoginDID1 = 0xf6;
  Proto_MGMLogControl = 0xf7;
  Proto_MGMRemoteManagement = 0xf8;
}

Proto_Hello

This packet’s content is empty (length=0). Sent to external server. Response is Proto_HelloAck.

Proto_HelloAck

Length is 0x10. Content is according to following structure:

struct HelloAck {
  struct sockaddr_in wan_addr;
}

Final word

I hope this very quick introduction will help someone and will lead to documenting the protocol, in enough part, that it will be possible to develop open source client application able to communicate with the cameras using it.

Security analysis of spy camera sold by Chinese suppliers: iMiniCam app

Lately, I bought some random, noname spy camera from usual Chinese sources. Just after unboxing, one detail seemed a bit suspicious to me: together with the camera there comes a link to Android app, that is meant to control the camera. What is worse, the link leads to some Chinese app shop, obviously with only one language available (notice the version – 1.0.33, as it is not the only one in the wild):

http://m.app.so.com/detail/index?pname=com.hbwy.fan.iminicams&id=3646142

As I’ve already seen apps that i.e. calls home, despite having no need to open a single socket and then sending data like IMEI, IMSI, MAC addresses and so on, red lamp turned on and I said to myself: stop. Let’s reverse engineer the thing first!

OSINT

Before I begun my reverse engineering, I attempted to ask uncle Google (and his friends πŸ™‚ ) to gain some knowledge. Usual virustotal gives me this:

Results of virustotal scan of original link (click for details)

Not bad for such a simple application. Searching other shops for the same name leaded to exactly the same application (shared objects included matches), but older version (1.0.25) and in their case virustotal stopped complaining. For me there are two possible reason for that:

  1. Google Play app is not supported by vendor anymore
  2. There is special edition for Chinese, with few surprises

In favor of the second option is the fact that Java code of only the China edition is heavily bloated and usual decompilers cannot reverse the code properly, so somebody tried to make reverse engineer’s life harder. In apk downloaded from downloadapk.net and apkplz.com, everything seem to be fine.

Below are the links, I’ve used in comparison:

  1. Google Play: https://play.google.com/store/apps/details?id=com.hbwy.fan.iminicams (1.0.25)
  2. downloadapk.net: https://downloadapk.net/down_IPLiveCam.html (1.0.25)
  3. apkplz.com: https://apkplz.com/android-apps/iplivecam-apk-download (1.0.25)

Just for completeness, below is the name and webpage of suspected vendor of the stuff, I found accidentally: Albert Technology Co., Ltd. Shenzhenshihong Bo, and their webpage is here: http://www.honest-tech.com/en/

Let’s reverse then

First thing,I’ve noticed after decompilation of APK’s Java classes is:

  public static String a(String paramString)
  {
    if (paramString == null)
      return "";
    if (paramString.length() == 0)
      return "";
    paramString = new BigInteger(paramString.getBytes());
    return new BigInteger("0933910847463829232312312").xor(paramString).toString(16);
  }

  public static String b(String paramString)
  {
    if (paramString == null)
      return "";
    if (paramString.length() == 0)
      return "";
    BigInteger localBigInteger = new BigInteger("0933910847463829232312312");
    try
    {
      paramString = new String(new BigInteger(paramString, 16).xor(localBigInteger).toByteArray());
      return paramString;
    }
    catch (Exception paramString)
    {
    }
    return "";
  }

So, it’s gonna be interesting… As it turns out this one seem to be unused. But, after going a bit further and searching for all hardcoded strings I found this:

cfans/ufo/sdk/d/a.java:37:    PPPP_APIs.PPPP_Initialize("EBGDEIBIKEJOGCJNEBGBFDEOHJNDHANKGEFCBNCJAPJILDLIDEABCLOKGMLNJBKDALMNLKDOONMGAFCJJKNHJN".getBytes());

PPPP is shared object linked to the app via NDK, so I started reversing this libPPPP_API.so. Not so long time later, I was able to see decoded string by running the library and connecting gdb to it:

(gdb) p (char*)$sp+0xc
$16 = 0xbefff78c "121.40.184.33,120.25.151.105,120.25.157.75,"

I’ve check the addresses in whois database and all of them leads to Hangzhou, Zhejiang, China.

Next to decoding of these addresses, there is UDP socket opening routine and at least two ports are used: 32100 and 32108, where first one is used to connect to these services (all three of them) and port 32108 seem to be the port of camera.

What is my IP address?

So far, I identified at least one request that for sure goes to the hardcoded server, so in fact it is used to call home. This is the so called hello request. It sends nothing at all, as can be seen below:

00000000  f1 00 00 00                                       |....|
00000004

[len] [magic] [typ]

And in response it gets packet like that:

00000000  f1 01 00 10 00 02 7b 3d  xx xx xx xx 00 00 00 00  |......{=........|
00000010  00 00 00 00                                       |....|
00000014

[len] [magic] [sockaddr_in] [typ]

Content of this packet is then cast to sockaddr_in packet and experimentally, I was able to tell that it contains WAN address of requesting side (this is the censored part). I haven’t noticed the original sockaddr_in, that was used to send this hello message in memory, so chances are this is the only purpose of that server.

However, there are a lot of other datagrams crafted from same function as this one (including so called P2P requests), so I cannot guarantee I haven’t missed anything. In my opinion, further dynamic analysis is required. And I want to do one either way.

Other indicators?

Beside these suspicious addresses, I noticed some ioctl usage, but this seem to be quite legit. It just checks configuration of network interfaces (the worst that can happen out of this is sending somewhere MAC address of all interfaces available, that is both WiFi and cellular).

I focused my analysis on libPPPP_API, but beside that, there exists also UFO SDK and Adpcm codec. While Adpcm can be considered safe, as it does not import anything weird, UFO SDK is suspicious, just like PPPP API. Maybe little less, as it does not seem to obfuscate any strings. Nevertheless, it opens some sockets,probably only for RTP, but still.

Conclusion

The most important recommendation is to never trust any source of APKs beside official shops, most notably Google Play, as in it there exist some administration staff, that can react in case of obvious malware (how often and in which cases they do is another story). As can be seen in this case, either app vendor, or a shop is attaching some stuff, which is recognized as malware. In case of Google Play variant (and other Play-cloning shops), things are much better and no alert is raised by VirusTotal.

Second point is how to treat the fact of calling home, especially in situation where app vendor tries to hide the fact by obfuscating the string that contains the addresses. In my opinion, it is enough to not let that app do anything outside some closed network (with no means of sending packet to the internet), at least to be able to check the behavior in practice. And this is probably what I am going to try next. However, I can imagine that I will be using the app, despite that, but after banning these three servers. I can also understand that most people will ignore that fact (or even more – will be unaware of that), as in today’s world there is such a huge amount of services that are gathering our data, that it is impossible to block them all.

To sum up, if you want to use it, download from Google Play and keep in mind, some data might leak somewhere on he Chinese coast.

PS: I am not planning to publish complete API, which I am in theory able to reverse engineer. However, expect some quick summary of what I already know about the protocl πŸ™‚

How to convert Android resource ID back into its name

Few weeks ago I made an attempt to reverse engineer some obscure Android APK. It was available only through some Chinese shop, obviously described in only one language there. Unfortunately, it turned out that every tool designed for reverse engineering APK files outputted source with mysterious resource IDs, as plain integers, which is not the most convenient way to read them. Therefore I started looking for any way to find some meaningful name from these ids. At the end of my development effort I found out, there is one file that usually might be used for that purpose – res/values/public.xml, as produced by apktool (if I remember correctly). However, according to its name it contains only public resources, so some of them are missing there (in my case at least some drawable type resources were missing). Therefore, I am publishing my program to do things even more reliably.

arscutils

This program requires my library created together, but which is separate project – libarsc. It is available, as usually through Github and also as a package to be downloaded from PyPI. Just type:

pip install libarsc

with proper privileges.

This is meant to be utility package, but for now it contains only one such tool: rid2name. Its purpose is to convert resource ID into name in format matching the one, programmers use in their Android apps. Therefore with its help it should be possible to make reversed program looks more similar to compiler input on the developer side. To use it, just feed it with resources.arsc file as first parameter, resource id as second one and optionally one of: fqdn, xmlid or json as third one. As a result you should get resource name as used in Java source, XML files or JSON meant for further processing. Example runs are:

$ python3 rid2name.py ../com.g_zhang.iMiniCam_39/original/resources.arsc 0x7f070000 xmlid
@com.g_zhang.iMiniCam:string/app_name
$ python3 rid2name.py ../com.g_zhang.iMiniCam_39/original/resources.arsc 0x7f070000 fqdn
com.g_zhang.iMiniCam.R.string.app_name
$ python3 rid2name.py ../com.g_zhang.iMiniCam_39/original/resources.arsc 0x7f070000 json
{"package": "com.g_zhang.iMiniCam", "type": "string", "key": "app_name"}
$ python3 rid2name.py ../com.g_zhang.iMiniCam_39/original/resources.arsc 0x7f070000
com.g_zhang.iMiniCam.R.string.app_name

There is also quite convenient interface inside Python source, so the file should be includable into bigger projects.

I have to give one warning now: my implementation of ARSC format is not complete, so some things might not work as expected, but from my tests of libarsc, out of 12 ARSC files, extracted from random APK files, I found on my phone, 3 of them failed (returned different MD5) to rebuild into exactly same binary (did not checked exactly what happened there).

libarsc

This is library that was used underneath arscutils. It is able to parse most of the ARSC file, with special treatment of naming part, that allowed creation of rid2name. It is still missing some important parts and if there will be need from my side to extracting some more things, I will implement the rest of the specification. I am also open to any pull requests to my Github repo.

Future

As you might noticed in usage listing, there is a topic of reverse engineering app, which name was shown there. In case I found something interesting inside, there will be another article, where I will try to share my findings.

Edit: my mistake, this is not my target app, just the package name was similar.

Setting up new v3 Hidden Service with ultimate security: Part 4: Installing client certificates to Firefox for Android

This post is a part of Tor v3 tutorial. Other parts are:

  1. Hidden Service setup
  2. PKI and TLS
  3. Client Authentication
  4. Installing client certificates to Firefox for Android

As we now have Hidden Service, requiring clients to authenticate themselves with proper certificate, it would be great to be able to use Android device to access the service. As I shown before, on desktop Firefox it was quite trivial. Unfortunately, things are different on Android. Mobile Firefox does not have any interface for adding any certificates. Furthermore, unlike Chrome, it does not use default Android certificate vault, providing it own instead. On the other hand, under the hood it is more or less the same Firefox, so the support itself is present. Therefore, we need to hack into Firefox internal databases and add the certificate there. In this part, I will show, how to do that.

Caution: similarly to desktop browser, you should not add any random certificates to your main browser. It is even worse idea to do the same with Orfox, as it might allow attackers to reveal your identity. Newer Androids have ability to create user accounts, furthermore Firefox has profiles features, just like on desktop, but harder to use. If you want to do, what is described here, separating this configuration from any other is first thing to do.

Installing CA certificate

Before we do that with user certificate, let’s start with CA. It is way easier, as Firefox has convenient feature allowing to install certificates by browsing them. All we need to provide is a valid MIME type – application/x-x509-ca-cert. So, all we need is some webserver, which we will configure to treat files with extension .crt to be treated as mentioned type. Just after opening certificate file, Firefox should ask if you are sure about adding the certificate and allow you to choose for what purpose it will be used. It also allows to view the certificate to make sure, it is the one we intended to add.

At first, check the certificate
Then use it only for website identification

In theory there is very similar MIME for user certs – application/x-x509-user-cert, but for some reason, what Firefox says after opening this type of file is:

“Couldn’t install because the certificate file couldn’t be read”

And the same effect is, no matter if the file is password protected or not.

Installing client certificate

  1. Go to /data/data/org.mozilla.firefox/files/mozilla on Android device (root required)
  2. Locate default Firefox profile. If there is only one directory in format [bloat].profile, this is it. If not, file profiles.ini should contain only one profile with Default=1. This is what we are looking for
  3. Download files cert9.db and key4.db to Linux machine
  4. Use pk12util to insert certificate into database:
$ pk12util -i [filename].p12 -d.
Enter password for PKCS12 file:
pk12util: no nickname for cert in PKCS12 file.
pk12util: using nickname: [email] - r4pt0r Test Systems
pk12util: PKCS12 IMPORT SUCCESSFUL
  1. Upload files back to Android. Make sure Firefox is not running
  2. Test it by opening your hidden service with Firefox. You should see messages similar to these:
Request for identification
Certificate details
Finally, working cgit via tor!

Setting up new v3 Hidden Service with ultimate security: Part 3: Client Authentication

This post is a part of Tor v3 tutorial. Other parts are:

  1. Hidden Service setup
  2. PKI and TLS
  3. Client Authentication
  4. Installing client certificates to Firefox for Android

As we now have working Public Key Infrastructure, we are ready to use it for more than encrypting traffic (which is already encrypted by Tor). We can very easily turn on client verification on our server. This will prevent anybody not having valid certificate issued by us from visiting our hidden webpage – just in case hiding domain name in hidden services version 3 leaks the name somehow (which should not happen anymore in v3). In this part we will issue client certificate (the procedure is almost identical to server certificate), then configure httpd to require client identification and finally configure Firefox to try sending the certificate. Let’s go!

Issuing user certificate

In my case tmp directory emulated client machine and ca is my Cerificate Authority, which issues certificates. We start by creating request on client side, then sign it on CA side.

$ mkdir tmp
$ cd tmp
$ openssl genrsa -out v3l0c1r4pt0r@gmail.com.key.pem 4096
Generating RSA private key, 4096 bit long modulus
........++
..............................................++
e is 65537 (0x010001)
$ openssl req -config ../ca/intermediate/openssl.cnf -key v3l0c1r4pt0r@gmail.com.key.pem -new -sha256 -out v3l0c1r4pt0r@gmail.com.csr.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:PL
State or Province Name [England]:lodzkie
Locality Name []:
Organization Name [Alice Ltd]:r4pt0r Test Systems
Organizational Unit Name []:
Common Name []:v3l0c1r4pt0r@gmail.com
Email Address []:v3l0c1r4pt0r@gmail.com
$ chmod 400 v3l0c1r4pt0r@gmail.com.*.pem
$ cp v3l0c1r4pt0r@gmail.com.csr.pem ../ca/intermediate/csr/
$ cd ../ca
$ openssl ca -config intermediate/openssl.cnf -extensions usr_cert -days 375 \
> -notext -md sha256 -in intermediate/csr/v3l0c1r4pt0r@gmail.com.csr.pem \
> -out intermediate/certs/v3l0c1r4pt0r@gmail.com.cert.pem
Using configuration from intermediate/openssl.cnf
Enter pass phrase for /home/r4pt0r/Research/cubie/newtor/ca/intermediate/private/intermediate.key.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4097 (0x1001)
        Validity
            Not Before: Feb 27 17:14:40 2018 GMT
            Not After : Mar  9 17:14:40 2019 GMT
        Subject:
            countryName               = PL
            stateOrProvinceName       = lodzkie
            organizationName          = r4pt0r Test Systems
            commonName                = v3l0c1r4pt0r@gmail.com
            emailAddress              = v3l0c1r4pt0r@gmail.com
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Cert Type:
                SSL Client, S/MIME
            Netscape Comment:
                OpenSSL Generated Client Certificate
            X509v3 Subject Key Identifier:
                ED:24:E6:FF:1D:9B:61:AC:29:66:39:59:FB:5D:77:25:F7:A3:55:47
            X509v3 Authority Key Identifier:
                keyid:3D:AC:8E:21:79:5A:AD:7B:7C:92:92:65:B7:19:D0:E8:00:0E:50:70

            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Client Authentication, E-mail Protection
Certificate is to be certified until Mar  9 17:14:40 2019 GMT (375 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
$ cd ../tmp
$ cp ../ca/intermediate/certs/v3l0c1r4pt0r@gmail.com.cert.pem ./
$ openssl pkcs12 -export -inkey v3l0c1r4pt0r@gmail.com.key.pem -in v3l0c1r4pt0r@gmail.com.cert.pem -out v3l0c1r4pt0r@gmail.com.p12
Enter Export Password:
Verifying - Enter Export Password:

Last step was packaging certificate and key into PKCS#12 container. That is for securing key (we can encrypt it with password), and is a form required by Firefox. After creation of .p12 (and verifying it is fine), we can (and SHOULD) delete source files, as they are not protected in any way.

Configuring httpd to require user certificate

To enforce client verification, following lines must be added to virtual host configuration, in our case it might go just after SSL certificate file paths.

    SSLVerifyClient require
    SSLVerifyDepth 2

We have to reload httpd for changes to take effect.

Installing certificate to Firefox

At last, to start using newly generated certificate, we should install it to Firefox. The procedure is similar to the one with CA certificate. We need to open Certificate Manager window. Then, instead of going to Authorities, we go to Your Certificates. Then we click on Import and select .p12 file.

Certificate Manager / Your Certificates

If the file has password, Firefox will ask for it and after successfully reading the content. If everything went well, you should see your certificate on the list. Now we can try connecting to our hidden service. We should see the window like this:

Server asks for client’s identity

Finally, after confirmation, you should see your hidden service content. Congrats!

Setting up new v3 Hidden Service with ultimate security: Part 2: PKI and TLS

This post is a part of Tor v3 tutorial. Other parts are:

  1. Hidden Service setup
  2. PKI and TLS
  3. Client Authentication
  4. Installing client certificates to Firefox for Android

After setting up working Tor hidden service, the next step to ultimate security is having properly implemented Public Key Infrastructure (PKI). For this step, there are a lot of tutorials already existing and there is not much that needs to be added to them. Personally, I was using tutorial available here for the second time now and I find it very well-written. Because I am going to follow this tutorial, I will just post commands that have to be executed.

Before starting, I have to add one important remark. To make our PKI really secure one, it is crucial to have root CA air-gapped, that is device, on which it will be generated should be disconnected permanently from the internet. Good candidate for such a device might be some old laptop or Raspberry Pi Zero, as it lacks Ethernet port and anything reasonable to connect to internet. It is also important to store generated certificate in a safe place and secure it with strong non-dictionary password, which will be saved only in our mind.

If the requirements are fulfilled, we can start the setup. Below are commands to type as well as output from them, for easier determination of whether the commands were successful or not.

Preparations

At first, we need to create following directory structure:

ca
β”œβ”€β”€ [drwxr-xr-x]  certs
β”œβ”€β”€ [drwxr-xr-x]  crl
β”œβ”€β”€ [-rw-r--r--]  index.txt
β”œβ”€β”€ [drwxr-xr-x]  intermediate
β”‚Β Β  β”œβ”€β”€ [drwxr-xr-x]  certs
β”‚Β Β  β”œβ”€β”€ [drwxr-xr-x]  crl
β”‚Β Β  β”œβ”€β”€ [drwxr-xr-x]  csr
β”‚Β Β  β”œβ”€β”€ [-rw-r--r--]  index.txt
β”‚Β Β  β”œβ”€β”€ [drwxr-xr-x]  newcerts
β”‚Β Β  β”œβ”€β”€ [drwx------]  private
β”‚Β Β  └── [-rw-r--r--]  serial
β”œβ”€β”€ [drwxr-xr-x]  newcerts
β”œβ”€β”€ [drwx------]  private
└── [-rw-r--r--]  serial

And file content is (enclosed between pipe symbols: |):

./index.txt: ||
./intermediate/index.txt: ||
./intermediate/serial: |1000
|
./serial: |1000
|

Then, we need to save this file into root/openssl.cnf and this file into root/intermediate/openssl.cnf. Inside them, the only thing that have to be changed is dir property in CA_default section. Use absolute path to your directory.

Root CA

Note: when giving values for certain fields, better give some country, state (I have just checked it’s necessary), ON, most importantly, Common Name and e-mail. Just in case some program will check if they exists.

$ openssl genrsa -aes256 -out private/ca.key.pem 8192
Generating RSA private key, 8192 bit long modulus
.................++
....++
e is 65537 (0x010001)
Enter pass phrase for private/ca.key.pem:
Verifying - Enter pass phrase for private/ca.key.pem:
$ chmod 400 private/ca.key.pem
$ openssl req -config openssl.cnf -key private/ca.key.pem -new -x509 -days 7300 \
> -sha256 -extensions v3_ca -out certs/ca.cert.pem
Enter pass phrase for private/ca.key.pem:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:PL
State or Province Name [England]:lodzkie
Locality Name []:
Organization Name [Alice Ltd]:r4pt0r Test Systems
Organizational Unit Name []:
Common Name []:r4pt0r Root CA
Email Address []:admin@example.com
$ chmod 444 certs/ca.cert.pem
$ openssl x509 -noout -text -in certs/ca.cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            9a:16:72:e8:ac:81:cd:be
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = PL, ST = lodzkie, O = r4pt0r Test Systems, CN = r4pt0r Root CA, emailAddress = admin@example.com
        Validity
            Not Before: Feb 20 17:22:27 2018 GMT
            Not After : Feb 15 17:22:27 2038 GMT
        Subject: C = PL, ST = lodzkie, O = r4pt0r Test Systems, CN = r4pt0r Root CA, emailAddress = admin@example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (8192 bit)
                Modulus:
                    00:dd:8c:8f:5d:be:f4:0f:63:91:9c:73:bf:a8:17:
<quite a lot of data>
                    6d:c1:3f:5c:05
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                29:53:8A:D2:ED:CF:35:C2:BB:A8:12:06:01:74:99:A3:B8:E5:DC:FE
            X509v3 Authority Key Identifier:
                keyid:29:53:8A:D2:ED:CF:35:C2:BB:A8:12:06:01:74:99:A3:B8:E5:DC:FE

            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
    Signature Algorithm: sha256WithRSAEncryption
         a9:6d:9e:d4:bf:1b:55:d8:f0:b5:e9:9d:56:e8:58:04:d6:c3:
<quite a lot of data>
         89:50:26:4f:3e:93:95:06:c7:38:08:c7:16:0e:d2:a2

Intermediate CA

$ openssl genrsa -aes256 -out intermediate/private/intermediate.key.pem 8192
Generating RSA private key, 8192 bit long modulus
.++
........................................................................................................................................................................................................................................................................................++
e is 65537 (0x010001)
Enter pass phrase for intermediate/private/intermediate.key.pem:
Verifying - Enter pass phrase for intermediate/private/intermediate.key.pem:
$ chmod 400 intermediate/private/intermediate.key.pem
$ openssl req -config intermediate/openssl.cnf -new -sha256 \
> -key intermediate/private/intermediate.key.pem -out intermediate/csr/intermediate.csr.pem
Enter pass phrase for intermediate/private/intermediate.key.pem:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:PL
State or Province Name [England]:lodzkie
Locality Name []:
Organization Name [Alice Ltd]:r4pt0r Test Systems
Organizational Unit Name []:
Common Name []:r4pt0r Intermediate CA
Email Address []:admin@example.com
$ openssl ca -config openssl.cnf -extensions v3_intermediate_ca -days 3650 \
> -notext -md sha256 -in intermediate/csr/intermediate.csr.pem -out intermediate/certs/intermediate.cert.pem
Using configuration from openssl.cnf
Enter pass phrase for ca/private/ca.key.pem:
Can't open ca/index.txt.attr for reading, No such file or directory
140341269315520:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:74:fopen('ca/index.txt.attr','r')
140341269315520:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:81:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4096 (0x1000)
        Validity
            Not Before: Feb 20 17:35:09 2018 GMT
            Not After : Feb 18 17:35:09 2028 GMT
        Subject:
            countryName               = PL
            stateOrProvinceName       = lodzkie
            organizationName          = r4pt0r Test Systems
            commonName                = r4pt0r Intermediate CA
            emailAddress              = admin@example.com
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                3D:AC:8E:21:79:5A:AD:7B:7C:92:92:65:B7:19:D0:E8:00:0E:50:70
            X509v3 Authority Key Identifier:
                keyid:29:53:8A:D2:ED:CF:35:C2:BB:A8:12:06:01:74:99:A3:B8:E5:DC:FE

            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
Certificate is to be certified until Feb 18 17:35:09 2028 GMT (3650 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
$ openssl x509 -noout -text -in intermediate/certs/intermediate.cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4096 (0x1000)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = PL, ST = lodzkie, O = r4pt0r Test Systems, CN = r4pt0r Root CA, emailAddress = admin@example.com
        Validity
            Not Before: Feb 20 17:35:09 2018 GMT
            Not After : Feb 18 17:35:09 2028 GMT
        Subject: C = PL, ST = lodzkie, O = r4pt0r Test Systems, CN = r4pt0r Intermediate CA, emailAddress = admin@example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (8192 bit)
                Modulus:
                    00:d4:c9:03:36:4a:dd:3d:ee:ca:bd:c1:d8:fe:51:
<quite a lot of data>
                    5a:ca:74:74:c8:a2:b2:69:0a:0c:c7:f9:d6:8a:58:
                    41:45:73:fc:2b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                3D:AC:8E:21:79:5A:AD:7B:7C:92:92:65:B7:19:D0:E8:00:0E:50:70
            X509v3 Authority Key Identifier:
                keyid:29:53:8A:D2:ED:CF:35:C2:BB:A8:12:06:01:74:99:A3:B8:E5:DC:FE

            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
    Signature Algorithm: sha256WithRSAEncryption
         15:04:2f:85:89:f6:77:82:c4:60:78:f0:4f:ac:39:ad:15:14:
<quite a lot of data>
         7c:71:95:db:16:02:de:01:70:fe:8f:48:94:92:11:1b
$ openssl verify -CAfile certs/ca.cert.pem intermediate/certs/intermediate.cert.pem
intermediate/certs/intermediate.cert.pem: OK
$ cat intermediate/certs/intermediate.cert.pem certs/ca.cert.pem > intermediate/certs/ca-chain.cert.pem
$ chmod 444 intermediate/certs/ca-chain.cert.pem

Server certificate

In the following parts, wherever [domain] appears, it should be changed to hostname of our hidden service.

At first, we need to generate certificate request (CSR) on our server:

$ openssl genrsa -out [domain].onion.key.pem 4096
Generating RSA private key, 4096 bit long modulus
.................++
..............................................................................++
e is 65537 (0x010001)
$ chmod 400 [domain].onion.key.pem
$ openssl req -config ca/intermediate/openssl.cnf \
> -key [domain].onion.key.pem -new -sha256 -out [domain].onion.csr.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:PL
State or Province Name [England]:lodzkie
Locality Name []:
Organization Name [Alice Ltd]:r4pt0r Test Systems
Organizational Unit Name []:
Common Name []:[domain].onion
Email Address []:admin@[domain].onion

Then, we will sign the request with intermediate CA private key, thus issuing the certificate. But first of all, we need to receive the CSR from the server, to intermediate/csr/ directory.

$ openssl ca -config intermediate/openssl.cnf -extensions server_cert -days 375 \
> -notext -md sha256 -in intermediate/csr/[domain].onion.csr.pem -out intermediate/certs/[domain].onion.cert.pem
Using configuration from intermediate/openssl.cnf
Enter pass phrase for ca/intermediate/private/intermediate.key.pem:
Can't open ca/intermediate/index.txt.attr for reading, No such file or directory
139810167087040:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:74:fopen('ca/intermediate/index.txt.attr','r')
139810167087040:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:81:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4096 (0x1000)
        Validity
            Not Before: Feb 20 17:52:13 2018 GMT
            Not After : Mar  2 17:52:13 2019 GMT
        Subject:
            countryName               = PL
            stateOrProvinceName       = lodzkie
            organizationName          = r4pt0r Test Systems
            commonName                = [domain].onion
            emailAddress              = admin@[domain].onion
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Cert Type:
                SSL Server
            Netscape Comment:
                OpenSSL Generated Server Certificate
            X509v3 Subject Key Identifier:
                DD:6E:E8:78:91:B9:F7:F4:0A:06:3F:D2:38:6D:11:4E:3C:D3:BC:E0
            X509v3 Authority Key Identifier:
                keyid:3D:AC:8E:21:79:5A:AD:7B:7C:92:92:65:B7:19:D0:E8:00:0E:50:70
                DirName:/C=PL/ST=lodzkie/O=r4pt0r Test Systems/CN=r4pt0r Root CA/emailAddress=admin@example.com
                serial:10:00

            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
Certificate is to be certified until Mar  2 17:52:13 2019 GMT (375 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
$ openssl x509 -noout -text -in intermediate/certs/[domain].onion.cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4096 (0x1000)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = PL, ST = lodzkie, O = r4pt0r Test Systems, CN = r4pt0r Intermediate CA, emailAddress = admin@example.com
        Validity
            Not Before: Feb 20 17:52:13 2018 GMT
            Not After : Mar  2 17:52:13 2019 GMT
        Subject: C = PL, ST = lodzkie, O = r4pt0r Test Systems, CN = [domain].onion, emailAddress = admin@[domain].onion
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:c5:d3:e2:a0:97:b8:4d:67:22:94:c9:be:17:e3:
<quite a lof of data>
                    49:76:cf
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Cert Type:
                SSL Server
            Netscape Comment:
                OpenSSL Generated Server Certificate
            X509v3 Subject Key Identifier:
                DD:6E:E8:78:91:B9:F7:F4:0A:06:3F:D2:38:6D:11:4E:3C:D3:BC:E0
            X509v3 Authority Key Identifier:
                keyid:3D:AC:8E:21:79:5A:AD:7B:7C:92:92:65:B7:19:D0:E8:00:0E:50:70
                DirName:/C=PL/ST=lodzkie/O=r4pt0r Test Systems/CN=r4pt0r Root CA/emailAddress=admin@example.com
                serial:10:00

            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
    Signature Algorithm: sha256WithRSAEncryption
         b0:92:d9:d5:3b:31:38:f6:b8:51:1f:41:e9:f7:d8:e6:33:67:
<quite a lot of data>
         ee:c4:eb:19:86:69:00:26:8d:04:7b:97:0b:8f:f5:76
$ openssl verify -CAfile intermediate/certs/ca-chain.cert.pem intermediate/certs/[domain].onion.cert.pem
intermediate/certs/[domain].onion.cert.pem: OK

httpd configuration

Finally, we can use generated files to set up HTTPS encryption on webserver. For this, I am using httpd as it is the most common webserver in use. We need following files:

  1. [domain].onion.key.pem – this is private key, that will be used to set up TLS session
  2. [domain].onion.cert.pem – this is certificate that will prove our identity, so web browser will not display any warnings as long as we will have CA certificate installed
  3. ca-chain.cert.pem – this is chain of certificates we created together with intermediate CA, that consists of both CAs – root and intermediate

Below is httpd configuration file, after enabling TLS:

Listen 666

<VirtualHost *:666>
    ServerAdmin admin@re-ws.pl
    DocumentRoot "/home/r4pt0r/tor/hs/public_html"
    ServerName 192.168.253.4
    ErrorLog "[path]/tor/hs/error_log"
    CustomLog "[path]/tor/hs/access_log" common
    ScriptAlias /cgit/ "/usr/lib/cgit/cgit.cgi/"
    Alias /cgit-css "/usr/share/webapps/cgit/"
    SSLEngine on
    SSLCertificateFile "[path]/tor/hs/tls/[domain].onion.cert.pem"
    SSLCertificateKeyFile "[path]/tor/hs/tls/[domain].onion.key.pem"
    SSLCACertificateFile "[path]/tor/hs/tls/ca-chain.cert.pem"
</VirtualHost>

As can be seen above, all necessary files had been moved to tls directory of our hidden service main directory.

Afterwards, one slight change is needed in torrc file:

HiddenServicePort 443 127.0.0.1:666

From now on, we need to use https://[domain].onion to visit our site, as it is now TLS-encrypted and using port 443, which is default for HTTPS. For convenience, we can set up another httpd vhost on different port, that will redirect all HTTP traffic through HTTPS and link it to port 80, so remembering about https in address will not be necessary. But, it is only optional, so I will leave it as an exercise to the reader.

Firefox

From this point it is useful to have Firefox that is not constantly reminding about insecure connection. To prevent this, we should install CA certificate into Firefox. One remark here: as we are going to hack Firefox to trust our certificate, now our whole browsing through that instance of Firefox relies on our CAs private key. So, it is best to not use the same instance for anything else unless you are really sure, the private keys for both root and intermediate are perfectly secure.

To install the certificate, follow the screenshots below:

On preferences page, go to security and scroll all the way down to View Certificates button
On Authorities tab, click Import and select your CA certificate
Confirm that this certificate will be able to identify websites
Finally, we are secure and no exclamation mark appears!