Setting up new v3 Hidden Service with ultimate security: Part 3: Client Authentication

This post is a part of Tor v3 tutorial. Other parts are:

  1. Hidden Service setup
  2. PKI and TLS
  3. Client Authentication
  4. Installing client certificates to Firefox for Android

As we now have working Public Key Infrastructure, we are ready to use it for more than encrypting traffic (which is already encrypted by Tor). We can very easily turn on client verification on our server. This will prevent anybody not having valid certificate issued by us from visiting our hidden webpage – just in case hiding domain name in hidden services version 3 leaks the name somehow (which should not happen anymore in v3). In this part we will issue client certificate (the procedure is almost identical to server certificate), then configure httpd to require client identification and finally configure Firefox to try sending the certificate. Let’s go!

Issuing user certificate

In my case tmp directory emulated client machine and ca is my Cerificate Authority, which issues certificates. We start by creating request on client side, then sign it on CA side.

$ mkdir tmp
$ cd tmp
$ openssl genrsa -out v3l0c1r4pt0r@gmail.com.key.pem 4096
Generating RSA private key, 4096 bit long modulus
........++
..............................................++
e is 65537 (0x010001)
$ openssl req -config ../ca/intermediate/openssl.cnf -key v3l0c1r4pt0r@gmail.com.key.pem -new -sha256 -out v3l0c1r4pt0r@gmail.com.csr.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:PL
State or Province Name [England]:lodzkie
Locality Name []:
Organization Name [Alice Ltd]:r4pt0r Test Systems
Organizational Unit Name []:
Common Name []:v3l0c1r4pt0r@gmail.com
Email Address []:v3l0c1r4pt0r@gmail.com
$ chmod 400 v3l0c1r4pt0r@gmail.com.*.pem
$ cp v3l0c1r4pt0r@gmail.com.csr.pem ../ca/intermediate/csr/
$ cd ../ca
$ openssl ca -config intermediate/openssl.cnf -extensions usr_cert -days 375 \
> -notext -md sha256 -in intermediate/csr/v3l0c1r4pt0r@gmail.com.csr.pem \
> -out intermediate/certs/v3l0c1r4pt0r@gmail.com.cert.pem
Using configuration from intermediate/openssl.cnf
Enter pass phrase for /home/r4pt0r/Research/cubie/newtor/ca/intermediate/private/intermediate.key.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4097 (0x1001)
        Validity
            Not Before: Feb 27 17:14:40 2018 GMT
            Not After : Mar  9 17:14:40 2019 GMT
        Subject:
            countryName               = PL
            stateOrProvinceName       = lodzkie
            organizationName          = r4pt0r Test Systems
            commonName                = v3l0c1r4pt0r@gmail.com
            emailAddress              = v3l0c1r4pt0r@gmail.com
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Cert Type:
                SSL Client, S/MIME
            Netscape Comment:
                OpenSSL Generated Client Certificate
            X509v3 Subject Key Identifier:
                ED:24:E6:FF:1D:9B:61:AC:29:66:39:59:FB:5D:77:25:F7:A3:55:47
            X509v3 Authority Key Identifier:
                keyid:3D:AC:8E:21:79:5A:AD:7B:7C:92:92:65:B7:19:D0:E8:00:0E:50:70

            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Client Authentication, E-mail Protection
Certificate is to be certified until Mar  9 17:14:40 2019 GMT (375 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
$ cd ../tmp
$ cp ../ca/intermediate/certs/v3l0c1r4pt0r@gmail.com.cert.pem ./
$ openssl pkcs12 -export -inkey v3l0c1r4pt0r@gmail.com.key.pem -in v3l0c1r4pt0r@gmail.com.cert.pem -out v3l0c1r4pt0r@gmail.com.p12
Enter Export Password:
Verifying - Enter Export Password:

Last step was packaging certificate and key into PKCS#12 container. That is for securing key (we can encrypt it with password), and is a form required by Firefox. After creation of .p12 (and verifying it is fine), we can (and SHOULD) delete source files, as they are not protected in any way.

Configuring httpd to require user certificate

To enforce client verification, following lines must be added to virtual host configuration, in our case it might go just after SSL certificate file paths.

    SSLVerifyClient require
    SSLVerifyDepth 2

We have to reload httpd for changes to take effect.

Installing certificate to Firefox

At last, to start using newly generated certificate, we should install it to Firefox. The procedure is similar to the one with CA certificate. We need to open Certificate Manager window. Then, instead of going to Authorities, we go to Your Certificates. Then we click on Import and select .p12 file.

Certificate Manager / Your Certificates

If the file has password, Firefox will ask for it and after successfully reading the content. If everything went well, you should see your certificate on the list. Now we can try connecting to our hidden service. We should see the window like this:

Server asks for client’s identity

Finally, after confirmation, you should see your hidden service content. Congrats!

Setting up new v3 Hidden Service with ultimate security: Part 1: Hidden Service setup

This post is a part of Tor v3 tutorial. Other parts are:

  1. Hidden Service setup
  2. PKI and TLS
  3. Client Authentication
  4. Installing client certificates to Firefox for Android

As a student I was lucky to have unlimited private Git repositories on Github, since they introduced that to their first paid plan. Unfortunately, I don’t have access to educational e-mail anymore, so I won’t be able to renew the service. This leads to a need to have that feature migrated to somewhere else. Some time ago, I installed cgit and gitolite on my single board computer (SBC). But, because of Github, there was no need to use that. Now it seems like a good replacement to Github’s Developer plan.

Few weeks ago, there was interesting event – Tor Project introduced new version of their Hidden Services – v3, which changes length of hidden service address in .onion domain and disables “feature” enabling some nodes in the network to index all existing service addresses. This seems like a good moment to give it a try and check, how fast (or rather how slow) will be the solution providing git through Tor on few-year-old SBC. By the way, I will show, how to configure things with maximum security in mind.

Disclaimer: I am not a person with deep knowledge of inner workings of Tor network, so I strongly encourage you to read thing or two, about how to use it safely. This article might contain errors that might reveal your identity, especially when used together with not-self-owned hidden services.

Prerequisites

Let’s start with summary of what we will need to make Tor v3 work:

  • tor in version 0.3.2.9 or higher
  • alternatively Tor Browser 7.5 or higher
  • for Android: Orbot and Orfox (at the moment of writing this, there is no support in current version of Orbot, so custom compilation is required – I am using Termux to provide tor binary)
  • httpd or any other HTTP server, able to provide service with only one vhost on separate TCP port

Because of the way, I am planning to configure hidden service in future, it might be a good idea to set up separate Tor browser at this moment, dedicated to this service, if it is going to be production configuration. If this is just an experiment, this advice could safely be ignored. However it is good to know, how to undo any modifications to the browser that will be done in the next parts.

httpd

What we need to do is to listen on localhost, on some random TCP port. Then we will set up httpd to provide only one virtual host on this custom port. It would be perfect to disable any other vhosts as our hidden service will work also as non-hidden service for local users, so if other service is buggy and allows to connect to other local services (see e.g. DNS rebinding), at least address of our hidden service will be compromised.

I have following configuration:

Listen 666

<VirtualHost *:666>
    ServerAdmin [email]@[domain]
    DocumentRoot "[path]/public_html"
    ServerName [domain].onion
    ErrorLog "[path]/error_log"
    CustomLog "[path]/access_log" common
</VirtualHost>

<Directory "[path]/public_html">
    DirectoryIndex index.html index.php index.txt
    AllowOverride All
    Options FollowSymlinks
    Require all granted
</Directory>

Furthermore, httpd must be able to traverse to public_html directory, so every directory from public_html up to root must have execute privilege for http user and directory itself as well as its content must be available (or better owned) by http.

After that and after starting httpd, it should be possible to visit http://localhost:666 via web browser and see content of public_html directory. If this is true, we can move on to tor configuration.

tor

SocksPort auto

HiddenServiceDir /etc/tor/hsv3
HiddenServiceVersion 3
HiddenServicePort 80 127.0.0.1:666

SafeLogging 0
Log notice stdout
Log notice file /etc/tor/hsv3/hs.log
Log info file /etc/tor/hsv3/hsinfo.log

Now, on the first startup of tor,  it should create keys for our new hidden service. We can look into /etc/tor/hsv3/hostname to see the .onion address. It is good idea to set key files and hostname file as readable as only user running tor service. In case of service started by systemd, this will probably be tor by default.

After starting tor service (systemctl start tor in case of systemd), we can check if everything works properly by visiting our hidden service with tor-enabled browser (using tor 0.3.2.9 or higher). That’s it.

Firefox for Android

At the time of writing this article there is still no upgrade for Orbot app, providing GUI interface for tor. Because of that, it might be required to use ordinary Firefox to use tor as a proxy, which is generally bad idea for connecting to any hidden services, because of privacy and anonymity. Fortunately, we can live with revealing our identity to ourselves 🙂 so we can do it only this single time.

What we need to change are following configuration options, available under about:config page:

  • network.proxy.socks to localhost
  • network.proxy.socks_port to 9050
  • network.proxy.socks_remote_dns to true
  • network.proxy.socks_version to 5, if any other (should be default)
  • network.proxy.type to 1 (0 means no proxy, 5 is system proxy)

Conclusion

Now we are ready to use our hidden service, from both desktop and mobile. Still, we use only HTTP protocol, which is not a big problem, as tor already provides encryption. Neverheless our next goal would be to configure HTTPS. And then we will configure client authentication for ultimate security of our hidden service.

LKV373A: porting objdump

This article is part of series about reverse-engineering LKV373A HDMI extender. Other parts are available at:

After part number four, we already have ELF file, storing all the data we found in firmware image, described in a way that should make our analysis easier. Moreover, we have ability to define new symbols inside our ELF file. The next step is to add support for our custom architecture into objdump and this is what I want to show in this tutorial.

Finding best architecture to copy

If we want to set up new architecture in objdump code, we need to learn interfaces that need to be implemented. It would be easier if we can use some existing code to do so. After some looking into the binutils’ code I learned that what is of special interest are bfd and opcodes libraries. They contain code dedicated to particular architectures. The first one seem to be related to object file handling (which in our case is ELF), so we should not tinker with it too much. Second one is related to disassembling binary programs, so is what we are looking for.

I did some quick examination of source code related to popular architectures and it seems not to be easy to adjust to our needs. Architecture I found to be best suitable for modification is Microblaze. Its source seem to be quite well-written, clean and short. Also from my research of architecture name for LKV373A (part 2, failed by the way) I also remember it is quite similar to the one present in LKV373A, so it is even better decision to use it.

Compiling objdump for target architecture

At first it is useful to learn how to compile objdump, so it will be able to disassemble program written for our target. Microblaze is not really a mainstream architecture, so there aren’t many programs compiled for it available online after typing 'microblaze program elf' into usual search engine. However, I was able to find 2 of them, so I was able to verify that compilation worked. If you can’t find any, I uploaded these to MEGA, so they can serve as test cases. First one is minimal valid file, the other one is quite huge.

Compilation is very easy. The only thing that needs to be done beside usual ./configure && make && make install is adding target option to configure script. So, the script looks as follows:

./configure --target=microblaze-elf

Of course, install step can safely be skipped as well as compilation of other tools, beside objdump. objdump itself seem to be built using make binutils/objdump. However it can’t be build successfully using that shortcut, so whole binutils package must be configured the way, everything not buildable is excluded from the build.

Setting up own architecture

Next step is to add support for our brand new, custom architecture to binutils’ configuration files and copy microblaze sources, so they will simulate our architecture, until we will write our own implementation. Then it should be possible to test objdump again, against our sample microblaze programs and disassembly should still work.

Even without any modification to binutils’ source or configs, it should be possible to configure it for any random architecture. The only constraint is format of the target string: ARCH-OS-FORMAT, where FORMAT is most likely to be elf. So, if we pass lkv373a-unknown-elf as target, it will work. -unknown part is usually skipped and this will not work. If we need it to work, config.sub must be modified. config.sub is used to convert any string, passed to configure into canonical form, so in our case lkv373a-unknown-elf. If it detects, that it is already in canonical form, it does nothing.

Final configure command will be slightly more complex, as we have to disable some parts, that are not of our interest and requires additional effort to work:

./configure --target=lkv373a-unknown-elf --disable-gas --disable-ld --disable-gdb

Although passing something random as target option works on configure stage, it will obviously fail on make stage. What make is doing at first is configuring all the sublibraries. What is of our interest is bfd and opcodes. And the first one fails. So this is the first problem, we need to get rid of.

bfd/config.bfd

The purpose of this file is to set some environment variables depending on target architecture. If it does not know the architecture, it returns error to caller, which is probably bfd’s configure script, called by make. According to documentation in file header, it sets following variables:

  1. targ_defvec – default vector. This links target to list of objects that will provide support for ELF file built for specific architecture (stored in bfd/configure.ac)
  2. targ_selvecs – list of other selected vectors. Useful e.g. when we need support for both 32- and 64-bit ELFs. Not needed here.
  3. targ64_selvecs – 64-bit related stuff. Used when target can be both 32- and 64-bit, meaningless in our case.
  4. targ_archs – name of the symbol storing bfd_arch_info_type structure. It provides description of architecture to support.
  5. targ_cflags – looks like some hack to add extra CFLAGS to compiler. We don’t care.
  6. targ_underscore – not sure what it is, should have no impact on our goals (possible values are yes or no)

To sum up, what we need to do on this step is to define default vector, we will later add to configure.ac and set name of architecture description structure. The structure itself will be defined later. Finally, I ended up with the following patch:

@@ -173,6 +173,7 @@ hppa*)     targ_archs=bfd_hppa_arch ;;
 i[3-7]86)   targ_archs=bfd_i386_arch ;;
 i370)     targ_archs=bfd_i370_arch ;;
 ia16)     targ_archs=bfd_i386_arch ;;
+lkv373a)  targ_archs=bfd_lkv373a_arch ;;
 lm32)           targ_archs=bfd_lm32_arch ;;
 m6811*|m68hc11*) targ_archs="bfd_m68hc11_arch bfd_m68hc12_arch bfd_m9s12x_arch bfd_m9s12xg_arch" ;;
 m6812*|m68hc12*) targ_archs="bfd_m68hc12_arch bfd_m68hc11_arch bfd_m9s12x_arch bfd_m9s12xg_arch" ;;
@@ -924,6 +925,10 @@ case "${targ}" in
     targ_defvec=iq2000_elf32_vec
     ;;

+  lkv373a*-*)
+    targ_defvec=lkv373a_elf32_vec
+    ;;
+
   lm32-*-elf | lm32-*-rtems*)
     targ_defvec=lm32_elf32_vec
     targ_selvecs=lm32_elf32_fdpic_vec

bfd/configure.ac

Now we need to define vector, we just declared to use for lkv373a architecture.

505     k1om_elf64_fbsd_vec)         tb="$tb elf64-x86-64.lo elfxx-x86.lo elf-ifunc.lo elf-nacl.lo elf64.lo $elf"; target_size=64 ;;
506     lkv373a_elf32_vec)           tb="$tb elf32-lkv373a.lo elf32.lo $elf" ;;
507     l1om_elf64_vec)              tb="$tb elf64-x86-64.lo elfxx-x86.lo elf-ifunc.lo elf-nacl.lo elf64.lo $elf"; target_size=64 ;;

Unfortunately, as we did modifications to .ac script, we now need to rebuild our configure. From my experience, any tinkering with autohell, after solving one problem, creates 5 more. We need to get into bfd directory and reconfigure project:

cd bfd
autoreconf

Now, if it worked for you, you should definitely go, play some lottery 🙂 . For me it said that I need exactly same version of autoconf as used by binutils’ developers. Because autoconf is so great, probably what I will show now is completely useless for anyone, but hacks I needed to do are at first to add:

20 m4_define([_GCC_AUTOCONF_VERSION], [2.69])

to the beginning of configure.ac file. Then bfd/doc/Makefile.am contains removed cygnus option at the beginning, in AUTOMAKE_OPTIONS, so we need to remove it. After that doing automake --add-missing, as autoreconf suggests, and then again autoreconf should solve the problem. But, as I said, this will probably not work for you. I can only wish you good luck.

(if were following the steps, you might have noticed that autoconf complained about not being in version 2.64 and we overridden version from 2.69 to 2.69 and it worked 🙂 , don’t ask me, why, please!)

After this step, compilation should start (and obviously will fail miserably on bfd as it misses few symbols). Now its time to make bfd compilable.

bfd/elf32-lkv373a.c

This file is meant to provide support for custom features of ELF file. As we don’t have any, we can safely do nothing here. Good template of such file is elf32-m88k.c as it does exactly this.

One thing that seem to be important here is EM value of architecture described. EM is an enum used in ELF file to define target architecture, so it might be required to adjust in our new elf32-lkv373a.c file. By the way definition of this value have to be added to include/elf/common.h:

433 /* LKV373A architecture */
434 #define EM_LKV373A              0x373a

It might also be a good idea to add it to elfcpp/elfcpp.h. To make the file compile, it is necessary to add following to bfd/bfd-in2.h as value of bfd_architecture enum:

2398   bfd_arch_lkv373a,    /* LKV373A */

bfd/archures.c

As we declared bfd_lkv373a_arch as symbol with CPU description structure, we now need to add this declaration to archures.c, as this is the file, where it will be used. We have to add:

611 extern const bfd_arch_info_type bfd_l1om_arch;
612 extern const bfd_arch_info_type bfd_lkv373a_arch;
613 extern const bfd_arch_info_type bfd_lm32_arch;

bfd/targets.c

Similar situation is in targets.c file. Here we have to provide declaration of our vector as bfd_target. This will be another structure, which seem to be generated automatically, so we should not care about it.

704 extern const bfd_target l1om_elf64_fbsd_vec;
705 extern const bfd_target lkv373a_elf32_vec;
706 extern const bfd_target lm32_elf32_vec;

bfd/cpu-lkv373a.c

This last file, we need in bfd, provides bfd_arch_info_type structure and… that’s it! Can be easily borrowed from cpu-microblaze.c with only slight modifications. One thing that needs explanation here is section_align_power. As far as I understand it, it is power of two to which the beginning of the section in memory must be aligned. It should be safe to put 0 here, as we are not going to load our ELF into memory.

This should close the bfd part of initialization. As you can see, there was no development at all to be done here. Let’s now go to opcodes library.

opcodes/configure.ac

At first we need to define objects to build for LKV373A architecture in opcodes library. This is quite similar to what we had to do in configure.ac of bfd library.

282         bfd_iq2000_arch)        ta="$ta iq2000-asm.lo iq2000-desc.lo iq2000-dis.lo iq2000-ibld.lo iq2000-opc.lo" using_cgen=yes ;;
283         bfd_lkv373a_arch)       ta="$ta lkv373a-dis.lo" ;;
284         bfd_lm32_arch)          ta="$ta lm32-asm.lo lm32-desc.lo lm32-dis.lo lm32-ibld.lo lm32-opc.lo lm32-opinst.lo" using_cgen=yes ;;

Hopefully, -dis file will be enough to be implemented. I’ve made a copy from microblaze configuration. The same way we will copy whole source file and any related headers in the next step.

Now, similarly to bfd’s configure.ac, we have to reconfigure it. And again, nobody knows what errors we will encounter.

opcodes/disassemble.c

The only thing that have to be done here is to set pointer of disassemble function. For this following snippets should be added:

53 #define ARCH_lkv373a
255 #ifdef ARCH_lkv373a
256     case bfd_arch_lkv373a:
257       disassemble = print_insn_lkv373a;
258       break;
259 #endif

And to disassemble.h:

62 extern int print_insn_lkv373a           (bfd_vma, disassemble_info *);

opcodes/lkv373a-dis.c

This is, where real stuff will happen. As our goal, for now, is not to make implementation of LKV373A architecture, but rather set everything up, so objdump will build, we can copy source file from microblaze-dis.c. It is also required to copy headers, related to MicroBlaze, used by this file, so:

  • opcodes/microblaze-dis.h
  • opcodes/microblaze-opc.h
  • opcodes/microblaze-opcm.h

And change include directives in them to link to lkv373a file, rather than microblaze ones.

Now, optionally we could change names of any symbols referring to name microblaze, but this should not be required, as original microblaze files should not be included in the build. The only change than need to be done is print_insn_microblaze into print_insn_lkv373a, as this is what we added to disassemble.c.

You should now be able to compile working objdump with LKV373A support (of course with wrong implementation, for now). We can now verify that everything works on slightly modified ELF file for MicroBlaze architecture (EM field must point to LKV373A – value must be 0x373a). Well done!

NOTE: all the steps, done till now are available on tutorial-setup tag in repository on Github.

Functions to implement

Now, finally the real fun starts. Bindings between opcodes library and objdump itself, require at least print_insn_lkv373a to be implemented.

What should happen inside this function is quite simple and can be described in following steps:

  1. Gets bfd_vma and struct disassemble_info (called info below) as parameters
  2. Read raw data containing instructions using info->read_memory_func
  3. Call info->memory_error_func in case of any errors
  4. Use info->fprintf_func to print disassembled instruction into info->stream
  5. Optionally use info->symbol_at_address_func to determine if there is any symbol declared at address decoded from instructions
  6. If symbol exists, call info->print_address_func
  7. Return number of bytes consumed

Following is some documentation, I wrote for easier implementation (mostly translated inline comments), of functions to be called:

  /**
   * \brief Function used to get bytes to disassemble
   *
   * \param memaddr Address of the current instruction
   * \param myaddr Buffer, where the bytes will be stored
   * \param length Number of bytes to read
   * \param dinfo Pointer to info structure
   *
   * \return errno value or 0 for success
   */
  int (*read_memory_func)
    (bfd_vma memaddr, bfd_byte *myaddr, unsigned int length,
     struct disassemble_info *dinfo);
  /**
   * \brief Call if unrecoverable error occurred
   *
   * \param status errno from read_memory_func
   * \param memaddr Address of current instruction
   * \param dinfo Pointer to info structure
   */
  void (*memory_error_func)
    (int status, bfd_vma memaddr, struct disassemble_info *dinfo);
  /**
   * \brief Pointer to fprintf
   *
   * \param stream Pass info->stream here
   * \param char Format string
   * \param ... vargs
   *
   * \return Number of characters printed
   */
  typedef int (*fprintf_ftype) (void *, const char*, ...) ATTRIBUTE_FPTR_PRINTF_2;
  /**
   * \brief Determines if there is a symbol at the given ADDR
   *
   * \param addr Address to check
   * \param dinfo Pointer to info structure
   *
   * \return If there is returns 1, otherwise returns 0
   * \retval 1 If there is any symbol at ADDR
   * \retval 0 If there is no symbol at ADDR
   */
  int (* symbol_at_address_func)
    (bfd_vma addr, struct disassemble_info *dinfo);
  /**
   * \brief Print symbol name at ADDR
   *
   * \param addr Address at which symbol exists
   * \param dinfo Pointer to info structure
   */
  /* Function called to print ADDR.  */
  void (*print_address_func)
    (bfd_vma addr, struct disassemble_info *dinfo);

For easier start of development, this commit can be used as template. You can find effects of implementation according to this description on lkv373a branch of my binutils fork on Github. After this step, you should have working objdump, able to disassemble architecture of your choice.

Alternative way

According to binutils’ documentation, porting to new architectures should be done using different approach. Instead of copying sources from other architectures, developers should write CPU description files (cpu/ directory) and then use CGEN to generate all necessary files. However, I found these files way too complicated comparing to goal, I wanted to achieve, therefore I used the shortcut. In reality, however, this might be a better way, as the final result should be the support for new architecture not only in objdump, but also in e.g. GAS (GNU assembler). If you want to go that way, another useful resource might be description of CPU description language.

Plans for the future

As I am now able to speed up reverse engineering of both instruction set and LKV373A firmware, I am planning to create public repository of my progress and guess operations done by some more opcodes as I already know only few of them. So, I will probably push some more commits to binutils repo as well. I hope this will enable me to gain some more knowledge about LKV373A and allow, me or someone else, to reverse engineer second part of the firmware, which seem to be way more interesting that the one, I was reverse engineering till now.

[Import]Wget with SSL/TLS support for Android

NOTE: This post was imported from my previous blog – v3l0c1r4pt0r.tk. It was originally published on 12th September 2016.

wget dependency tree

Lately I have tried to download some file from a website to my Android smartphone. Simple thing, yeah? Well, not really. Unfortunately mobile browser developers removed many features from their mobile distributions. One of them is a possibility of downloading random page to disk as is. Instead (this is the case at least with Mozilla’s product) they are forcing “Download as PDF” feature. I had a bit of luck, because the file I was trying to download was MP4 movie, which is downloadable, maybe not in an intuitive way, but it is. But before I have found that feature hidden in a player’s context menu, I tried another solution – wget. Since I am great fan of terminals, I have busybox installed on my phone. Those of you, who know what exactly is busybox should know that this is set of lightweight variants of most standard UNIX tools. So, if they are lightweight, they had to cut some part of tool functionality, right? And in case of my busybox’s wget, they cut HTTPS support. And today, it is more likely to encounter site which is only HTTPS than one that is only HTTP, at least when talking about popular sites. So I had to get my own distribution of wget, that will not be such constrained one.

Not to get you bored too much, here you can find binary distribution of what I achieved to compile. It was compiled for ARMv7 platform using NDKr12b and API level 24 (Nougat), so it will probably not work on most of current Android phones, but if you read later, it is probably working on your device or even is outdated. If you are interested in recompiling binaries yourself, you can find detailed how-to in the next part of this article.

Dependencies

Before compiling wget itself, you have to have whole bunch of its dependencies. But at first, you of course need Android compiler. It is distributed as part of NDK and I won’t describe its installation here. Sources of every program compiled here can be grabbed from its official sites (list at the end of this post). The only exception is libtasn1, which required few hacks to be done to make it compile with Android bionic libc. Its source, ported to Android can be get from my github repo.

Let’s start with programs that does not depend on anything. For all projects, the procedure is more or less the same and can be described with simplified bash script:

tar -zxvf program-1.00.tar.gz
mkdir build
mkdir install
cd build
CC=arm-linux-androideabi-gcc AR=arm-linux-androideabi-ar RANLIB=arm-linux-androideabi-ranlib CFLAGS=-pie \
    ../program-1.00/configure --host=arm-linux --prefix=/data/local/root
make
make install DESTDIR=$(dirname `pwd`)/install/
cd ../install
tar -zcvf program.tar.gz *

gmp, libidn and libffi

For these three programs, the procedure above should work without any modification.

nettle

Since nettle depends on gmp, it has to be configured with paths to gmp binaries and headers in its CFLAGS and LDFLAGS variables. They should look like this:

CFLAGS="-pie -I`pwd`/../../gmp/install/data/local/root/include"
LDFLAGS="-L`pwd`/../../gmp/install/data/local/root/lib"

when invoking configure script.

libtasn1

This was the hardest part for me, but should go smoothly now. Script below should do the work correctly:

git clone git@github.com:v3l0c1r4pt0r/android_external_libtasn1.git
mkdir build
mkdir install
cd build
CC=arm-linux-androideabi-gcc AR=arm-linux-androideabi-ar RANLIB=arm-linux-androideabi-ranlib CFLAGS=-pie \
    ../libtasn1/configure --host=arm-linux --prefix=/data/local/root --disable-doc
make
make install DESTDIR=$(dirname `pwd`)/install/
cd ../install
tar -zcvf libtasn1.tar.gz

p11-kit

This is the last dependency of gnutls which is the only, but very important dependency of wget. Just embedding libtasn1 and libffi should do the job well.

CFLAGS="-pie -I`pwd`/../../libtasn1/install/data/local/root/include"
LDFLAGS="-L`pwd`/../../libtasn1/install/data/local/root/lib -L`pwd`/../../libffi/install/data/local/root/lib"

Notice that libffi has no headers, so we add it just to CFLAGS here!

gnutls

This one was more complicated than the rest. As I mentioned above, it is very important to wget functionality. However wget’s dependency on it could probably be turned off, we would not have TLS support then. When compiling it I had some problems that seemed to be serious. There were a few errors while making it, so I had to call make twice and even though it failed. Despite that it seem to work after make install, which obviously failed too. In my case following script did the job:

mkdir build
mkdir install
cd build
CC=arm-linux-androideabi-gcc AR=arm-linux-androideabi-ar RANLIB=arm-linux-androideabi-ranlib \
    CFLAGS="-pie -I`pwd`/../../gmp/install/data/local/root/include -I`pwd`/../../nettle/install/data/local/root/include -I`pwd`/../../libtasn1/install/data/local/root/include -I`pwd`/../../libidn/install/data/local/root/include -I`pwd`/../../p11-kit/install/data/local/root/include" \
    LDFLAGS="-L`pwd`/../../gmp/install/data/local/root/lib -L`pwd`/../../nettle/install/data/local/root/lib -L`pwd`/../../libtasn1/install/data/local/root/lib -L`pwd`/../../libidn/install/data/local/root/lib -L`pwd`/../../p11-kit/install/data/local/root/lib" \
    ../gnutls-3.4.9/configure --host=arm-linux --prefix=/data/local/root --disable-cxx --disable-tools
make || make
make install DESTDIR=$(dirname `pwd`)/install/ || true
cd ../install
tar -zcvf file.tar.gz *

Compilation

Since we should now have all dependencies compiled, we can try compiling wget itself. The procedure here is the same as with dependencies. We just have to pass path to gnutls. And then standard configure, make, make install should work. However if your NDK installation is fairly new and you were not hacking it before, you most likely don’t have <sys/fcntl.h> header and make should complain about that. Luckily Android itself have this header present, but for reason unknown it is kept in include directory directly. To make wget, and any other program that uses it, compile you can just point “sys/” instance to <fcntl.h> with symlink or do something like that:

echo "#include <fcntl.h>" > $TOOLCHAIN/sysroot/usr/include/sys/fcntl.h

where $TOOLCHAIN/sysroot is path at which you have your headers placed. Depending on tutorial you were using for making it work it may have different structure.

Installation

All commands I presented above implies that you have your custom-compiled binaries in “/data/local/root”. I made it that way to have clear separation between default and busybox binaries. If you want to have them somewhere else, you should pass it to configure scripts of all programs you are compiling. After successful compilation of all tools, I have made single tarball containing all compilation output (this file’s link was placed above). Its content can be installed into Android by typing

tar -zxvf wget-with-deps.tar.gz -C/

using adb shell or terminal emulator.

Sources

Below you can find links to sources of all programs nedded to follow this tutorial.

[Import]HDCB – new way of analysing binary files under Linux

NOTE: This post was imported from my previous blog – v3l0c1r4pt0r.tk. It was originally published on 10th February 2016.

As any observer of my projects spotted, most of the biggest projects I do involves binary file analysis. Currently I am working on another one that requires such analysis.
Unfortunately such analysis is not an easy task and anything that will ease this or speed it up is appreciated. Of course there are some tools that will make it a bit easier. One of them is hexdump. Even IDA Pro can make it easier a bit. Despite them I always felt that something is missing here. When creating xSDM and delz utils, I was using hexdump output with LibreOffice document to mark different structure members with different colors. It worked, but selecting 100-byte buffer line by line was just wasting precious time.

Solution

SDC file analyzed by HDCB script

So why not automate whole process? What we really need here is just hexdump output and terminal emulator with color support. And that’s why I’ve made HDCB – HexDump Coloring Book. Basically it is just extension to bash scripting language. Goal was to create simple script that will hide as much of its internals from end-user and allow user to just start it will his shell using old good ./scriptname.ext and that’s it. HDCB is masked as if it was standalone scripting language. It uses shebang, known from bash or python scripts to let user shell know what interpreter to use (#!/usr/bin/env hdcb). Those who are python programmers should recognize usage of env binary.

In fact it is just simple extension to bash language, so users are still able to utilize any features available in bash. Main extensions are two commands: one (define) for defining variables and the other (use) for defining field or array of that defined type. Such scripts should be started with just one argument – file that is meant to be hexdumped and analyzed.

Internals

Bash scripts are just some kind of a cover of real program. Main core of the program is colour utility. It gets unlimited number of parameters grouped in groups of four. They are in order: offset of byte being colored, length of the field, background and foreground colors. As standard input, hexdump output (in fact only hexdump -C or hexdump -Cv are supported) is provided. Program colors the hexdump with rules provided as arguments. This architecture allows clever hacker to build that cover mentioned in virtually any programming language.

Downloads, documentation and more

As usual, program is available on my Github profile. Sources are provided on GPLv3 license so you are free to contribute to the project and you are strongly encouraged to do so or make proposals of a new functions. Program is meant to be expanded according to my future needs, but I will try to implement any good idea. Whole documentation, installation instructions and the like are also available on Github.

[Import]Airlive WN-151ARM UART pinout & root access

NOTE: This post was imported from my previous blog – v3l0c1r4pt0r.tk. It was originally published on 24th November 2015.

Airlive WN-151ARM pinout

For curious ones. Here is pinout of serial connection. As you can see UART pins are at J4 header (should have pin 4 labeled and 1 be square).

J4 header
Num. Function
1 VCC
2 RX
3 TX
4 GND

Edit: Oh, and one more thing: goldpin header, you see in the picture is soldered by me, so do not be surprised if you have to hold wires all the time during the transmission.

Root access

There is also possibility to gain root access without removing the cover and possibly voiding the warranty. You have to connect to router’s AP and enter

http://192.168.1.254/system_command.htm

into your browser (panel authentication required). Now you can execute any command you want with root privileges! So let’s type

/usr/sbin/utelnetd -d &

into Console command field and press Execute button. If everything went well, you should now be able to connect to your router using telnet at its default TCP port 23. After that you should see BusyBox banner and command prompt.

It is worth noting that this hidden console cannot be accessed by unauthorized person, so only router administrator can use this (in theory, in practice there are surely a lot of routers using default credentials and security of httpd binary is unknown).

[Import]CP210x (USB to TTL adapter) for Android devices

NOTE: This post was imported from my previous blog – v3l0c1r4pt0r.tk. It was originally published on 22nd August 2014.

Few days ago, while searching for something interesting I can do with my Android phone I found out that Sony is publishing kernel source of all their (and old SE ones too) ROMs. Beside hacking the kernel itself I was wondering if it is possible to compile additional kernel modules (stock ROM provides only internal WiFi module’s drivers). Well.. it was. The first one I tried was driver of my cheap, Chinese USB to serial adapter based on CP2102 chip.

Prerequisites

  • kernel source – it is important to be the same kernel as the one working on your device. Otherwise it would probably be necessary to boot your system with kernel compiled with source you have and it will not be described in this tutorial. The reason it is important is that between kernel version compatibility is not guaranteed. Different configuration could mess with functionality too (but not tried myself). I have Sony Ericsson device so I downloaded from its official repository.
  • cross-compiler – while compiling programs for Android you definitely should get special version of a compiler because Android is different than usual Linux box so compiler options are different too. The easiest way is to download official NDK which has built-in compilers and easy-to-use script to make standalone toolchain out of them. Same toolchain should also work as compiler for kernel but I haven’t tested it myself. In case it didn’t there is very powerful tool for making your own toolchain for any platform supported by GCC so in practice any you can imagine called crosstool-ng (on Arch available on AUR).
  • uucp source code

Kernel module

We will start with a kernel module. First of all we will unpack our kernel. In case of SE kernel need to be uncompressed and then unpacked so I did:

bzcat 4_1_B_0_431_tar.bz2 | tar -xv
cd kernel

And changed dir to kernel. Next thing I had to do was patching Makefile, because it complained about unused variables. If you are using Sony or SE kernel you probably need to do it too. If so create file named Makefile.patch with your favorite editor, i.e.

vim Makefile.patch

and paste following content:

--- Makefile  2012-05-25 12:07:05.000000000 +0200
+++ Makefile.new  2014-08-20 21:16:50.642703198 +0200
@@ -342,7 +342,6 @@
 KBUILD_CPPFLAGS := -D__KERNEL__

 KBUILD_CFLAGS   := -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs \
-		   -Werror \
 		   -fno-strict-aliasing -fno-common \
 		    -Werror-implicit-function-declaration \
 		    -Wno-format-security \

Now you can patch it with:

patch Makefile < Makefile.patch

Then we can start the compilation process which should end in a second. I assume you have working toolchain in your $PATH and it is prefixed with arm-unknown-eabi- (arm-unknown-eabi-gcc, etc.). If its name is different change all occurrences of arm-unknown-eabi- below.

make mrproper
ARCH=arm CROSS_COMPILE=arm-unknown-eabi- make semc_iyokan_defconfig
ARCH=arm CROSS_COMPILE=arm-unknown-eabi- make prepare
ARCH=arm CROSS_COMPILE=arm-unknown-eabi- make modules_prepare
ARCH=arm CROSS_COMPILE=arm-unknown-eabi- make modules SUBDIRS=drivers/usb/serial CONFIG_USB_SERIAL=m CONFIG_USB_SERIAL_CP210X=m

If everything worked you should have your module compiled in drivers/usb/serial directory. It is worth to note you could compile any other module the same way. It is only important to change SUBDIRS to your module location and ensure it is configured to be built by issuing make menuconfig or setting appropriate CONFIG-* variable to ‘m’.

Now you can copy modules to your devices. With adb it will be:

adb push drivers/usb/serial/usbserial.ko /mnt/sdcard/others
adb push drivers/usb/serial/cp210x.ko /mnt/sdcard/others

We have two modules to copy here since cp210x (and any other serial device driver) depends on usbserial and without it being already in kernel it would be impossible to insert cp210x and furthermore it will give you strange error message (tested :). So now, again with help of ADB, we can insert our modules.

adb shell
su
cd /mnt/sdcard/others
insmod ./usbserial.ko
insmod ./cp210x.ko

After that you can connect your device and device you compiled module for and test if it works. Serial adapters should create new device file in /dev and, at least with CP210x, it uses ttyUSB* name so you can list it with:

ls -l /dev/ttyUSB*

You can also read from your port as simply as by typing

cat /dev/ttyUSB0

Now after you connect some device talking UART you should see its output.

Program to easily control our port – cu and the rest of uucp package

Warning: in the following steps I assume that you have directories like /data/local/bin and /data/local/etc already on your device. As far as I remember there is only /data/local by default.

At first we need to unpack our source and create some useful directories. Then we will configure our script in build-uucp so we won’t get garbage in source directory and could easily clean things when something goes wrong.

tar -zxvf uucp-1.07.tar.gz
mkdir build-uucp
mkdir install-uucp
cd build-uucp/

Then we will patch our program so it will work on Android out of the box. If we omitted following patch it would be necessary to create configuration file, configure program to read it before work and whole bunch of similar fun. So it is easier to let him know where he could store his files now. You can obviously use the one you want to. If you stay with my config you won’t be able to start cu as normal user which is no problem because by default you won’t have permission to use serial terminal.

Now the procedure is the same as with previous patch. I assume you used policy.h.patch name and the content is:

--- policy.h  2003-05-29 08:08:45.000000000 +0200
+++ policy.h.new  2014-08-20 12:03:45.595405893 +0200
@@ -297,7 +297,7 @@
    systems the lock files are placed in /etc/locks.  On some they are
    placed in /usr/spool/locks.  On the NeXT they are placed in
    /usr/spool/uucp/LCK.  */
-/* #define LOCKDIR "/usr/spool/uucp" */
+#define LOCKDIR "/data/local/etc/spool/uucp"
 /* #define LOCKDIR "/etc/locks" */
 /* #define LOCKDIR "/usr/spool/locks" */
 /* #define LOCKDIR "/usr/spool/uucp/LCK" */
@@ -572,7 +572,7 @@
 /* The name of the default spool directory.  If HAVE_TAYLOR_CONFIG is
    set to 1, this may be overridden by the ``spool'' command in the
    configuration file.  */
-#define SPOOLDIR "/usr/spool/uucp"
+#define SPOOLDIR "/data/local/etc/spool/uucp"
 /* #define SPOOLDIR "/var/spool/uucp" */

 /* The name of the default public directory.  If HAVE_TAYLOR_CONFIG is
@@ -580,7 +580,7 @@
    configuration file.  Also, a particular system may be given a
    specific public directory by using the ``pubdir'' command in the
    system file.  */
-#define PUBDIR "/usr/spool/uucppublic"
+#define PUBDIR "/data/local/etc/spool/uucppublic"
 /* #define PUBDIR "/var/spool/uucppublic" */

 /* The default command path.  This is a space separated list of
@@ -644,21 +644,21 @@
 /* The default log file when using HAVE_TAYLOR_LOGGING.  When using
    HAVE_TAYLOR_CONFIG, this may be overridden by the ``logfile''
    command in the configuration file.  */
-#define LOGFILE "/usr/spool/uucp/Log"
+#define LOGFILE "/data/local/etc/spool/uucp/Log"
 /* #define LOGFILE "/var/spool/uucp/Log" */
 /* #define LOGFILE "/var/log/uucp/Log" */

 /* The default statistics file when using HAVE_TAYLOR_LOGGING.  When
    using HAVE_TAYLOR_CONFIG, this may be overridden by the
    ``statfile'' command in the configuration file.  */
-#define STATFILE "/usr/spool/uucp/Stats"
+#define STATFILE "/data/local/etc/spool/uucp/Stats"
 /* #define STATFILE "/var/spool/uucp/Stats" */
 /* #define STATFILE "/var/log/uucp/Stats" */

 /* The default debugging file when using HAVE_TAYLOR_LOGGING.  When
    using HAVE_TAYLOR_CONFIG, this may be overridden by the
    ``debugfile'' command in the configuration file.  */
-#define DEBUGFILE "/usr/spool/uucp/Debug"
+#define DEBUGFILE "/data/local/etc/spool/uucp/Debug"
 /* #define DEBUGFILE "/var/spool/uucp/Debug" */
 /* #define DEBUGFILE "/var/log/uucp/Debug" */

@@ -669,17 +669,17 @@
 /* The default log file when using HAVE_V2_LOGGING.  When using
    HAVE_TAYLOR_CONFIG, this may be overridden by the ``logfile''
    command in the configuration file.  */
-#define LOGFILE "/usr/spool/uucp/LOGFILE"
+#define LOGFILE "/data/local/etc/spool/uucp/LOGFILE"

 /* The default statistics file when using HAVE_V2_LOGGING.  When using
    HAVE_TAYLOR_CONFIG, this may be overridden by the ``statfile''
    command in the configuration file.  */
-#define STATFILE "/usr/spool/uucp/SYSLOG"
+#define STATFILE "/data/local/etc/spool/uucp/SYSLOG"

 /* The default debugging file when using HAVE_V2_LOGGING.  When using
    HAVE_TAYLOR_CONFIG, this may be overridden by the ``debugfile''
    command in the configuration file.  */
-#define DEBUGFILE "/usr/spool/uucp/DEBUG"
+#define DEBUGFILE "/data/local/etc/spool/uucp/DEBUG"

 #endif /* HAVE_V2_LOGGING */

@@ -692,16 +692,16 @@
    be replaced by the system name (if there is no appropriate system,
    "ANY" will be used).  No other '%' character may appear in the
    string.  */
-#define LOGFILE "/usr/spool/uucp/.Log/%s/%s"
+#define LOGFILE "/data/local/etc/spool/uucp/.Log/%s/%s"

 /* The default statistics file when using HAVE_HDB_LOGGING.  When using
    HAVE_TAYLOR_CONFIG, this may be overridden by the ``statfile''
    command in the configuration file.  */
-#define STATFILE "/usr/spool/uucp/.Admin/xferstats"
+#define STATFILE "/data/local/etc/spool/uucp/.Admin/xferstats"

 /* The default debugging file when using HAVE_HDB_LOGGING.  When using
    HAVE_TAYLOR_CONFIG, this may be overridden by the ``debugfile''
    command in the configuration file.  */
-#define DEBUGFILE "/usr/spool/uucp/.Admin/audit.local"
+#define DEBUGFILE "/data/local/etc/spool/uucp/.Admin/audit.local"

 #endif /* HAVE_HDB_LOGGING */

If you prefer I have it on my gist so you can just issue one command and get it. Then we are patching as usually:

patch ../uucp-1.07/policy.h < policy.h.patch

As mentioned above you can change path of uucp’s files by issuing the following (remember to escape every occurrence of slash with backslash, otherwise it will fail):

sed -i "s/\/data\/local\/etc/[your-path]/" ../uucp-1.07/policy.h

Now you are ready to compile. It can be done with following commands. Your compiler should have same name (at least if you use NDK’s compiler). It is important to note that I had to switch off HAVE_SYSCONF flag since it was causing ugly errors. In my case makescript couldn’t also find a rule to make ftw.o so I had to make it myself. If you have no trouble here, just omit the line after make.

CC=arm-linux-androideabi-gcc AR=arm-linux-androideabi-ar RANLIB=arm-linux-androideabi-ranlib \
../uucp-1.07/configure --prefix=`pwd`/../install-uucp/ --host=arm-linux-androideabi
sed -i "s/#define HAVE_SYSCONF 1/#define HAVE_SYSCONF 0/" config.h
make
cd unix; make ftw.o; cd ..; make
make install
adb push ../install-uucp/bin/cu /mnt/sdcard/others
adb shell
su
cp /mnt/sdcard/others/cu /data/local/bin/

Finally you can test the program with the following and you should be able to talk RS232 with just a phone/tablet!

cu -lttyUSB0 -s115200

where 115200 is the speed the device you connect to transmits.

BTW: uucp have few other tools and by following this tutorial you compiled them all so you can explore them on your own.

[Import]How to bypass Secure Download Manager while downloading from Dreamspark

NOTE: This post was imported from my previous blog – v3l0c1r4pt0r.tk. It was originally published on 1st June 2014.

NOTE2: As people are reporting, THIS METHOD DOES NOT WORK anymore. Also I don’t have access to Imagine, so I would not be able to provide any help. Therefore, this article is left only for historic purposes, or for those that have some SDC files downloaded, when it was still valid and have valid decryption keys.

About a month or so ago I had an urgent need to download a copy of Microsoft Windows from Dreamspark. Unfortunately I haven’t Windows installed then so had to do this using Linux. After successful transaction I was given a link to SDX file and program called SDM. It looked that it would be easy. But it wasn’t. Program that I was encouraged to download was archive with .pkg extension. As I discovered few minutes later it was OS X application package. So the next step was to try to download Windows version and try to execute it with help of Wine. It failed. Then I tried to find some tips on the Net. I found a few other people having the same problem. Some of them could download using Wine and some not. For me there was only one solution: do it myself. As you probably guessed that way was a (almost) full success.

Solution

If you already have SDC file please do not skip since you probably still don’t have a key needed to unpack the file. The first step is to open SDX file in your favorite text editor. You will see a link. Open it in a web browser. Now you need to get to page source. The way it can be done depends on your web browser. Now we need to find few strange values in the code. The easiest way to achieve this is to search for keyword ‘edv*’ where * is the number of file you want to download counting from 1 (they are on ‘Items’ list on page you opened). Now you need to copy somewhere values of the following variables: ‘oiopu*’, ‘oiop*’, ‘fileID*’ (*-see above). The last one we need is ‘dlSelect*’ but for that one you need to search cause it is in a different place. Now you can build URL that will let you to file containing two interesting values: file URL and decryption string. This URL’s format is:

http://[SDXdomain]/WebStore/Account/SDMAuthorize.ashx?oiopu=[oiopu]&f=[fileID]&oiop=[oiop]&dl=[dlSelect]

Now you should see XML file that looks similar to this:

<information>
 <oiopua>01234abcd-0123-4567-890a-0123456789ab</oiopua>
 <edv>0123456789^^0123456789QwErTyUiOpAsDfGhJkLzXc0123456789QwErTyUiOpAsDfGhJkLzXc12345678</edv>
 <linkAvailable>1</linkAvailable>
 <errorTextKey/>
 <invokeExternalDownload>0</invokeExternalDownload>
 <fileUrl><![CDATA[http://software.dreamspark.com/dreamspark/ENGLISH/SDCfileName.sdc]]></fileUrl>
</information>

The last step here will be downloading file from fileUrl and saving edv value in file. The important thing is that the file with a key should be named exactly as SDC file with addition of ‘.key’ suffix.

Update:

I’ve just discovered that things are getting a bit different when the file size exceeds 2.0 GB. In that case Dreamspark is splitting file in two or more files. That situation could be easily recognized, because sdc file name’s suffix is: ‘.01.sdc’. In that case you need to try to download file which URL differs by only that one digit, ie. ‘.01.sdc’, ‘.02.sdc’, ‘.03.sdc’. When you encounter last file it should have smaller size than the rest and incrementing that number by one should give you BlobNotFound error.

After downloading all the files they just need to be joined into one. It can be easily achieved with dd, ie.

dd if=pl_windows_7_professional_with_sp1_x64_dvd_u_676944.02.sdc >> pl_windows_7_professional_with_sp1_x64_dvd_u_676944.01.sdc

and then optionally

dd if=pl_windows_7_professional_with_sp1_x64_dvd_u_676944.03.sdc >> pl_windows_7_professional_with_sp1_x64_dvd_u_676944.01.sdc

After that you will get sdc file prepared to unpack.

Unpacking SDC

Now since you have SDC file you can start unpacking it. The previous part was, at least for me, very easy. The problem started when I tried to discover how the file is stored in that container. But don’t worry, I’ve written simple program to do it for you. As of now (1st June) it is still in really early alpha stage and have lot of constraints. It is able to unpack containers that contains only one file packed, doesn’t create any directories, cannot verify file’s checksum and probably few other problems I don’t remember or don’t know about.

If you were searching a bit in the Internet, you probably found out that someone cracked that container in the past. Unfortunately Microsoft changed format since then. It is also possible that in response to this article it will be changed again. To make it a bit harder for them to block my software I’m publishing source code on github and after the process of reverse engineering is finished will write second article describing how things works under the hood and describe sdc file format.

But let’s get back to unpacking. Now you need to download xSDM from github. The newest version can be downloaded by typing

git clone https://github.com/v3l0c1r4pt0r/xSDM.git

in your terminal (of course you need to have git installed). Nevertheless I advice you to download newest tagged release. You can do this by clicking on releases on project page and then choosing the one on the top (or first beta/stable if any) and clicking on “tar.gz”. tar.gz can be unpacked by typing

tar -zxvf xSDM-[tag-name].tar.gz

into console. Then get into xSDM directory by typing

cd xSDM

(or your release directory) and compile the program by standard

./configure
make
make install

where installation is optional. Now to unpack your file you just need to type

src/xsdm [path-to-your-sdc-file]

And that’s it, you should now be able to open file you downloaded. As mentioned above the program is in very early alpha so I cannot guarantee that it will work in any case. If you will encounter any problems feel free to open issue on project page at github.

[Import]Graphic LCD based on ST7565 chip under Raspberry Pi

NOTE: This post was imported from my previous blog – v3l0c1r4pt0r.tk. It was originally published on 25th December 2013.

Hi, today I’d like to show you how to connect and use gLCD module with Raspberry Pi as host. The display I have is only bare display without any board not like the one in Adafruit’s offer. It can be powered by 3V3 Raspberry but the display itself needs a bit more power so we will need a few capacitors to build a circuit for that purpose. It will also make the connection a bit complicated (can be seen in the photo on the right). Nevertheless I think that the process is still rather easy.

Overview

Connected display

As far as I know ST7565 based displays can be connected on two ways: parallel and serial. In case of serial connection, which I used to save few GPIO’s, it is possible to program it using SPI or just only GPIO’s. The display that I have is a bit different than most of the others because it has external backlight, which is additionally single LED so it is very power-saving (15mA). The only problem with that backlight was that the vendor didn’t told anything about parameters of that diode so I needed to figure it out myself.. The second problem while connecting the display itself might be amount of cables that need to be connected when using breadboard. Despite these two facts the whole process should be easy.

Physical connections

Connection scheme

As said before the only step that may be a bit complicated is connecting so called step-up voltage circuit, made of 4 capacitors. The capacitors that we will use are 1uF electrolytic caps. Beside that we need to use another 5 caps (same as before) to connect parallel inputs to ground. So in sum we need 9 of them. Now we only need to connect VDD to 3V3 pin on Raspberry, ground from the schematic on the right with GND pin, SDATA to SMOSI on Pi, SCLK to SCLK and A0, RST and CS to any free GPIO. It is good to remember their numbers cause we will need it in a moment 🙂 It is important to use numbers used by Linux kernel, not wiringPi which has its own names. At last we need to connect the backlight. As said I have ECO backlight so I had to connect mine using 10 Ohm resistor. You can connect it to 3V3 or if you like to have control during runtime use GPIO, just like any other LED.

Configuring the program

Now I have to mention something about a program itself, because depending on how your vendor implemented the things your display will almost surely need a bit different settings. General procedure will look the same on every ST7565-based display. Main differences will be on particular commands during setup procedure.

 
uint8_t init() 
{ 
  if (!bcm2835_init()) { 
    return 0; 
  } 
  bcm2835_gpio_fsel(LCD_BACK,BCM2835_GPIO_FSEL_OUTP);   //backlight 
  bcm2835_gpio_fsel(LCD_A0,BCM2835_GPIO_FSEL_OUTP);   //A0 
  bcm2835_gpio_fsel(LCD_RST,BCM2835_GPIO_FSEL_OUTP);    //RST 
  bcm2835_gpio_fsel(LCD_CS,BCM2835_GPIO_FSEL_OUTP);   //CS 
  bcm2835_gpio_write(LCD_CS,HIGH);        //set CS to high to indicate the bus as free 
  bcm2835_gpio_write(LCD_RST,LOW); 
  bcm2835_delayMicroseconds(1); 
  bcm2835_gpio_write(LCD_RST,HIGH);       //hardware reset 
  //setup SPI 
  bcm2835_spi_begin(); 
  bcm2835_spi_chipSelect(BCM2835_SPI_CS_NONE);      //manual CS control 
  bcm2835_spi_setClockDivider(BCM2835_SPI_CLOCK_DIVIDER_4); //set speed to 62.5MHz (fastest supported) 
 
  int i; 
  bcm2835_gpio_write(LCD_CS,LOW); 
  for(i = 0; i < sizeof(initcmd)/sizeof(uint8_t); i++) 
    transfer(initcmd[i],0); 
  bcm2835_gpio_write(LCD_CS,HIGH); 
  bcm2835_gpio_write(LCD_BACK,HIGH);        //turn backlight on 
 
  return 1; 
} 

I think that the code above should be generally clear. The most important for us is for loop that is executing every byte from initcmd array. Its content will look like that:

const uint8_t initcmd[] =
{
  0xa1,             //screen orientation
  0x41,             //set starting line
  0xc0,             //page count direction
  0xa3,             //1/7 bias
  0x2c,             //vc
  0x2e,             //vc+vr
  0x2f,             //vc+vr+vf
  0x24,             //voltage regulator (0x20-0x27)
  0xa6,             //do not reverse the display
  0xaf,             //display on
  0xa4,             //display from ram
  0x81,             //turn on brightness regulation
  0x18              //set brightness (0x0-0x40)
};

The most important values here are:

  • voltage regulator – 0x20 means the darkest, as seen above in my case 0x24 worked
  • bias – I saw displays that had 1/9 so you need to make sure how is in yours and set it according to chips documentation linked at the end

You may also want to play with commands like screen orientation, page direction, display reverse or brightness to fit them to your needs. Now you have tell the program which GPIO you used as backlight (if you weren’t using GPIO you will now need to comment out few lines that sets backlight up), CS, RST and A0.

The program itself

To compile the program you will need to use external library named libbcm2835. It can be installed on ArchLinux ARM by issuing pacman -S libbcm2835 as root. If you are ready you can compile the program by typing: gcc -o lcd lcd.c -lbcm2835 assuming you didn’t change the filename. The simple program I’ve written, basing on the one posted on Gist by tmeissner here and ST7565’s documentation supports transferring single byte (commands too), whole framebuffer, or writing 5×8 single ASCII character or character string. Basing on both codes: mine and Meissner’s I think it is possible to do anything you could think about with that display.

Font creation

Standard ASCII table and traditional ‘Hello World!’:)

Ending slowly it’s time to tell something about fonts. As I said it is possible to simply write characters on the screen. To understand how all that thing works you need to know how the pixels are transfered to the display. The best explanation of the ST7565 display’s work can be in my opinion found here. TL;DR: the whole pixel space is divided into eight, 8-pixel high, horizontal pages divided into 128 columns that are 8 pixels high. If you didn’t understand, try link above. Nevertheless single letter is 8-pixel high and 5-pixel long so we need 5 bytes to store one letter. Its pixel map starts at left, top corner so it’s our (0,0) point and setting LSBof the first byte lights highest pixel. The font that is available in the code is Fixed 5×8 if someone is curious, it’s one of the default fonts in Xorg. To speed up conversion of the font to the display’s format I made simple OpenGL program to do the job for me. The code is of course available to download (check out my github).

DLs

Traditionally, at last some downloads:

[Import]Using GNU Radio Companion – simple FM radio tutorial

NOTE: This post was imported from my previous blog – v3l0c1r4pt0r.tk. It was originally published on 1st November 2013.

This post is the second part of the Instructable mentioned on the previous post. That’s tutorial on how to make simple FM radio using GRC. That task is in my opinion the simplest project that can be made using GRC so it’s in fact beginners guide to GNU Radio which is really capable software. I hope that it is only the first my project using that software.

It isn’t hard to find GRC projects that implements FM radio functionality, that’s a fact. But when it comes to reproduce them so they are working just like SDR# you’d realize it’s a bit harder task. At least I did. I found about a three such projects but there was never any included project file and the only materials was video or pictures. I admit the video has pretty good explanation but it still wasn’t enough. The rest was even worse. They were only blog posts with few screen shots and short description. So finally to reproduce functionality of SDR#’s FM radio I had to think on my own. And after few hours I did what I tried to do.

A bit of theory

RTL-SDR Source Properties

But let’s start at the beginning. The first thing that you need to do to start creating your own FM receiver using GRC is to find FM station that can be received without disruptions so you can check if your program works as well as i.e. SDR#. It can be done with help of SDR# and when you found one you are ready to learn some theory about FM signal processing. In fact it could be omitted, but in my opinion it is better to know a bit. I personally am not any specialist in radio theory so it will be really simple and I might be wrong somewhere so if you find any mistake in that what I will write here just let me know in comments.

The simplest FM radio consists of few elements:

  • signal source – in our case it would be RTL-SDR dongle
  • low pass filter
  • WBFM demodulator
  • audio output – your PC’s sound card

There can be few other elements depending on input and output sample rate, if it will be possible to match them using only above elements there won’t be any other. So our task will be getting signal using RTL-SDR, passing it through low pass filter and FM demodulator and outputting on PC’s sound card. Meanwhile we will also need to match the sample rate of the input to the one of the output (2 MHz in to 48 KHz out). All elements of this circuit can be found by writing part of its name while list on the right is selected. Element’s names are the ones used as section header below.

RTL-SDR source

Low Pass Filter Properties

Our signal input. We need to set its sample rate to 2M. It can be done by editing samp_rate variable and setting its value to 2e6. We need also to set the frequency of the station we want to receive. It is good practice to add every value that might be changed in the future as standalone variable. This can be done by adding Variable block from Variables category or, if you want to have possibility to edit it during the runtime block named WX GUI Slider or WX GUI Text Box and then just writing variable name as value in block’s properties.

Rational Resampler

Now we need to convert sample rate from 2M (samp_rate) to 500K which I realized is the best value for low pass filter’s input rate. To do that we need to add another variable, named i.e. quadrature, set its value to 500e3 (means: 500K). With that we need to add rational resampler’s block and set its decimation value to int(samp_rate/quadrature). Of course its input on the schematic should be connected to output of RTL-SDR Source (can be achieved by clicking on blue out on one block and then blue in on another).

Low Pass Filter

The next step is to filter out frequencies other than the one we centered in th previous step. In that task we will use a low pass filter block. We here set cutoff frequency to 100e3. This is because that’s standard band’s width. I don’t know what correct value for transition width should be, but trying to change that I found that the higher value the better so it’s finally set to 1M. Obviously it should be connected to resampler’s output.

WBFM Receive

Volume slider Properties

Now we could do the nearly final and the most important step: placing the FM demodulator. In my case its quadrature rate equals 500k (that’s the same value as before so I set this to the value of quadrature variable). I don’t know if it could be changed to something else so if you are not sure just leave it as is. In that setting it should work.

Next Rational Resampler and Multiply Const

These elements’ job is to match the signal’s rate to the one supported by sound card and provide volume regulation. At first we need to convert rate from 500k to 48k so we need to decimate by 500 and then multiply by 48. In the second block we set const to volume. Now we can create variable or place WX GUI Slider with variable name set to volume. As you can see here color of WBFM demodulator is different than rational resampler’s and multiplier’s in and out. To change that you need to select them and use up and down arrows on your keyboard.

Audio Sink

That one’s function is to output signal on our sound card. Now its only required to edit its rate and choose 48k from a drop down list and that’s it! If you don’t have that option just choose the biggest value and edit value in rational resampler and it should work. Now you should be able to execute your program and test if it works. To hear anything it will probably be required to increase volume to about 50.

Finally: some links

If you don’t know how some part has been done or what value should have a particular variable below you can download projects that I’ve made first learning to use GRC myself and then preparing this tutorial. There is one project presented here and one made earlier.

Finished FM radio schematic