ANNOUNCE: codespell 1.2

Since I created a mailing list for codespell, the announcements here will not have as many details as before. Checkout the new version of codespell: http://groups.google.com/group/codespell/browse_thread/thread/bce1a6f83d4bbd85

One of the issues I with codespell was that it was trying to parse cscope.out, since it’s a text file. On Linux Kernel this file can get very big and besides taking much longer, sometimes it was running out of memory :-). Now codespell has an option to ignore files, even text ones. It’s as easy as passing –skip=”*.eps,cscope.out” (notice this is useful for ignoring eps images too). Another useful thing (not much for Linux Kernel though) is proper detection of encoding by using chardet.

Attending AI classes

For those who didn’t know, Stanford is offering some online courses, starting from next month. During my graduation I already have some AI classes but I thought it would be interesting to participate in this class with thousand of students.

One area that I like on AI is the on to teach computers how to play games. I like that so much that during my AI classes in Politecnico di Milano I developed a computer game to play tic-tac-toe in NxNxN dimensions. It’s actually fun to play, and you can find some screenshots (and the code) on a past blog post. I didn’t have time to continue maintaining it, but the part that I liked more was actually designing the algorithm, using MinMax, and eventually finding out that I couldn’t win the game anymore when playing against the computer. This was both motivating and demotivating because I knew that if I did developed it right, I couldn’t possibly win anymore. Maybe I’ll update this project when I have some time. If you would like to try it and find any problem, send me a patch and I will happily apply. If you think you have a better algorithm, it’s very easy to develop your own Player and plug it in the game. I challenge you to win my MinMax player ;-).

During the time I was developing this game, a friend of mine that lived with me in Italy had a dream of making money playing poker. Because of this he was attending the AI classes too and eventually playing online to develop his strategies (in sites such as partypoker). It was very fun talking to him and his plans and strategies, though I don’t know if he succeeded.

Even if you frequented some AI class before like me and my friend, I think it will be very interesting to participate on this one. If you didn’t register yet, you still have some time: go to http://www.ai-class.com and subscribe.

Seminário de Linux Embarcado 2011

No último final de semana participei do Seminário de Linux Embarcado, em que palestrei sobre “systemd: repensando a inicialização”. O feedback que tive do pessoal foi positivo, mostrando bastante interesse nos diversos tipos de inicialização sob demanda, exclusivos desse sistema de init. Os slides estão disponíveis aqui no blog.

Gostei bastante do evento, que teve também participação do Antognolli, que trabalha comigo, sobre interfaces gráficas em sistemas embarcados usando o conjunto de bibliotecas EFL. Outro conhecido meu de outras conferências, Glauber Costa, falou sobre QEMU.

Benchmarking Javascript engines for EFL

The Enlightenment Foundation Libraries has several bindings for other languages in order to ease the creation of end-user applications, speeding up its development. Among them, there’s a binding for Javascript using the Spidermonkey engine. The questions are: is it fast enough? Does it slowdown your application? Is Spidermonkey the best JS engine to be used?

To answer these questions Gustavo Barbieri created some C, JS and Python benchmarks to compare the performance of EFL using each of these languages. The JS benchmarks were using Spidermonkey as the engine since elixir was already done for EFL. I then created new engines (with only the necessary functions) to also compare to other well-known JS engines: V8 from Google and JSC (or nitro) from WebKit.

Libraries setup

For all benchmarks EFL revision 58186 was used. Following the setup of each engine:

  • Spidermonkey: I’ve used version 1.8.1-rc1 with the already available bindings on EFL repository, elixir;
  • V8: version 3.2.5.1, using a simple binding I created for EFL. I named this binding ev8;
  • JSC: WebKit’s sources are needed to compile JSC. I’ve used revision 83063. Compiling with CMake, I chose the EFL port and enabled the option SHARED_CORE in order to have a separated library for Javascript;

Benchmarks

Startup time: This benchmark measures the startup time by executing a simple application that imports evas, ecore, ecore-evas and edje, bring in some symbols and then iterates the main loop once before exiting. I measured the startup time for both hot and cold cache cases. In the former the application is executed several times in sequence and the latter includes a call to drop all caches so we have to load the library again from disk

Runtime – Stress: This benchmark executes as many frames per second as possible of a render-intensive operation. The application is not so heavy, but it does some loops, math and interacts with EFL. Usually a common application would do far less operations every frame because many operations are done in EFL itself, in C, such as list scrolling that is done entirely in elm_genlist. This benchmark is made of 4 phases:

  • Phase 0 (P0): Un-scaled blend of the same image 16 times;
  • Phase 1 (P1): Same as P0, with additional 50% alpha;
  • Phase 2 (P2): Same as P0, with additional red coloring;
  • Phase 3 (P3): Same as P0, with additional 50% alpha and red coloring;

The C and Elixir’s versions are available at EFL repository.

Runtime – animation: usually an application doesn’t need “as many FPS as possible”, but instead it would like to limit to a certain amount of frames per second. E.g.: iphone’s browser tries to keep a constant of 60 FPS. This is the value I used on this benchmark. The same application as the previous benchmark is executed, but it tries to keep always the same frame-rate.

Results

The first computer I used to test these benchmarks on was my laptop. It’s a Dell Vostro 1320, Intel Core 2 Duo with 4 GB of RAM and a standard 5400 RPM disk. The results are below.

Benchmarks on Dell 1320 laptop

First thing to notice is there are no results for “Runtime – animation” benchmark. This is because all the engines kept a constant of 60fps and hence there were no interesting results to show. The first benchmark shows that V8′s startup time is the shortest one when considering we have to load the application and libraries from disk. JSC was the slowest and  Spidermonkey was in between.

With hot caches, however, we have another complete different scenario, with JSC being almost as fast as the native C application. Following, V8 with a delay a bit larger and Spidermonkey as the slowest one.

The runtime-stress benchmark shows that all the engines are performing well when there’s some considerable load in the application, i.e. removing P0 from from this scenario. JSC was always at the same speed of native code; Spidermonkey and V8 had an impact only when considering P0 alone.

 

Next computer to consider in order to execute these benchmarks was  a Pandaboard, so we can see how well the engines are performing in an embedded platform. Pandaboard has an ARM Cortex-A9 processor with 1GB of RAM and the partition containing the benchmarks is in an external flash storage drive. Following the results for each benchmark:

 

Benchmarks on Pandaboard

Once again, runtime-animation is not shown since it had the same results for all engines. For the startup tests, now Spidermonkey was much faster than the others, followed by V8 and JSC in both hot and cold caches. In runtime-stress benchmark, all the engines performed well, as in the first computer, but now JSC was the clear winner.

 

There are several points to be considered when choosing an engine to be use as a binding for a library such as EFL. The raw performance and startup time seems to be very near to the ones achieved with native code. Recently there were some discussions in EFL mailing list regarding which engine to choose, so I think it would be good to share these numbers above. It’s also important to notice that these bindings have a similar approach of elixir, mapping each function call in Javascript to the correspondent native function. I made this to be fair in the comparison among them, but depending on the use-case it’d  be good to have a JS binding similar to what python’s did, embedding the function call in real python objects.

ESC Brazil – Realtime Linux with RT_PREEMPT

Two weeks ago I’d give a talk about realtime Linux at ESC Brazil: “Usando Linux como Sistema de Tempo Real” (Using Linux as a realtime OS). Sadly some days before while playing soccer  I broke my fibula and I had to have a surgery. I regret I couldn’t attend this conference.

At least in the company I work for there are more people with knowledge in this area. Gustavo Barbieri went there in my place and had a good feedback from the attendees.

Now I have stay home. At least for 1 or 2 months :-(.

 

codespell 1.1-rc1

I’m glad to announce the first RC of codespell 1.1. I decided to let the biggest feature for the next version and release 1.1 with the small but important features that were already implemented. This new version comes with the following features:

  • Verbosity level: tired of seeing so many things printed while the fixes are taking place? Now you can filter what you see
  • Exclusion list: there are cases in which the codespell spots a false positive, but disabling the entry in the dictionary will prevent it from fixing many other places. This is particularly true when there are names in source code. In Linux kernel I’ve seen some names with “Taht” and “Teh” that were incorrectly fixed to “That” and “The”. Now we have a file with lines that are exclude from the ones codespell will fix. Hopefully such lines will not change very often and we can maintain a file per project for future executions of codespell in each project. I’m maintaining one for the Linux kernel. It’s in data/linux-kernel.exclude.
  • Interactive mode: for those fixes that are not done automatically (because they have more than one possible fix), now we can interactively decide on each one. I recommend for everyone interested in this feature to run codespell once without this option to fix the automatic ones and another time to go through the other fixes.
  • Stats (summary) of the changes: are you interested how many times a word was misspelled? Now codespell can display a summary of all the fixes it has done.

I was particularly worried about the increase in runtime when using the exclusion list. However it proved to be very fast when excluding the lines by using their hashes. I can successfully parse the entire Linux kernel tree within 1min30s in my laptop with slow HD. The biggest feature that I’ve left for the next version is to allow changes to be applied only to parts of the source code like comments and strings. I expect to implement this for a future 1.2 version.

Besides these new features, there are some fixes to the dictionary. Thanks to all of you who have sent me fixes and suggestions. I’m glad to see patches generated by codespell been applied to other opensource projects that I’m not the one to send. As of now, I’ve seen patches been applied to: Linux Kernel, oFono, ConnMan, FreeBSD, LLVM, clang, EFL and others that I don’t remember right now.

For those who prefer to wait for a stable release, I’m also releasing codespell 1.0.2 with fixes only to the dictionary.

As always, you can download codespell packages from:

http://packages.profusion.mobi/codespell/

Repositories are available at:

http://git.profusion.mobi/cgit.cgi/lucas/codespell/

https://github.com/lucasdemarchi/codespell

Por uma web melhor e mais segura

Hoje adicionei no site um script para alertar usuários que usam versões antigas (e inseguras) de browsers. O projeto é o “Salve a Web, por favor”. Ele é opensource e pode ser visto no repositório do github. Basta carregar o script com a seguinte entrada na sua página:

1
<script type="text/javascript" src="http://sawpf.com/1.0.js"></script>

Acho que pela própria natureza dos posts no meu blog, a divisão dos browsers não é parecida com aquela global, em que o Internet Explorer ainda é o browser mais usado. Porém acho importante alerr os usuários. Quem sabe poderemos ter uma web melhor e mais segura futuramente? Veja abaixo as estatísticas de acesso a esse blog no últimoo ano.

 

Browsers usados no acesso ao blog

Sistemas Operacionais usados no acesso ao blog

 

Os poucos visitantes com browsers antigos que leem esse blog vão ver o seguinte banner acima (obtido mudando o user agent do Chromium para um IE 6):

Internet Explorer desatualizado

Cross-compiling with icecc/icecream

Very common questions I hear when dealing with  compiling open source projects are:

  1. How do I cross-compile a project using icecc/icecream?
  2. How to use a different compiler version for compiling my project and still benefiting from icecc/icecream?

Note: from now on I’ll always refer to icecream instead icecc/iceccd/distcc for the name of the project.

Given you already created your cross toolchain  (or downloaded from somewhere else, e.g.  CodeSourcery/Linaro) these two questions are essentially the same. All you have to do is to follow the two steps below:

1. Create the “compiler environment”

Understanding this part is really understanding how this magic remote-compiling works. When you want to compile a source remotely, what icecream does is sending a copy of your compiler and the things it needs to the remote machine, executing the process and getting back the result. By “things it needs” I mean: assembler, linker, libc, libgcc and some other libraries like libm, libgmp, libstdc++, libz, etc. Creating this environment with icecream is dead easy: call “icecc –build-native”. Following is the output I get on my Archlinux box with gcc 4.6.0 as default compiler:

$ icecc --build-native
adding file /usr/bin/gcc
adding file /lib/libc.so.6
adding file /lib/ld-linux-x86-64.so.2
adding file /usr/bin/g++
adding file /usr/bin/as
adding file /usr/lib/libopcodes-2.21.0.20110430.so
adding file /usr/lib/libbfd-2.21.0.20110430.so
adding file /lib/libdl.so.2
adding file /usr/lib/libz.so.1
adding file /usr/bin/cc1=/usr/lib/gcc/x86_64-unknown-linux-gnu/4.6.0/cc1
adding file /usr/lib/libcloog-isl.so.1
adding file /usr/lib/libisl.so.6
adding file /usr/lib/libgmp.so.10
adding file /usr/lib/libppl_c.so.4
adding file /usr/lib/libppl.so.9
adding file /usr/lib/libgmpxx.so.4
adding file /usr/lib/libstdc++.so.6
adding file /lib/libm.so.6
adding file /usr/lib/libgcc_s.so.1
adding file /usr/lib/libpwl.so.5
adding file /usr/lib/libmpc.so.2
adding file /usr/lib/libmpfr.so.4
adding file /usr/bin/cc1plus=/usr/lib/gcc/x86_64-unknown-linux-gnu/4.6.0/cc1plus
adding file /usr/lib/gcc/x86_64-unknown-linux-gnu/4.6.0/liblto_plugin.so
adding file /etc/ld.so.conf=/tmp/icecc_ld_so_confPaVA2Q
creating 6b1d2b44080a004a88c0746fe128172e.tar.gz

Note that in the last line it created a .tar.gz file. This is the environment that will be sent to other machines. If you want to use another compiler, you need to create another environment that will be later passed to icecream in the second step.

To create an environment for a compiler that is not the default in your machine, first thing you need is to have it in your PATH, pointing to the icecc binary. Here sometimes I use GCC 4.4 instead of the default compiler, so I’ll use it as an example. In my machine, GCC 4.4 is installed in /usr/bin/gcc-4.4 and icecream is installed in /opt/icecream/bin:

$ which gcc-4.4
/usr/bin/gcc-4.4
$ which icecc
/opt/icecream/bin/icecc

Go to where icecc is installed and make a symlink to icecc with the same name of the compiler you want to use:

$ sudo ln -s icecc gcc-4.4
$ sudo ln -s icecc g++-4.4

Now, tell icecream to create the environment for this compiler:

$ cd /tmp
$ export ICECC_CC=gcc-4.4
$ export ICECC_CXX=g++4.4
$ icecc --build-native

Now your environment is ready. Copy the file generated to somewhere you’ll remember later (you can give it whatever name you like):

$ sudo cp my-environment-built-above.tar.gz /var/icecream/gcc-4-4.4_x86-64.tar.gz

This step will be done only once, as opposed to the second step below that is repeated whenever you compile a new source.

2. Tell icecream which environment to use

When compiling a source code, tell icecream which environment it will send to other hosts. You do this by exporting some env vars:

$ export ICECC_CC=gcc-4.4
$ export ICECC_CXX=g++-4.4
$ export ICECC_VERSION=/var/icecream/gcc-4-4.4_x86-64.tar.gz

Now you can compile your source code as usual, be it calling gcc directly or through makefiles or other build systems. For example:

$ gcc-4.4 helloworld.c -o helloworld

If you manage a handful of machines running icecream, I’d recommend a software we developed at ProFUSION called Picolé.

UPDATE: if you want a recommendation on how to build a cross toolchain, crossdev it is. The steps are the same as above, replacing gcc-4.4 with the name given to your compiler (e.g. arm-elf-gcc-4.6.0)

by Lucas De Marchi