When your testsuite starts to grow you’ll soon find yourself wishing you had spent more time creating simple unittests with mockups and created less of those fancy Symfony2 functional tests.

Here are some tips that you can use to speed up any PHPUnit testssuite - with a Symfony2 bonus tip.

I start off with this Xdebug configuration:

zend_extension=/usr/lib/php5/20090626/xdebug.so

xdebug.show_mem_delta = 1
xdebug.profiler_enable_trigger=1
xdebug.max_nesting_level=600
xdebug.profiler_enable=0
xdebug.coverage_enable=0

Now, my research found tips on how to disable xdebug by adding and removing a config file from PHPs INI_SCAN_DIR, but I found that that was way too much hassle for me. Instead I enable xdebug by default and disable all options.( http://www.boxuk.com/blog/fast-phpunit-and-xdebug-code-coverage/)

Tip 1: Disable as much as possible of Xdebug.

For your normal tests, disable xdebug as much as you can:

<php>
    <ini name="xdebug.default_enable" value="0" />
    <ini name="xdebug.remote_autostart" value="0" />
    <ini name="xdebug.remote_enable" value="0" />
    <ini name="xdebug.overload_var_dump" value="0" />
    <ini name="xdebug.show_mem_delta" value="0" />
</php>

Tip 2: Only enable coverage when you need it.

Keep a separate phpunit file for coverage tests where you enable coverage:

<php>
    <ini name="xdebug.default_enable" value="1" />
    <ini name="xdebug.enable_coverage" value="1" />
    <ini name="xdebug.remote_autostart" value="0" />
    <ini name="xdebug.remote_enable" value="0" />
    <ini name="xdebug.overload_var_dump" value="0" />
    <ini name="xdebug.show_mem_delta" value="0" />
</php>

Bonus tip:

Originally this tip comes from Kris Wallsmith.

Add this to your Symfony2 kernel:

class AppKernel extends Kernel
{
    // ...

    protected function initializeContainer()
    {
        static $first = true;

        if ('test' !== $this->getEnvironment()) {
            parent::initializeContainer();
            return;
        }

        $debug = $this->debug;

        if (!$first) {
            // disable debug mode on all but the first initialization
            $this->debug = false;
        }

        // will not work with --process-isolation
        $first = false;

        try {
            parent::initializeContainer();
        } catch (\Exception $e) {
            $this->debug = $debug;
            throw $e;
        }

        $this->debug = $debug;
    }
}

And your tests will run much faster because Symfony2 doesn’t check if any resources have been changed between each time it starts a new kernel.

Many javascript test frameworks have a consept of async tests where your tests follow this pattern:

1
2
3
4
5
6

    "test http request": function(done) {
        $.post("/some/url", {}, function(result, success) {
            asset.equals("ok", result);
            done();
        });
    }

I spent quite some time debugging a testcase looking like this one, can you spot the error?

The problem with the code above is that done() only gets executed on success. If your request fails, then your test will timeout instead of telling you what happened.

The fixed testcase looks like this:

1
2
3
4
5
6
7
8
9
10
11

    "test http request": function(done) {
        $.post("/some/url", {}, function(result, success) {
            asset.equals("ok", result);
            done();
        },
        function(error) {
            assert(false, "Ajax failed");
            done();
        }
        );
    }

At least now you test will fail in a way where you will understand what happened.

Lessons for writing JS modules.

Most JS code you want to write is bigger than this example. It is also usually hidden within a module. This implies that you should take note of the done function early when developing the module and make sure it is present and also that it is injectable. Here’s one way:

1
2
3
4
5
6
7
8
9
10
11
12
13

    window.kraken = function(done) {
        var done = done;

        return {
            fetch: function(url) {
                $.post(url, this.success, this.error);
            },
            success: function(arg, arg2) {
                done();
            }
            ...
        };
    }

Ok, but what if your interactions contain multiple async callbacks?

Unittest purists will tell you that you are testing too much, which is true, but sometimes you have a complex setup with more than one callback firing that you want to test. For this I use a countdownLatch:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

function countdownLatch(num, callback) {
    return {
        num:num,
        callback:callback,
        done: function(msg) {
            this.num--;
            console.log("Done",this.num,  msg);
            if (this.num == 0) {;
                callback();
            }
        }
    }
}
    "test http request": function(done) {
        var latch = countdownLatch(2,function() {
                assert.equals(2, $("#sum").val());
                done();
                });
        var obj = window.kraken(latch);
        obj.fetch("1");
        obj.fetch("2");
    }

The latch above also lets me pass a messate to the done call so I can log progress, something that can be usefull for debugging.

Summary

1. When you write new JS modules, add support for a done callback for testing.
2. Remember to wire up errorhandlers for all requests.
3. You can use the done function for debugging as well.

I’ve been working with the very promising new JS testframework named buster.js which seems to be shaping up to become the next great thing in JS testing.

One of the reasons for this I thing is because they try to give you the whole stack, not just a toy for testing your latest implementation of the fibonaci series.

Loading HTML into the DOM of the test is still missing and someting that is quite usefull. JS code does not live in a vacum. They will fix it, see this issue,but until that happens - here’s how you can do it.

First setup your config file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

    var config = module.exports;
    var fs = require("fs");

    config["someTest"] = {
        rootPath: ".",
        environment: "browser", // or "node"
        libs:  [
         "src/js/mootools-core.js",
        ],
        src: [
            "src/js/my.class.js",
        ],
        tests: [
            "tests/js/test.my.class.js",
        ],
        resources: [
            {path: "/menu", content: fs.readFileSync('src/tests/fixtures/menu.html')},
        ]
    };

In this example I use Mootools, but that is because I’m using mootools on the project I’m working on right now. Pick your poison.

In the test file I load the html fragment and inject it into the dom.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

    buster.testCase("Test menu handling", {
        setUp: function () {
            var r = new Request.HTML({
                async: false,
                url: buster.env.contextPath + "/menu",
                onSuccess: function(responseTree, responseElements, responseHTML, responseJavaScript) {
                    var body = new Element(document.body);
                    body.adopt(responseElements);
                }
                });
            r.get();
        },
        "Test something with dom": function () {

        }
    }

Note: With this method, the menu.html file should not be a complete html page, but just an html fragment that you can load into the dom.

This is a quick howto for working with Apache Sqoop.

I used Clouderas packages of Hadoop and Sqoop for this work. Both the maven packages and the debs.

A basic Sqoop import - MR workflow consist of:

1. Dumping the table info HDFS as sequencefiles:

sqoop import --options-file ./sqoop-options.txt  --as-sequencefile --columns contentobject_id,item --table someTable  --class-name=com.example.MyClass --target-dir=/user/tarjei/db-sqoop/

2. Generate the class that is used in the sequecefile:

sqoop codegen  --options-file ./sqoop-options.txt --outdir src/main/java/ --class-name com.example.MyClass --table someTable

Create a Mapper that works with the data:

The mapper will have the format, LongWritable,Myclass:

1
2
3
4
5
6

    public class SomeMapper extends Mapper<LongWritable,MyClass,Text,Text>{
        public void map(LongWritable id, MyClass, Context context) throws IOException, InterruptedException {
            String someValue = context.getValue();
            ...
        }
    }

Thats it. The parts I missed in the documentation were the part that Sqoop generates a class for you to use when reading the dump files.

Links

The Sqoop user guide is very usefull. Also the Apache Sqoop page.

While working with Symfony2 I’ve ended up defining a few aliases that might be of use for you. They make it a bit faster to do the things you do all the time, i.e. test and clear caches.

Here they are:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

PROJECT_DIR=`pwd`
alias cc="$PROJECT_DIR/app/console cache:clear"
alias ccp="$PROJECT_DIR/app/console cache:clear --env=prod"
alias cct="$PROJECT_DIR/app/console cache:clear --env=test"
alias cca="cc --env=prod ; cc --env=test"
alias itest="phpunit -c /$PROJECT_DIRapp/phpunit.xml.ci"
alias con="$PROJECT_DIR/app/console"
alias routes="$PROJECT_DIR/app/console router:debug"
alias log='tail -n 100 $PROJECT_DIR/app/logs/dev.log'
alias plog='tail -n 100 $PROJECT_DIR/app/logs/prod.log'
alias tlog='tail -n 100 $PROJECT_DIR/app/logs/test.log'
alias clear_prod_cache='frigg sudo -u www-data /webroot/vhosts/beta.scanmine.com/nywly/app/console cache:clear --env=demo'
#alias test="phpunit -c $PROJECT_DIR/app/phpunit.xml"
function test {
    
if [ $# -gt 0 ]
then
    echo "Filtering on: $1"
    phpunit -c $PROJECT_DIR/app/phpunit.xml --debug --filter $1
else
phpunit -c $PROJECT_DIR/app/phpunit.xml --verbose 
fi
}
# setup project perms:
# sudo setfacl -R -m u:www-data:rwx -m u:tarjei:rwx app/cache app/logs

Just a quick rundown:

• con is an alias to the app/console object.
• cc clears the cache
• cca clears all the caches
• routes lis an alias to the console router:debug command.

test is the command I use the most

It’s implemented as a function and the usage is as follows:

Run all tests

% test

Run the tests that hit by a filter

% test MyController::testIndex

Just remember that the aliases override the test function in bash so don’t reuse the same shell!

It does not show up when you google for it, but there are ways to handle PEAR packages in Symfony2.

I Found two bundles on symfony2bundles.org:

• Pearbundle
• PyrusBundle

None of them worked perfectly. I ended up updating the PyrusBundle to something that works fairly well.

You can find the new version up at github/tarjei/PyrusBundle

I added a README to it as well so usage should be fairly simple.

I have converted this blog from using Drupal to using Octopress - I didn’t need a CMS to handle it. Being a developer, it is easier to handle everything using git and vim.

When I get the time, I’ll try to post the conversion scripts. A problem now is that a lot of codelistings do not look good at all. I hope to fix them soon.

UPDATE: I improved the metod. See using Symfony2 with PEAR using Pyrus

I’ve just started using Symfony2 (http://www.symfony.com) and one of the first problems I stumbled across was how to integrate Pear packages into it.

The way I ended up doing this was to create a php directory in my verdors directory and install the packages into this directory using pyrus.

php pyrus.phar mypear ~/SymfonyProject/vendors
php pyrus.phar install pear/HTTP_Request2-2.0.0RC1

Then I added the directories to app/autoload.php:

1
2
3
4
5

    $loader->registerPrefixes(array(
       'Net_'             => __DIR__.'/../vendor/php',
       'HTTP_'             => __DIR__.'/../vendor/php',
    ));
    

This isn’t perfect as it clutters up the vendors directory with a lot of unneeded directories that follow a Pear installation - it would have been nicer if I could just download a cuple of phar files - so it is probably a good idea to have a separare pear directory under vendors to handle this.

When you get this error it is because you need to import the annotation classes into the file you are using them in.

For @Route and @Template this is:

use Sensio\Bundle\FrameworkExtraBundle\Configuration\Route;use Sensio\Bundle\FrameworkExtraBundle\Configuration\Template;

This week I’ve been fighting Hibernate again. That is, last week I should have implemented a quick job to export 30 million entities from a DB and to a new interchange format.

Instead the job blew up due to memoryproblems. Below you’ll find teh solution.

First we thought this was due to a Hibernate bug, so we upgraded Hibernate - and spent a day fixing the consequences (thank god, we had tests). This didn’t help.

The problem was that the Hibernate querycache didn’t evict anything.. The heap was filling up with org.hibernate.engine.query.QueryPlanCache objects.

Now we did what we should have done in the first place. Google’d the terms”hibernate batch jobs”. We got these gems:

• Hibernate queries should be parameterized or you’ll have a problem.
• Batch processing in Hibernate

The changes we did in the end were:

1. Remove the use of Spring autowired DAOs to ensure control over queries.

This is important both to ensure that the number of connections don’t spiral out of control AND to clear the Hibernate entity caches.

If you are using a Spring SessonFactory, you can get the session using the SessionFactoryUtils:

Session s = SessionFactoryUtils.getSession(sessionFactory, true);try {doExport(s);} finally {s.flush();s.clear();SessionFactoryUtils.closeSession(s);}

The s.flush() and s.clear() are essential. If not, Hibernate will cache the objects it has created for you forever.

2. Make sure you use parameterized queries.

The point here is that this query:

Query query = s.createQuery("SELECT C.* from Blog where C.id="+ id);

Will be cached separately for EACH time it is called. If you call it 100 000 times, the QueryPlanCache will contain 100 000 entries. The solution is this:

Query query = s.createQuery("SELECT C.* from Blog where C.id=:id");query.setInteger("id", id);

DUH!

3. Love thy jmap

Use Eclipse to analyze jmap heapdumps. It’s easy and rewarding. Never try to solve a problem you do not know you have.

4.Bonus camel tip

This does not do what you thought it would:

//.. in some camel route ....to("seda:myQueue").to("file://...")

With this you are essentially sending the message two places. I should have known :)