It's all connected

Patterns for testing asynchronous javascripts.

2012-03-23T10:47:00+01:00

Many javascript test frameworks have a consept of async tests where your tests follow this pattern:

someTestCode.js

    "test http request": function(done) {
        $.post("/some/url", {}, function(result, success) {
            asset.equals("ok", result);
            done();
        });
    }

I spent quite some time debugging a testcase looking like this one, can you spot the error?

The problem with the code above is that done() only gets executed on success. If your request fails, then your test will timeout instead of telling you what happened.

The fixed testcase looks like this:

someTestCode.js

    "test http request": function(done) {
        $.post("/some/url", {}, function(result, success) {
            asset.equals("ok", result);
            done();
        },
        function(error) {
            assert(false, "Ajax failed");
            done();
        }
        );
    }

At least now you test will fail in a way where you will understand what happened.

Lessons for writing JS modules.

Most JS code you want to write is bigger than this example. It is also usually hidden within a module. This implies that you should take note of the done function early when developing the module and make sure it is present and also that it is injectable. Here's one way:

kraken.js

    window.kraken = function(done) {
        var done = done;

        return {
            fetch: function(url) {
                $.post(url, this.success, this.error);
            },
            success: function(arg, arg2) {
                done();
            }
            ...
        };
    }

Ok, but what if your interactions contain multiple async callbacks?

Unittest purists will tell you that you are testing too much, which is true, but sometimes you have a complex setup with more than one callback firing that you want to test. For this I use a countdownLatch:

someTestCode.js

function countdownLatch(num, callback) {
    return {
        num:num,
        callback:callback,
        done: function(msg) {
            this.num--;
            console.log("Done",this.num,  msg);
            if (this.num == 0) {;
                callback();
            }
        }
    }
}
    "test http request": function(done) {
        var latch = countdownLatch(2,function() {
                assert.equals(2, $("#sum").val());
                done();
                });
        var obj = window.kraken(latch);
        obj.fetch("1");
        obj.fetch("2");
    }

The latch above also lets me pass a messate to the done call so I can log progress, something that can be usefull for debugging.

Summary

When you write new JS modules, add support for a done callback for testing.
Remember to wire up errorhandlers for all requests.
You can use the done function for debugging as well.

Injecting HTML With BusterJS

2012-03-23T08:32:00+01:00

I've been working with the very promising new JS testframework named buster.js which seems to be shaping up to become the next great thing in JS testing.

One of the reasons for this I thing is because they try to give you the whole stack, not just a toy for testing your latest implementation of the fibonaci series.

Loading HTML into the DOM of the test is still missing and someting that is quite usefull. JS code does not live in a vacum. They will fix it, see this issue,but until that happens - here's how you can do it.

First setup your config file:

buster.js

    var config = module.exports;
    var fs = require("fs");

    config["someTest"] = {
        rootPath: ".",
        environment: "browser", // or "node"
        libs:  [
         "src/js/mootools-core.js",
        ],
        src: [
            "src/js/my.class.js",
        ],
        tests: [
            "tests/js/test.my.class.js",
        ],
        resources: [
            {path: "/menu", content: fs.readFileSync('src/tests/fixtures/menu.html')},
        ]
    };

In this example I use Mootools, but that is because I'm using mootools on the project I'm working on right now. Pick your poison.

In the test file I load the html fragment and inject it into the dom.

buster.js

    buster.testCase("Test menu handling", {
        setUp: function () {
            var r = new Request.HTML({
                async: false,
                url: buster.env.contextPath + "/menu",
                onSuccess: function(responseTree, responseElements, responseHTML, responseJavaScript) {
                    var body = new Element(document.body);
                    body.adopt(responseElements);
                }
                });
            r.get();
        },
        "Test something with dom": function () {

        }
    }

Note: With this method, the menu.html file should not be a complete html page, but just an html fragment that you can load into the dom.

Howto Export From Mysql to Hadoop Using Sqoop

2012-02-07T10:18:00+01:00

This is a quick howto for working with Apache Sqoop.

I used Clouderas packages of Hadoop and Sqoop for this work. Both the maven packages and the debs.

A basic Sqoop import - MR workflow consist of:

1. Dumping the table info HDFS as sequencefiles:

sqoop import --options-file ./sqoop-options.txt  --as-sequencefile --columns contentobject_id,item --table someTable  --class-name=com.example.MyClass --target-dir=/user/tarjei/db-sqoop/

2. Generate the class that is used in the sequecefile:

sqoop codegen  --options-file ./sqoop-options.txt --outdir src/main/java/ --class-name com.example.MyClass --table someTable

Create a Mapper that works with the data:

The mapper will have the format, LongWritable,Myclass:

SomeMapper.java

    public class SomeMapper extends Mapper<LongWritable,MyClass,Text,Text>{
        public void map(LongWritable id, MyClass, Context context) throws IOException, InterruptedException {
            String someValue = context.getValue();
            ...
        }
    }

Thats it. The parts I missed in the documentation were the part that Sqoop generates a class for you to use when reading the dump files.

Links

The Sqoop user guide is very usefull. Also the Apache Sqoop page.

Handy Symfony2 aliases

2012-01-12T12:47:00+01:00

While working with Symfony2 I've ended up defining a few aliases that might be of use for you. They make it a bit faster to do the things you do all the time, i.e. test and clear caches.

Here they are:

.aliases

PROJECT_DIR=`pwd`
alias cc="$PROJECT_DIR/app/console cache:clear"
alias ccp="$PROJECT_DIR/app/console cache:clear --env=prod"
alias cct="$PROJECT_DIR/app/console cache:clear --env=test"
alias cca="cc --env=prod ; cc --env=test"
alias itest="phpunit -c /$PROJECT_DIRapp/phpunit.xml.ci"
alias con="$PROJECT_DIR/app/console"
alias routes="$PROJECT_DIR/app/console router:debug"
alias log='tail -n 100 $PROJECT_DIR/app/logs/dev.log'
alias plog='tail -n 100 $PROJECT_DIR/app/logs/prod.log'
alias tlog='tail -n 100 $PROJECT_DIR/app/logs/test.log'
alias clear_prod_cache='frigg sudo -u www-data /webroot/vhosts/beta.scanmine.com/nywly/app/console cache:clear --env=demo'
#alias test="phpunit -c $PROJECT_DIR/app/phpunit.xml"
function test {
    
if [ $# -gt 0 ]
then
    echo "Filtering on: $1"
    phpunit -c $PROJECT_DIR/app/phpunit.xml --debug --filter $1
else
phpunit -c $PROJECT_DIR/app/phpunit.xml --verbose 
fi
}
# setup project perms:
# sudo setfacl -R -m u:www-data:rwx -m u:tarjei:rwx app/cache app/logs

Just a quick rundown:

con is an alias to the app/console object.
cc clears the cache
cca clears all the caches
routes lis an alias to the console router:debug command.

test is the command I use the most

It's implemented as a function and the usage is as follows:

Run all tests

% test

Run the tests that hit by a filter

% test MyController::testIndex

Just remember that the aliases override the test function in bash so don't reuse the same shell!

Using PEAR packages in Symfony2 with Pyrus

2011-08-10T10:42:00+02:00

It does not show up when you google for it, but there are ways to handle PEAR packages in Symfony2.

I Found two bundles on symfony2bundles.org:

None of them worked perfectly. I ended up updating the PyrusBundle to something that works fairly well.

You can find the new version up at github/tarjei/PyrusBundle

I added a README to it as well so usage should be fairly simple.

Converted to Octopress

2011-08-09T08:47:00+02:00

I have converted this blog from using Drupal to using Octopress - I didn't need a CMS to handle it. Being a developer, it is easier to handle everything using git and vim.

When I get the time, I'll try to post the conversion scripts. A problem now is that a lot of codelistings do not look good at all. I hope to fix them soon.

Using Pear Packages With Symfony2

2011-08-08T19:58:00+02:00

UPDATE: I improved the metod. See using Symfony2 with PEAR using Pyrus

I've just started using Symfony2 (http://www.symfony.com) and one of the first problems I stumbled across was how to integrate Pear packages into it.

The way I ended up doing this was to create a php directory in my verdors directory and install the packages into this directory using pyrus.

php pyrus.phar mypear ~/SymfonyProject/vendors
php pyrus.phar install pear/HTTP_Request2-2.0.0RC1

Then I added the directories to app/autoload.php:

app/autoload.php

    $loader->registerPrefixes(array(
       'Net_'             => __DIR__.'/../vendor/php',
       'HTTP_'             => __DIR__.'/../vendor/php',
    ));

This isn't perfect as it clutters up the vendors directory with a lot of unneeded directories that follow a Pear installation - it would have been nicer if I could just download a cuple of phar files - so it is probably a good idea to have a separare pear directory under vendors to handle this.

The annotation Template in method SomeClass\SomeBundle\Controller\RestController::addAction() was never imported.

2011-08-05T10:48:00+02:00

When you get this error it is because you need to import the annotation classes into the file you are using them in.

For @Route and @Template this is:

use Sensio\Bundle\FrameworkExtraBundle\Configuration\Route;use Sensio\Bundle\FrameworkExtraBundle\Configuration\Template;

Running large batches with Hibernate

2010-06-25T08:36:00+02:00

This week I've been fighting Hibernate again. That is, last week I should have implemented a quick job to export 30 million entities from a DB and to a new interchange format.

Instead the job blew up due to memoryproblems. Below you'll find teh solution.

First we thought this was due to a Hibernate bug, so we upgraded Hibernate - and spent a day fixing the consequences (thank god, we had tests). This didn't help.

The problem was that the Hibernate querycache didn't evict anything.. The heap was filling up with org.hibernate.engine.query.QueryPlanCache objects.

Now we did what we should have done in the first place. Google'd the terms"hibernate batch jobs". We got these gems:

The changes we did in the end were:

1. Remove the use of Spring autowired DAOs to ensure control over queries.

This is important both to ensure that the number of connections don't spiral out of control AND to clear the Hibernate entity caches.

If you are using a Spring SessonFactory, you can get the session using the SessionFactoryUtils:

Session s = SessionFactoryUtils.getSession(sessionFactory, true);try {doExport(s);} finally {s.flush();s.clear();SessionFactoryUtils.closeSession(s);}

The s.flush() and s.clear() are essential. If not, Hibernate will cache the objects it has created for you forever.

2. Make sure you use parameterized queries.

The point here is that this query:

Query query = s.createQuery("SELECT C.* from Blog where C.id="+ id);

Will be cached separately for EACH time it is called. If you call it 100 000 times, the QueryPlanCache will contain 100 000 entries. The solution is this:

Query query = s.createQuery("SELECT C.* from Blog where C.id=:id");query.setInteger("id", id);

DUH!

3. Love thy jmap

Use Eclipse to analyze jmap heapdumps. It's easy and rewarding. Never try to solve a problem you do not know you have.

4.Bonus camel tip

This does not do what you thought it would:

//.. in some camel route ....to("seda:myQueue").to("file://...")

With this you are essentially sending the message two places. I should have known :)

Groovy Camel MainSupport

2010-06-17T10:48:00+02:00

I've been using Apache Camel more and more lately and one thing that has irritated me is creating a big project with maven, multiple classes etc for just a simple task. I thought Scala would be my savior, instead Groovy is.

This post shows how to get groovy scripts to block when stating up.

The examples I've seen on the web says that all you need is:

def camelCtx = new  DefaultCamelContext()camelCtx.addRoutes(new SampleRoute())camelCtx.start()

(fromGroovy example: ActiveMQ broker and Apache Camel).

Somehow this didn't work for me, and according to the very helpful guys on #camel it shouldn't either.

What happens is that the start() method returns to the calling thread while starting up Camel in the background. The main thread then exits - thus shutting down everything.

I still do not understand how the examples are supposed to work - but I can hope to learn some day :)

According to Claus Ibsen on #camel (and one of the main commiters to the project), thecorrectway to get this to work is to implement a subclass of the abstract class MainSupport that creates the CamelContext and starts it.

There is one implementation of MainSupport in camel-spring that supports loading Spring contexts etc when starting a Camel route, but I wanted something much simpler that would work with as little fuss as possible.

Here's what I came up with:

/*** This is a simple implementation of MainsSupport to handle running simple scripts* as camel services.** Note that the implementation only does one context.* @author Tarjei Huse,tarjei@scanmine.com**/public class GroovyMain extends MainSupport {CamelContext context = new DefaultCamelContext();public GroovyMain() {this.addChildService contextthis.camelContexts.add context}protected  ProducerTemplate findOrCreateCamelTemplate() {return context.createProducerTemplate()}@Overrideprotected void doStart() throws Exception {super.doStart();//context.addRoutes this.getRouteBuilders()postProcessContext(); // does add routes}protected  Map<String, CamelContext> getCamelContextMap() {return ['main': context]}/*** Add a component to the context* @param name* @param component*/public void addComponent(String name, Component component ) {this.context.addComponent(name, component);}public CamelContext getCamelContext() {return context;}protected ModelFileGenerator createModelFileGenerator() throws JAXBException {return null;}/*** Enables the hangup support. Gracefully stops by calling stop() on a* Hangup signal.*/public void enableHangupSupport() {HangupInterceptor interceptor = new HangupInterceptor(this);Runtime.getRuntime().addShutdownHook(interceptor);}}/*** A class for intercepting the hang up signal* and do a graceful shutdown of the Camel.* Taken from the camel-spring implementation of MainSupport.*/class HangupInterceptor extends Thread {MainSupport mainInstance;public HangupInterceptor(MainSupport main) {mainInstance = main;}@Overridepublic void run() {try {mainInstance.stop();} catch (Exception ex) {Logger.getLogger("Shutdown handler").warn(ex);}}}

The implementation only handles one simple camel context. It also adds support for adding components to the context.

Munin FAQ Entry: Why aren't my graphs updating

2010-03-28T21:46:00+02:00

I just upgraded to Munin 1.4.3 on Centos 5.4 (from 1.3.x I think) the other day and after some time noticed that my Graphs were not updating.

After a long bout of debugging and cursing Munin's inability to provide any indications of what was wrong, I suddenly understood my error: I was deeplinking into pages that were not updated anymore because the Munin upgrade changed the way generated files are structured.

The following is the quick fix:

su -s /bin/bash muninfind . -name"*.html"-ls -exec rm -f {} \;/usr/bin/munin-cron

I.e delete all html files generated by munin and ask it to generate them over again _as the munin user_.

I sure hope this helps someone!

Loading different fixtures based on environment

2009-12-16T10:51:00+01:00

A few days ago someone asked on #symfony if it is possible to make symfony load fixtures from different places depending on the environment.

To do this, you have to define your own task. Here's one I just created. As a bonus, this task also loads sql files from a separate sql directory.

Here's the task:

<?php/*** This task does almost the same as doctrine:data-load, but it also* loads the sql from the files in the data/sqlextra directory And takes into account the current environment.*** Use instead of doctrine:data-load*/class nnv_load_dataTask extends sfBaseTask{protected function configure(){$this->addOptions(array(new sfCommandOption('application', null, sfCommandOption::PARAMETER_REQUIRED,'The application name'),new sfCommandOption('env', null, sfCommandOption::PARAMETER_REQUIRED,'The environment','dev'),new sfCommandOption('connection', null, sfCommandOption::PARAMETER_REQUIRED,'The connection name','doctrine'),// add your own options here));$this->namespace        ='data';$this->name             ='load-data';$this->briefDescription ='Load fixtures and sql based on environment';$this->detailedDescription =<<<EOFThis [nnv_load_data|INFO] task does almost the same as doctrine:data-load, but it alsoloads the sql from the files in the data/sqlextra/$env directory and loads fixtures from the data/fixtures/$env dir.Use instead of doctrine:data-loadCall it with:[php symfony load_extra_sql|INFO]EOF;}protected function execute($arguments = array(), $options = array()){// initialize the database connection$databaseManager = new sfDatabaseManager($this->configuration);$connection = $databaseManager->getDatabase($options['connection'] ? $options['connection'] : null)->getConnection();Doctrine::loadData(sfConfig::get('sf_root_dir').'/data/fixtures/'. $env);$env = $options['env'];$files = glob(sfConfig::get('sf_root_dir')."/data/sqlextra/$env/*.sql");foreach ($files as $file) {print"Loading". $file ."...";$sql = file_get_contents($file) ;try {$n = $connection->exec($sql);print"ok: $n\n";} catch (Exception $e) {print"Error loading statement: \n$stmt\n:". $e->getMessage();}}}}

Testing symfony with sqlite::memory:

2009-12-03T12:41:00+01:00

One thing I love about Java is that there are a number of databases that you can load into memory and run functional tests against. This is also an option in PHP thanks to Sqlite.

To do this in Symfony, you must configure the test db like this:

test:doctrine:param:      dsn:"sqlite::memory:"

And setup the test to reload the db each time you call it:

<?php require_once dirname(__FILE__).'/../bootstrap/functional.php';$configuration = ProjectConfiguration::getApplicationConfiguration('admin','test', true);new sfDatabaseManager($configuration);Doctrine::createTablesFromModels(dirname(__FILE__).'/../../lib/model');Doctrine::loadData(sfConfig::get('sf_test_dir').'/fixtures');$t = new lime_test(2);

Update:

I had some trouble loading fixtures where the ids could not be autogenerated. In the end I ended up defining a separate function for loading files like these. Here's the complete bootstrap file I've ended up with:

<?php/** This file is part of the symfony package.* (c) Fabien Potencier<fabien.potencier@symfony-project.com>** For the full copyright and license information, please view the LICENSE* file that was distributed with this source code.*/$_test_dir = realpath(dirname(__FILE__).'/..');// configurationrequire_once dirname(__FILE__).'/../../config/ProjectConfiguration.class.php';$configuration = ProjectConfiguration::hasActive() ? ProjectConfiguration::getActive() : new ProjectConfiguration(realpath($_test_dir.'/..'));// autoloader$autoload = sfSimpleAutoload::getInstance(sfConfig::get('sf_cache_dir').'/project_autoload.cache');$autoload->loadConfiguration(sfFinder::type('file')->name('autoload.yml')->in(array(sfConfig::get('sf_symfony_lib_dir').'/config/config',sfConfig::get('sf_config_dir'),)));$autoload->register();// limeinclude $configuration->getSymfonyLibDir().'/vendor/lime/lime.php';//$configuration = ProjectConfiguration::getApplicationConfiguration('admin','test', true);new sfDatabaseManager(ProjectConfiguration::getApplicationConfiguration('admin','test', true));Doctrine::createTablesFromModels(sfConfig::get('sf_root_dir').'/lib/model');Doctrine::loadData(sfConfig::get('sf_test_dir').'/fixtures');/*** Load testdata from an sqldump. This is used when running tests where the* id's are significant (i.e. fylker, kommuner and postnummer)* this function expects a connection to exist* @param $file string path to the file to load*/function loadSqlDump($file) {$sql = file_get_contents($file);if ($sql =='') throw new Exception("No sql found in $file");$conn2 = Doctrine_Manager::connection();$sql = trim($sql);$dbo = $conn2->getDbh();// sqlite doesn't like multiple statements$statements = explode(";", $sql);foreach ($statements as $stmt) {try {$n = $dbo->exec($stmt);} catch (Exception $e) {print"Error loading statement: \n$stmt\n:". $e->getMessage();}}}

Note that I place these DB tests in the unittest directory.

Converting from DBCP to C3P0 with Spring

2009-11-16T11:42:00+01:00

Just a few quick notes on things to do when converting from using the DBCP connectionpool implementation to using C3P0

You start by changing the dataSourcein your Spring config from:

        class="org.springframework.jdbc.datasource.DriverManagerDataSource">








To:

          class="com.mchange.v2.c3p0.ComboPooledDataSource"

          destroy-method="close">

       com.mysql.jdbc.Driver







       3

       200

       0

       3



1. Name differences:
There are some variable name differences in the xml above. Note:

url becomes jdbcUrl
driverClassName becomes driverClass
username becomes user

2. Some variables must be moved into the sessionFactory configuration
According tothisyou must configure some of the variables into the hibernate config. The article is from 2006 so this might not be true anymore.

I add the properties like this so that they can be in the same config file:








               true

               create

          1

           3

         50

               org.hibernate.dialect.MySQLDialect

Some other notes regarding connection pooling with the above:

If you use DBUnit, remember to close the connection so it gets returned to the pool after you have set up the db.

If you use SessionFactoryUtils to get a session, remember to do SessionFactoryUtils.releaseSession() when done.

Using MySQL for storing images

2009-11-15T15:29:00+01:00

You do not always have the luxury to decide how things are done. This weekend the image service at work decided it was its turn to wreck itself.

The whole story starts of with me rewriting a part of the image service - the component that inserts an image into the service.

The service has two parts: One that saves images to the datastore and one that retrieves images from the datastore and delivers them to some client, usually as a thumbnail. This should be fairly simple stuff that should never go wrong and usually just work. The problem was in the chosen datastore: MySQL.

MySQL is good for many things, but far too often have I seen it used as a hammer where a screwdriver should have been used.

The schema for image database looked like this:

  CREATE TABLE IF NOT EXISTS \`image\` (  
      \`objectid\` int(11) NOT NULL default'0',  
      \`imagedata\` longblob,  
      \`imagewidth\` int(4) default NULL,  
      \`imageheight\` int(4) default NULL,  
      \`imagemime\` varchar(4) default NULL,  
      \`imageurl\` text,  
      PRIMARY KEY  (\`objectid\`) ) ENGINE=MyISAM;

I'll comment on a few of the fields above. The worst field here is probably the imagemime field. Notice that it is a varchar of max 4 characters. The idea is to only store the specific part of the mimetype, i.e. pjpeg from image/pjpeg. Notice that the author thought that imagetypes only come in names of less than 4 characters (jpeg, png and gif) and thus we would have no problems with the size of the field.

The error caused by the service trying to thumnail pjpe images was what made me look at the image service.

To fix this bug we issued

 "ALTER TABLE \`image\` CHANGE \`imagemime\` \`imagemime\` VARCHAR( 10 )"

, thinking that this should work ok.

Bad idea.

The problem is that because this command should work atomically, MySQL will do all the work in a separate file and then mv the file over the original file when done. What do you think happens when there is too little space left on the disk? MySQL hangs, crashes and burns.

Lesson learned: Always have more space left on the MySQL data partition than the size of the largest table(.MYD) file.

If not, you cannot perform recovery!

Or in my case, change the table.

The next thing I tried was deleting a large amount of the images in the store as they were old and had no value. This again blew up because of too little space.The solution to this mess was to move the whole database over to another partition, run myisamchk and then move it back.

How should this service have been designed?

I am not an expert on designing online storage facilities for images, but one thing that I would like to note is the storing of metadata together with the file - but not in the file itself.

Most image formats for the web store image height and width in the start of the image and you can use metadata formats like IPTC to store even more information into the image. Storing metadata in the file makes it easier to retrieve the file since you do not need to fetch additional metadata that is already in the file.This would have made it possible to simplify the storage schema. You may want to keep a copy of the metadata in a store to be able to query it, but for most operations it will not be needed since you want the file at the same time anyway.

Storing the files in the db is also a bad idea (IMHO). It may well be faster to serve binary objects from a database, but this comes at a price. The service is harder to backup, dependent on an extra component (the db) and thus much more complicated.There are two designs I've considered for a simple image service.

Either build a tiny layer aroundMogile FSor useGlusterFSto replicate the filesystem where the files are stored.

Note: I would probably have done the same mistake myself if I got asked to create a system like this a year ago. I just hope someone reads this in time to not repeat the mistakes.

Attendum about ActiveMQ: Sometimes you do not have the luxury to keep every service separated on different machines. When the disk filled up due to the MySQL problems above I also learned that if you run Active MQ with the Kaha persistent storage db then you should keep it on a separate partition - or else loose all messages when you run out of disk space.

Using Pythons logging module with unittest

2009-10-08T08:39:00+02:00

Problem: setUp() is called for every method but I only want to start logging once. Solution:Add a check on logger.handlers to see if any handlers have been registered with the logger:def startLogging(): logger = logging.getLogger() if len(logger.handlers): return logger stdout = logging.StreamHandler() stdout.setLevel(logging.DEBUG) logger.addHandler(stdout) return loggerclass MyTest(unittest.TestCase): def setUp(self): startLogging() def test_something(self): some_function_that_uses_logging()

TransactionManagerLookupFactory - No TransactionManagerLookup configured

2009-09-01T15:55:00+02:00

I spent the best part of yesterday afternoon fighting the message above. I post this so that the next person fighting it may save some time. One of the problems was that none ofthe solutions suggestedhelped. After some beer, sleep and a fresh start I foundthis messagethat talks about replacing

<tx:annotation-driven transaction-manager="transactionManager" />

With

<code><bean class="org.springframework.transaction.aspectj.AnnotationTransactionAspect"factory-method="aspectOf"><property name="transactionManager" ref="transactionManager"></property></bean></code>

And suddenly it all worked. Another example of GDD - Google Driven Development. If anyone knows why this works - please explain.

Update: There seems to be an issue when it comes to combining @Confgurable and @Transactional in a project that relates to the comment above. Also, I later experienced that doing sessionFactory.getCurrentsession() is not a good idea when using spring transactions. Instead, use

Update 2:None of the examples of Transactional tests using spring show it, but you'll need the TransactionalTestExecutionListener to get @AfterTransaction and @BeforeTransaction to work. Det it up like this:

@TestExecutionListeners( { TransactionalTestExecutionListener.class })

umount: device is busy

2009-08-28T10:29:00+02:00


  #umount /var<br /><br />umount: device is busy<br />

Who hasn't experienced this one?

Anyhow, I foundthisshellscript that does the important checks:

!/bin/sh

Attempts to umount a mountpoint or device. If it fails, give info on why it might be busy.

/bin/umount $ || (
    if [ -z"$" ]
    then
       exit
    fi
    echo `pwd`/"$1" | sed's/^{.*\/\//\//g'|} ( read abspath

    mtab_entry=`grep"$abspath"< /etc/mtab`
    echo $mtab_entry | if grep"/"> /dev/null
    then
       device=`echo $mtab_entry | sed"s/ .$//g"`
       mountpoint=`echo $mtab_entry | sed"s/[^ ] //" | sed"s/ .*$//g"`
       #echo $device $mountpoint

       ( echo The mountpoint $mountpoint is in use by:
        lsof | grep" $mountpoint" | head     #list files open
           cat /etc/mtab | grep" $mountpoint/" #list sub mount points
       ) 1>&2
    fi
))

Securing the web: PHP CMSs should autodestruct

2009-07-20T09:31:00+02:00

I've been using Drupal a bit lately for some simple sites and just happened to come overthisdebate on the relative security between Drupal and Plone. I think we need to go beond looking at the number of Security advisories and start discussing how we protect vulnerable versions of software. The problem with the larger PHP CMS'es (Drupal, Joomla etc) isn't that they do not care about security - but that that their users do not.One thing I've noticed is that there are a lot of simple sites out there that get hacked because they do not upgrade their CMS'es to the latest version. Often the hacker keeps the site up so that it is impossible to notice that that the account is being used for all kinds of spamming - for example by adding a bunch of"invisible"links at the bottom of the site. I've tried to contact some ISP's that hosts sites like this (Servetheworld amongst others), but their reply is that they cannot disable a customers account or clean up the customers site just because it is hacked. IMHO we need a way to protect users against themselves here. A simple protocol that checks the version of the running CMS against a registry somewhere and then either shuts down the CMS and/or notifies the user.At least 50% of the people running small sites do not know enough about the security to keep their site up to date. We need to provide protection for them.

MYSQL datatruncation errors

2009-06-22T08:38:00+02:00

Few things can provide as much frustration as refactoring a legacy system bit by bit. Today I have been wrestling with a field defined in MySQL as"timestamp default'0000-00-00 00:00:00'. In my (reveng'ed) POJO the field is defined like this:

@Temporal(TemporalType.TIMESTAMP)    @Column(name ="httpstatus_time", nullable = false, length = 19,
    columnDefinition="timestamp default'0000-00-00 00:00:00'")    
    public Date getHttpstatusTime() {       
      return this.httpstatusTime;
    }

Now, it turns out that JDBC cannot deal with date values that are before UNIX epoch, something 0000-00-00 clearly is. So, what does JDBC do in these situations? It will throw an exception. To get around this, you can add'zeroDateTimeBehavior=convertToNull'to your jdbc url. After setting zeroDateTimeBehaviour, the value will be set to a Javanull. This provides you with another issue: When you get an object from the db, and you modify it, you may have to also modify some other values and set them to something other than null.

Note: I tried setting the dates to"new Date(1)" but that didn't work - it is too close to 0 so JDBC still throws a fit. I ended up with"new Date(10000)".

If you do not check your null's, you'll get this: (I've omitted parts of the stacktrace for brevity)

org.springframework.dao.DataIntegrityViolationException: not-null property references a null or transient value: com.scanmine.scheduler.server.model.Source.httpstatusTime;
 nested exception is org.hibernate.PropertyValueException: not-null property references a null or transient value: com.example.Myclass.httpstatusTime    at org.springframework.orm.hibernate3.SessionFactoryUtils.convertHibernateAccessException...Caused by: org.hibernate.PropertyValueException: not-null property references a null or transient value:com.example.Myclass.httpstatusTime...Caused by: org.hibernate.PropertyValueException: not-null property references anull or transient value: com.example.Myclass.httpstatusTime

Not fun.Bonus tipIf you use DBUnit to test your legacy db and want to add values lik'0000-00-00 00:00:00'you will hit the same bug as described above. To get around it, add jdbcCompliantTruncation=false to the connection url.

Links

http://bugs.mysql.com/bug.php?id=3331