This is a quick howto for working with Apache Sqoop.

I used Clouderas packages of Hadoop and Sqoop for this work. Both the maven packages and the debs.

A basic Sqoop import - MR workflow consist of:

1. Dumping the table info HDFS as sequencefiles:

sqoop import --options-file ./sqoop-options.txt  --as-sequencefile --columns contentobject_id,item --table someTable  --class-name=com.example.MyClass --target-dir=/user/tarjei/db-sqoop/

2. Generate the class that is used in the sequecefile:

sqoop codegen  --options-file ./sqoop-options.txt --outdir src/main/java/ --class-name com.example.MyClass --table someTable

Create a Mapper that works with the data:

The mapper will have the format, LongWritable,Myclass:

1
2
3
4
5
6

    public class SomeMapper extends Mapper<LongWritable,MyClass,Text,Text>{
        public void map(LongWritable id, MyClass, Context context) throws IOException, InterruptedException {
            String someValue = context.getValue();
            ...
        }
    }

Thats it. The parts I missed in the documentation were the part that Sqoop generates a class for you to use when reading the dump files.

Links

The Sqoop user guide is very usefull. Also the Apache Sqoop page.

While working with Symfony2 I’ve ended up defining a few aliases that might be of use for you. They make it a bit faster to do the things you do all the time, i.e. test and clear caches.

Here they are:

PROJECT_DIR=`pwd`
alias cc="$PROJECT_DIR/app/console cache:clear"
alias ccp="$PROJECT_DIR/app/console cache:clear --env=prod"
alias cct="$PROJECT_DIR/app/console cache:clear --env=test"
alias cca="cc --env=prod ; cc --env=test"
alias itest="phpunit -c /$PROJECT_DIRapp/phpunit.xml.ci"
alias con="$PROJECT_DIR/app/console"
alias routes="$PROJECT_DIR/app/console router:debug"
alias log='tail -n 100 $PROJECT_DIR/app/logs/dev.log'
alias plog='tail -n 100 $PROJECT_DIR/app/logs/prod.log'
alias tlog='tail -n 100 $PROJECT_DIR/app/logs/test.log'
alias clear_prod_cache='frigg sudo -u www-data /webroot/vhosts/beta.scanmine.com/nywly/app/console cache:clear --env=demo'
#alias test="phpunit -c $PROJECT_DIR/app/phpunit.xml"
function test {
    
if [ $# -gt 0 ]
then
    echo "Filtering on: $1"
    phpunit -c $PROJECT_DIR/app/phpunit.xml --debug --filter $1
else
phpunit -c $PROJECT_DIR/app/phpunit.xml --verbose 
fi
}
# setup project perms:
# sudo setfacl -R -m u:www-data:rwx -m u:tarjei:rwx app/cache app/logs

Just a quick rundown:

• con is an alias to the app/console object.
• cc clears the cache
• cca clears all the caches
• routes lis an alias to the console router:debug command.

test is the command I use the most

It’s implemented as a function and the usage is as follows:

Run all tests

% test

Run the tests that hit by a filter

% test MyController::testIndex

Just remember that the aliases override the test function in bash so don’t reuse the same shell!

It does not show up when you google for it, but there are ways to handle PEAR packages in Symfony2.

I Found two bundles on symfony2bundles.org:

• Pearbundle
• PyrusBundle

None of them worked perfectly. I ended up updating the PyrusBundle to something that works fairly well.

You can find the new version up at github/tarjei/PyrusBundle

I added a README to it as well so usage should be fairly simple.

I have converted this blog from using Drupal to using Octopress - I didn’t need a CMS to handle it. Being a developer, it is easier to handle everything using git and vim.

When I get the time, I’ll try to post the conversion scripts. A problem now is that a lot of codelistings do not look good at all. I hope to fix them soon.

UPDATE: I improved the metod. See using Symfony2 with PEAR using Pyrus

I’ve just started using Symfony2 (http://www.symfony.com) and one of the first problems I stumbled across was how to integrate Pear packages into it.

The way I ended up doing this was to create a php directory in my verdors directory and install the packages into this directory using pyrus.

php pyrus.phar mypear ~/SymfonyProject/vendors
php pyrus.phar install pear/HTTP_Request2-2.0.0RC1

Then I added the directories to app/autoload.php:

1
2
3
4

    $loader->registerPrefixes(array(
       'Net_'             => __DIR__.'/../vendor/php',
       'HTTP_'             => __DIR__.'/../vendor/php',
    ));

This isn’t perfect as it clutters up the vendors directory with a lot of unneeded directories that follow a Pear installation - it would have been nicer if I could just download a cuple of phar files - so it is probably a good idea to have a separare pear directory under vendors to handle this.

When you get this error it is because you need to import the annotation classes into the file you are using them in.

For @Route and @Template this is:

use Sensio\Bundle\FrameworkExtraBundle\Configuration\Route;use Sensio\Bundle\FrameworkExtraBundle\Configuration\Template;

This week I’ve been fighting Hibernate again. That is, last week I should have implemented a quick job to export 30 million entities from a DB and to a new interchange format.

Instead the job blew up due to memoryproblems. Below you’ll find teh solution.

First we thought this was due to a Hibernate bug, so we upgraded Hibernate - and spent a day fixing the consequences (thank god, we had tests). This didn’t help.

The problem was that the Hibernate querycache didn’t evict anything.. The heap was filling up with org.hibernate.engine.query.QueryPlanCache objects.

Now we did what we should have done in the first place. Google’d the terms”hibernate batch jobs”. We got these gems:

• Hibernate queries should be parameterized or you’ll have a problem.
• Batch processing in Hibernate

The changes we did in the end were:

1. Remove the use of Spring autowired DAOs to ensure control over queries.

This is important both to ensure that the number of connections don’t spiral out of control AND to clear the Hibernate entity caches.

If you are using a Spring SessonFactory, you can get the session using the SessionFactoryUtils:

Session s = SessionFactoryUtils.getSession(sessionFactory, true);try {doExport(s);} finally {s.flush();s.clear();SessionFactoryUtils.closeSession(s);}

The s.flush() and s.clear() are essential. If not, Hibernate will cache the objects it has created for you forever.

2. Make sure you use parameterized queries.

The point here is that this query:

Query query = s.createQuery("SELECT C.* from Blog where C.id="+ id);

Will be cached separately for EACH time it is called. If you call it 100 000 times, the QueryPlanCache will contain 100 000 entries. The solution is this:

Query query = s.createQuery("SELECT C.* from Blog where C.id=:id");query.setInteger("id", id);

DUH!

3. Love thy jmap

Use Eclipse to analyze jmap heapdumps. It’s easy and rewarding. Never try to solve a problem you do not know you have.

4.Bonus camel tip

This does not do what you thought it would:

//.. in some camel route ....to("seda:myQueue").to("file://...")

With this you are essentially sending the message two places. I should have known :)

I’ve been using Apache Camel more and more lately and one thing that has irritated me is creating a big project with maven, multiple classes etc for just a simple task. I thought Scala would be my savior, instead Groovy is.

This post shows how to get groovy scripts to block when stating up.

The examples I’ve seen on the web says that all you need is:

def camelCtx = new  DefaultCamelContext()camelCtx.addRoutes(new SampleRoute())camelCtx.start()

(fromGroovy example: ActiveMQ broker and Apache Camel).

Somehow this didn’t work for me, and according to the very helpful guys on #camel it shouldn’t either.

What happens is that the start() method returns to the calling thread while starting up Camel in the background. The main thread then exits - thus shutting down everything.

I still do not understand how the examples are supposed to work - but I can hope to learn some day :)

According to Claus Ibsen on #camel (and one of the main commiters to the project), thecorrectway to get this to work is to implement a subclass of the abstract class MainSupport that creates the CamelContext and starts it.

There is one implementation of MainSupport in camel-spring that supports loading Spring contexts etc when starting a Camel route, but I wanted something much simpler that would work with as little fuss as possible.

Here’s what I came up with:

/*** This is a simple implementation of MainsSupport to handle running simple scripts* as camel services.** Note that the implementation only does one context.* @author Tarjei Huse,tarjei@scanmine.com**/public class GroovyMain extends MainSupport {CamelContext context = new DefaultCamelContext();public GroovyMain() {this.addChildService contextthis.camelContexts.add context}protected  ProducerTemplate findOrCreateCamelTemplate() {return context.createProducerTemplate()}@Overrideprotected void doStart() throws Exception {super.doStart();//context.addRoutes this.getRouteBuilders()postProcessContext(); // does add routes}protected  Map<String, CamelContext> getCamelContextMap() {return ['main': context]}/*** Add a component to the context* @param name* @param component*/public void addComponent(String name, Component component ) {this.context.addComponent(name, component);}public CamelContext getCamelContext() {return context;}protected ModelFileGenerator createModelFileGenerator() throws JAXBException {return null;}/*** Enables the hangup support. Gracefully stops by calling stop() on a* Hangup signal.*/public void enableHangupSupport() {HangupInterceptor interceptor = new HangupInterceptor(this);Runtime.getRuntime().addShutdownHook(interceptor);}}/*** A class for intercepting the hang up signal* and do a graceful shutdown of the Camel.* Taken from the camel-spring implementation of MainSupport.*/class HangupInterceptor extends Thread {MainSupport mainInstance;public HangupInterceptor(MainSupport main) {mainInstance = main;}@Overridepublic void run() {try {mainInstance.stop();} catch (Exception ex) {Logger.getLogger("Shutdown handler").warn(ex);}}}

The implementation only handles one simple camel context. It also adds support for adding components to the context.

I just upgraded to Munin 1.4.3 on Centos 5.4 (from 1.3.x I think) the other day and after some time noticed that my Graphs were not updating.

After a long bout of debugging and cursing Munin’s inability to provide any indications of what was wrong, I suddenly understood my error: I was deeplinking into pages that were not updated anymore because the Munin upgrade changed the way generated files are structured.

The following is the quick fix:

su -s /bin/bash muninfind . -name"*.html"-ls -exec rm -f {} \;/usr/bin/munin-cron

I.e delete all html files generated by munin and ask it to generate them over again _as the munin user_.

I sure hope this helps someone!

A few days ago someone asked on #symfony if it is possible to make symfony load fixtures from different places depending on the environment.

To do this, you have to define your own task. Here’s one I just created. As a bonus, this task also loads sql files from a separate sql directory.

Here’s the task:

<?php/*** This task does almost the same as doctrine:data-load, but it also* loads the sql from the files in the data/sqlextra directory And takes into account the current environment.*** Use instead of doctrine:data-load*/class nnv_load_dataTask extends sfBaseTask{protected function configure(){$this->addOptions(array(new sfCommandOption('application', null, sfCommandOption::PARAMETER_REQUIRED,'The application name'),new sfCommandOption('env', null, sfCommandOption::PARAMETER_REQUIRED,'The environment','dev'),new sfCommandOption('connection', null, sfCommandOption::PARAMETER_REQUIRED,'The connection name','doctrine'),// add your own options here));$this->namespace        ='data';$this->name             ='load-data';$this->briefDescription ='Load fixtures and sql based on environment';$this->detailedDescription =<<<EOFThis [nnv_load_data|INFO] task does almost the same as doctrine:data-load, but it alsoloads the sql from the files in the data/sqlextra/$env directory and loads fixtures from the data/fixtures/$env dir.Use instead of doctrine:data-loadCall it with:[php symfony load_extra_sql|INFO]EOF;}protected function execute($arguments = array(), $options = array()){// initialize the database connection$databaseManager = new sfDatabaseManager($this->configuration);$connection = $databaseManager->getDatabase($options['connection'] ? $options['connection'] : null)->getConnection();Doctrine::loadData(sfConfig::get('sf_root_dir').'/data/fixtures/'. $env);$env = $options['env'];$files = glob(sfConfig::get('sf_root_dir')."/data/sqlextra/$env/*.sql");foreach ($files as $file) {print"Loading". $file ."...";$sql = file_get_contents($file) ;try {$n = $connection->exec($sql);print"ok: $n\n";} catch (Exception $e) {print"Error loading statement: \n$stmt\n:". $e->getMessage();}}}}