Installing and running PIG 0.11.1 with Hadoop 1.2.1

1. Download pig from pig.apache.org
2. copy it to the hadoop user /home/hduser/
3. extract , make sure the extracted directory has the same permission (hduser:hadoop), insert path to $PATH
4. Start the hadoop server (./start-all.sh), check with jps
5. Copy the data to HDFS,
>hadoop dfs -copyFromLocal /user/hduser/<file-name>
6. run script using pig –
>pig <script-name>
7. check if output is generated
>hadoop dfs -ls /user/hduser/<file-name>/ [Note that , the output directory is automatically created and has the same name as the input file]
8. check output file
> hadoop dfs -cat /user/hduser/<output-dir>/part-r-00000|head -5

How to Install Scalding 2.10.2 on Ubuntu 12.04 with Hadoop 1.2.1

1. Download Scalding from repo on GitHub:

$ git clone git://github.com/twitter/scalding.git

it will create a local folder called scalding

2. cd scalding
3. export SCALDING_HOME=`pwd` # /home/hduser/scalding (optional?)

4. Install [sbt 0.12.2](<a href="http://www.scala-sbt.org/release/docs/Getting-Started/Setup.html" rel="nofollow">http://www.scala-sbt.org/release/docs/Getting-Started/Setup.html</a>) (sorry, but the assembly plugin is sbt version dependent).
5“`sbt update“` (takes 2 minutes or more)
6. “`sbt test“

7. open project/Build.Scala – change

scalaVersion := “2.10.2″, from scalaVersion := “2.9.3″

8“`sbt assembly“` (needed to make the jar used by the scald.rb script)

9. rename target scala-core/target/scala-2.10 folder to scala-2.10.2

mv ../scalding-core/target/scala-2.10/ ../scalding-core/target/scala-2.10.2

10.open scald.rb and change

CONFIG_DEFAULT = begin
...
{
...
"localmem" =&gt; "1g"
...
}

In case your VM fails to start

11. Run tutorial , include –scalaversion

./scripts/scald.rb –local –scalaversion=2.10.2 tutorial/Tutorial1.scala
[/code[
General run command syntax:

./path/to/scald.rb --local --scalaversion=2.10.2 somefile.scala --input ~/input.txt --output ~/output.txt
Example:

./scalding/scripts/scald.rb --local --scalaversion=2.10.2 WordCountJob.scala --input ~/mary.txt --output ~/mary_wordcount.txt

Irrlicht – An Ideal Graphics Engine for Newbies


For sometime I have been studying this Graphics Engine called Irrlicht. I came to know about this one from the monthly game development talks at BDOSN.org, the lecture was given by Al-Mamoon Shohag, Technical Leader at S-Games Ltd (http://www.s-games.co.uk, Previously known as Deep Red Studio Ltd.). So thanks to him and BDOSN. I think this is a really good graphics engine, because even if you’re just new to graphics/game programming you’ll be able to pick up its features within shortest amount of time. And that is not because of its simplicity but also for the wide range of documentations and tutorials found over the net.

Irrlicht supports 3D rendering via OpenGL, DirectX 8 and 9, and internal software rasterizers. External renderers can be written and plugged via a simple interface, giving rise to a community-made SDL video driver. The engine comes with a library of standard material renderers, allowing fallback materials where user hardware is unable to handle advanced techniques. New materials can be added to the engine at run-time, allowing users to write their own where required. In addition to legacy fixed-function pipeline materials, programmable Pixel and Vertex Shaders (1.1 to 3.0), ARB Fragment and Vertex Programs, HLSL and GLSL materials are supported.

The engine supports most common 3D mesh and image formats by default, and more have been written as external plugins. Lights, cameras and 3D objects are managed as a tree of ‘Scene Nodes’, arbitrary groupable entities which are responsible for their own behaviour. Nodes can be managed by generic animators, by each other, or manually by the user. A large number of built-in node types exist and can be used together to make complex indoor and outdoor scenes, new nodes are trivial to make and can be added at runtime, many extra ones are provided by the community. Internal node types include a terrain renderer and sky domes/boxes for outdoor rendering, BSPs for indoor rendering, bone based animated meshes, stencil shadows, billboards and particle systems, water surfaces and primitives.

A skinnable 2D GUI is available, sporting many controls and the ability for users to plug in their own (or community made) custom controls at runtime, access to a full event system is provided for responding to GUI, mouse and keyboard events without having to rely on external libraries.

Filesystem access is abstracted allowing platform-independent file and folder access, and transparent access to files within Zip archives. Other IO features include an XML reader and writer, the ability to take screenshots, manipulate images and save them in several different file formats.

Rudimentary collision detection is also included, but for more serious physics uses, users are recommended to use a full featured Physics engine.

Links for starters:

1. http://www.irrlicht3d.org/wiki/
2. http://irrlicht.sourceforge.net/tutorials.html
3. http://irrlicht.sourceforge.net/phpBB2/viewtopic.php?p=136776 (Highly recommended, but there maybe some bugs in the code snippets. Read the whole topic posting to resolve those issues).
4. http://comsci.liu.edu/~murali/index.html (Some basic examples, very helpful to get u started)
5. http://www.codeguru.com/cpp/misc/samples/games/article.php/c10469__1/
(Review of Irrlicht engine with some cool sample codes.)

Here’s a list of some open source games using Irrlicht. If you’re intersted in game development using Irrlicht, maybe these examples might help.

1. Crimson Glory – http://www.crimsonengine.com/
2. Supremacy – http://supremacy.sourceforge.net/
3. First-King – http://first-king.sourceforge.net/
4. http://aresfps.sourceforge.net/