Archive

Archive for the ‘Java’ Category

Setting up Hadoop HDFS in Pseudodistributed Mode

April 19th, 2016 No comments

Well, new to the big data world.

Following Appendix A in the book Hadoop: The Definitive Guide, 4th Ed, just get it to work. I’m running Ubuntu 14.04.

1. Download and unpack the hadoop package, and set environment variables in your ~/.bashrc.

export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
export HADOOP_HOME=~/hadoop-2.5.2
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

Verify with:

# hadoop version
Hadoop 2.5.2
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r cc72e9b000545b86b75a61f4835eb86d57bfafc0
Compiled by jenkins on 2014-11-14T23:45Z
Compiled with protoc 2.5.0
From source with checksum df7537a4faa4658983d397abf4514320
This command was run using /home/gonwan/hadoop-2.5.2/share/hadoop/common/hadoop-common-2.5.2.jar

The 2.5.2 distribution package is build in 64bit for *.so files, use the 2.4.1 package if you want 32bit ones.

2. Edit config files in $HADOOP_HOME/etc/hadoop:

<?xml version="1.0"?>
<!-- core-site.xml -->
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost/</value>
  </property>
</configuration>
<?xml version="1.0"?>
<!-- hdfs-site.xml -->
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

3. Config SSH:
Hadoop needs to start daemons on hosts of a cluster via SSH connection. A public key is generated to avoid password input.

# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Verify with:

# ssh localhost

4. Format HDFS filesystem:

# hdfs namenode -format

5. Start HDFS:

# start-dfs.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/gonwan/hadoop-2.5.2/logs/hadoop-gonwan-namenode-gonwan-mate17.out
localhost: starting datanode, logging to /home/gonwan/hadoop-2.5.2/logs/hadoop-gonwan-datanode-gonwan-mate17.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/gonwan/hadoop-2.5.2/logs/hadoop-gonwan-secondarynamenode-gonwan-mate17.out

Verify running with jps command:

# jps
2535 NameNode
2643 DataNode
2931 Jps
2828 SecondaryNameNode

6. Some tests:

# hadoop fs -ls /
# hadoop fs -mkdir /test
# hadoop fs -put ~/.bashrc /
# hadoop fs -ls /
Found 2 items
-rw-r--r--   1 gonwan supergroup        215 2016-04-19 16:07 /.bashrc
drwxr-xr-x   - gonwan supergroup          0 2016-04-19 16:06 /test

7. Stop HDFS:

# stop-dfs.sh

8. If there is an error like:

Error: JAVA_HOME is not set and could not be found.

Just edit $HADOOP_HOME/etc/hadoop/hadoop-env.sh and export JAVA_HOME explicitly again here. It does happen under Debian. Not knowing why the environment variable is not passed over SSH.

9. You can also set HADOOP_CONF_DIR to use a separate config directory for convenience. But make sure you have the whole directory copied from the Hadoop package. Otherwise, nasty errors may occur.

Categories: Java Tags: ,

OO Impelementation in Java

March 13th, 2011 No comments

Continue with last article, we will try to write an identical application to use OO features including: encapsulation, inheritance, polymorphism, properties, meta info and event-driven mechanism. Java supports the 3 basic features in language level. It uses interfaces to implements event-driven. To implements properties and meta info, we have to write our own code. We want to implements API like someObject.setProperty(prop-name, prop-value). I write my own NewObject class:

package my;

import java.lang.reflect.Field;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;

import my.annotation.ClassInfo;
import my.annotation.ClassInfoList;
import my.annotation.Property;
import my.annotation.PropertyAccess;

/**
 * We just set/get values as Object type. End users know the exact type of the
 * property, and they can do the conversion themselves.
 */
public class NewObject {

    private static String makeGetPropertyName(Field field) {
        String fieldName = field.getName();
        if (fieldName == null || fieldName.equals("")) {
            return null;
        }
        return "get" + Character.toUpperCase(fieldName.charAt(0)) + fieldName.substring(1);
    }

    private static String makeSetPropertyName(Field field) {
        String fieldName = field.getName();
        if (fieldName == null || fieldName.equals("")) {
            return null;
        }
        return "set" + Character.toUpperCase(fieldName.charAt(0)) + fieldName.substring(1);
    }

    /**
     * Get property
     * @param name
     * @return
     */
    public Object getProperty(String name) {
        Class klass = this.getClass();
        Field field = null;
        while (!klass.getName().equals(NewObject.class.getName())) {
            try {
                field = klass.getDeclaredField(name);
                break;  // found
            } catch (NoSuchFieldException e) {
                // noop
            }
            klass = klass.getSuperclass();
        }
        if (field != null) {
            Property property = field.getAnnotation(Property.class);
            if (property.value() == PropertyAccess.WRITE
                    || property.value() == PropertyAccess.READWRITE) {
                String methodName = makeGetPropertyName(field);
                try {
                    /**
                     * We can also get the value directly as below, but this
                     * bypass the getter function which is wrong. 
                     * <code>
                     * field.setAccessible(true);
                     * Object value = field.get(this);
                     * field.setAccessible(false);
                     * return value;
                     * </code>
                     */
                    Method getMethod = klass.getMethod(methodName);
                    return getMethod.invoke(this);
                } catch (IllegalAccessException e) {
                    // noop
                } catch (IllegalArgumentException e) {
                    // noop
                } catch (InvocationTargetException e) {
                    // noop
                } catch (NoSuchMethodException e) {
                    // noop
                } catch (SecurityException e) {
                    // noop
                }
            }
        }
        return null;
    }

    /**
     * Set property
     * @param name
     * @param value
     */
    public void setProperty(String name, Object value) {
        Class klass = this.getClass();
        Field field = null;
        while (!klass.getName().equals(NewObject.class.getName())) {
            try {
                field = klass.getDeclaredField(name);
                break;  // found
            } catch (NoSuchFieldException e) {
                // noop
            }
            klass = klass.getSuperclass();
        }
        if (field != null) {
            Property property = field.getAnnotation(Property.class);
            if (property.value() == PropertyAccess.WRITE
                    || property.value() == PropertyAccess.READWRITE) {
                String methodName = makeSetPropertyName(field);
                try {
                    Method setMethod = klass.getMethod(methodName, field.getType());
                    setMethod.invoke(this, value);
                } catch (IllegalAccessException e) {
                    // noop
                } catch (IllegalArgumentException e) {
                    // noop
                } catch (InvocationTargetException e) {
                    // noop
                } catch (NoSuchMethodException e) {
                    // noop
                } catch (SecurityException e) {
                    // noop
                }
            }
        }
    }

    /**
     * Dump class info by given class
     * @param klass
     */
    public static void dumpClassInfo(Class klass) {
        System.out.println(klass.getCanonicalName() + "(");
        ClassInfo[] klassInfos = klass.getAnnotation(ClassInfoList.class).value();
        for (int i = 0; i < klassInfos.length; i++) {
            System.out.println(klassInfos[i].name() + "=" + klassInfos[i].value());
        }
        System.out.println(")");
    }

    /**
     * Dump class info of current object
     */
    public void dumpClassInfo() {
        Class klass = this.getClass();
        System.out.println(klass.getCanonicalName() + "(");
        ClassInfo[] klassInfos = klass.getAnnotation(ClassInfoList.class).value();
        for (int i = 0; i < klassInfos.length; i++) {
            System.out.println(klassInfos[i].name() + "=" + klassInfos[i].value());
        }
        System.out.println(")");
    }

    /**
     * Get class info by given name
     * @param name
     * @return
     */
    public String getClassInfo(String name) {
        Class klass = this.getClass();
        ClassInfo[] klassInfos = klass.getAnnotation(ClassInfoList.class).value();
        for (int i = 0; i < klassInfos.length; i++) {
            if (klassInfos[i].name().equals(name)) {
                return klassInfos[i].value();
            }
        }
        return null;
    }

}

To use our setProperty()/getProperty() method, all classes should derive from the NewObject class. To be consistent with the JavaBean convention, we assume that the getter/setter function to be “get”/”set” + capitalize_first_letter_of(member-variable-name).

Property annotation and PropertyAccess enum are defined to indicate properties:

// PropertyAccess.java
package my.annotation;

public enum PropertyAccess {
    READ, WRITE, READWRITE
}
// Property.java
package my.annotation;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.FIELD)
public @interface Property {
    /* "value" seems to be a magic name */
    public PropertyAccess value();
}

ClassInfo and ClassInfoList annotation are defined to indicate class meta info:

// ClassInfo.java
package my.annotation;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
public @interface ClassInfo {
    public String name();
    public String value();
}
// ClassInfoList.java
package my.annotation;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
public @interface ClassInfoList {
    ClassInfo[] value();
}

Let’s see how to use them, our Base is defined as:

// Base.java
package fake;

import java.util.LinkedList;
import java.util.List;

import my.NewObject;
import my.annotation.ClassInfo;
import my.annotation.ClassInfoList;
import my.annotation.Property;
import my.annotation.PropertyAccess;

@ClassInfoList({ 
    @ClassInfo(name = "author", value = "gonwan"),
    @ClassInfo(name = "version", value = "1.0.0") 
})
public class Base extends NewObject {
    @Property(PropertyAccess.READWRITE)
    private int id;
    @Property(PropertyAccess.READWRITE)
    private String name;

    private List<IPrintInt> basePrintIntListeners;
    private List<IPrintString> basePrintStringListeners;

    public Base() {
        basePrintIntListeners = new LinkedList<IPrintInt>();
        basePrintStringListeners = new LinkedList<IPrintString>();
    }

    public int getId() {
        return id;
    }

    public void setId(int id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public void virtualDump() {
        System.out.printf("Base(virtual): id=%d, name=\"%s\"\n", getId(), getName());
    }

    final public void nonvirtualDump() {
        System.out.printf("Base(nonvirtual): id=%d, name=\"%s\"\n", getId(), getName());
    }

    public void addBasePrintIntListener(IPrintInt listener) {
        basePrintIntListeners.add(listener);
    }

    public void addBasePrintStringListener(IPrintString listener) {
        basePrintStringListeners.add(listener);
    }

    public void fireBasePrintIntEvent(int i) {
        for (IPrintInt listener : basePrintIntListeners) {
            listener.printInt(i);
        }
    }

    public void fireBasePrintStringEvent(String str) {
        for (IPrintString listener : basePrintStringListeners) {
            listener.PrintString(str);
        }
    }

}

Since our implementation of properties are simply methods, they can be inherited by subclasses. But the class meta info cannot be retrieved in subclasses. They just get their own.

I do not want to demo events/listeners code here, just find them in source code in my skydrive: http://cid-481cbe104492a3af.office.live.com/browse.aspx/share/dev/TestOO. In the TestJavaObject-{date}.zip file.

Categories: Java Tags:

Zip Compression Bug in JDK was Fixed

May 14th, 2009 No comments

The long-living zip compression file name encoding bug in JDK was finally fixed. Since the original ZipEntry class will encode/decode the file name in platform’s native encoding, one zip file created under one codepage cannot be decoded correctly under another codepage. The workaround is to use the ZipEntry class in the ant project. But in current JDK7 early access, the ZipEntry class also added the encoding support as the ant project.

This feature was found occasionally when I checked the JDK7 project’s changeset: http://download.java.net/jdk7/changes/jdk7-b57.html

Categories: Java Tags: