Securing your JBoss JMX Invoker Layer

If you use JBoss and have a nicely secured JMX Console and/or Web Console it’s a fairly safe bet that, like me, you haven’t secured the invoker layer; meaning any old monkey can most likely shutdown your container whenever they feel like it.

Recently I implemented an MBean in JBoss to use as a batch trigger from a ControlM implementation and was surprised (probably shouldn’t have been though) that all my carefully crafted security for the JMXConsole and Web Console was ignored with complete impunity by the tool (twiddle.sh) that I used to invoke my MBean. Since then I’ve been through a pile of pain trying to get an RMI call to a JBoss XMBean to require authentication and I thought I’d put some instructions in plain language on how to do it.

I do this for two reasons:

  1. because I bet a lot of developers miss this one; and
  2. because the documentation and other information I find online is limited and confusing.

For demonstration I’m going to use a standard JBoss MBean for setting system properties in a running application container.

A Simple Example of Setting a System Property in JBoss using Twiddle

Using the default JBoss version of twiddle.sh (in the bin directory beneath JBoss home) against the default JNP location of JBoss (localhost:1099) you can execute the following to set a system property in a running container.

# this assumes you're in the bin directory of JBoss Home.
./twiddle.sh -s localhost:1099 invoke "jboss:type=Service,name=SystemProperties" set myprop mypropvalue
'null'

To verify that you have been succesful (assuming you didn’t get an exception in the last operation) you can do the following:

# this assumes you're in the bin directory of JBoss Home.
./twiddle.sh -s localhost:1099 invoke "jboss:type=Service,name=SystemProperties" get myprop
mypropvalue

This example will work from anywhere on your network where you’re not prevented from reaching the JNP URL of the container (prevented by a firewall or IP filter for example) regardless of the JMX Console and Web Console security you’ve put in place (there is plenty of documentation around for securing the JMX Console and Web Console). This is because the JMX Console and Web Console are HTTP based and as such are secured in the normal way you would secure a website on JBoss (i.e. in web-inf.xml and jboss-web.xml) whereas the invoker layer is not HTTP based and as such must use an alternate method of security; the key file in this operation is the jmx-invoker-service.xml file in the JBoss deploy directory.

Securing the Invoker Layer

The invoker layer is the one you are calling through when you query or invoke on an MBean via RMI (i.e. with twiddle.sh – as above). This layer is not subject to the security constraints you will have placed on your HTTP based JMX Console or Web Consoles.

To make this layer secure the key file you’re interested in is the jmx-invoker-service.xml in the JBoss deploy directory; and the key operation configuration you will need to change is for ‘invoke’.

The default configuration of the invoke operation in this file is:

<server>

	<!-- excluded for brevity -->

	<mbean code="org.jboss.jmx.connector.invoker.InvokerAdaptorService" name="jboss.jmx:type=adaptor,name=Invoker" xmbean-dd="">
		<xmbean>
			<description>The JMX Detached Invoker Service</description>
			<class>org.jboss.jmx.connector.invoker.InvokerAdaptorService</class>

			<!-- excluded for brevity -->

			<operation>
				<description>The detached invoker entry point</description>
				<name>invoke</name>
				<parameter>
					<description>The method invocation context</description>
					<name>invocation</name>
					<type>org.jboss.invocation.Invocation</type>
				</parameter>
				<return-type>java.lang.Object</return-type>
				<descriptors>
					<interceptors>

						<!-- Uncomment to require authenticated users -->
						<!-- <interceptor code="org.jboss.jmx.connector.invoker.AuthenticationInterceptor" securityDomain="java:/jaas/jmx-console"/> -->

						<!-- Interceptor that deals with non-serializable results -->
						<interceptor code="org.jboss.jmx.connector.invoker.SerializableInterceptor" policyClass="StripModelMBeanInfoPolicy"/>

					</interceptors>
				</descriptors>
			</operation>
		</xmbean>
	</mbean>
</server>

So to switch on authentication we do what it says and ‘Uncomment to require authenticated users’:

<server>

	<!-- excluded for brevity -->

	<mbean code="org.jboss.jmx.connector.invoker.InvokerAdaptorService" name="jboss.jmx:type=adaptor,name=Invoker" xmbean-dd="">
		<xmbean>
			<description>The JMX Detached Invoker Service</description>
			<class>org.jboss.jmx.connector.invoker.InvokerAdaptorService</class>

			<!-- excluded for brevity -->

			<operation>
				<description>The detached invoker entry point</description>
				<name>invoke</name>
				<parameter>
					<description>The method invocation context</description>
					<name>invocation</name>
					<type>org.jboss.invocation.Invocation</type>
				</parameter>
				<return-type>java.lang.Object</return-type>
				<descriptors>
					<interceptors>

						<!-- Uncomment to require authenticated users -->
						<interceptor code="org.jboss.jmx.connector.invoker.AuthenticationInterceptor" securityDomain="java:/jaas/jmx-console"/>

						<!-- Interceptor that deals with non-serializable results -->
						<interceptor code="org.jboss.jmx.connector.invoker.SerializableInterceptor" policyClass="StripModelMBeanInfoPolicy"/>

					</interceptors>
				</descriptors>
			</operation>
		</xmbean>
	</mbean>
</server>

If you haven’t changed the default security realm for your JMX Console (i.e. java:/jaas/jmx-console) you will now have an invoker layer secured with the same credentials as for your JMX Console. To change this add a new security realm to your global login-config.xml in the conf directory of your container and match the name you give it in the securityDomain attribute of the Authentication Interceptor.

I’ve not yet delved too deeply into setting a specific set of roles, at this point I set my invoker user to JBossAdmin which means that user can do pretty much anything exposed to JMX. That’s ok for my purposes tho (feel free to write a response with the details of setting roles for particular JMX functions :-)).

Invoking on a Secure Invoker Layer

Ok so now that it’s secure how do you invoke an operation on it?

With the default JBoss twiddle.sh utility there are arguments -u (or –user=) for user and -p (–password=) for password.

# this assumes you're in the bin directory of JBoss Home.
./twiddle.sh -s localhost:1099 --user=myuser --password=mypassword invoke "jboss:type=Service,name=SystemProperties" get myprop
mypropvalue

These arguments work fine except your password is now in clear text and even worse is visible in the process list while it’s executing – in clear text with ‘password=’ conveniently placed for extraction by a simple script!! This seems a bit of an oversight in the tool to me.

To get around this issue in my environment I took the source and modified the main class of twiddle.jar to accept a password from Standard In (patch is below – no promises or guarantees though) which prevents the password showing in your password list and allows you to use standard encryption utilities to decrypt and pipe it into the process without ever making it visible clear text.

You would now invoke as follows:

# this assumes you're in the bin directory of JBoss Home.
mypassword | ./twiddle.sh -s localhost:1099 --user=myuser invoke "jboss:type=Service,name=SystemProperties" get myprop
mypropvalue

or better; from an encrypted password file (or better yet a repository) such as follows:

# this assumes you're in the bin directory of JBoss Home and have previously encrypted your password and encryption key into ~/.<username>.key and ~/.<username>.psw.
KEY=`cat ~/.<execution username>.key`
PWD=`cat ~/.<execution username>.psw | crypt $KEY`

PWD | ./twiddle.sh -s localhost:1099 --user=myuser invoke "jboss:type=Service,name=SystemProperties" get myprop
mypropvalue

to encrypt your password to be used as above you might do:

echo "<password>" | crypt > ~/.<execution username>.psw

which will request an encryption key which you would save as follows (for this example anyway):

cat "<encryption key>" > ~/.<execution username>.key

These files would, of course, be accessable only from your execution user.

Securing the JMX Console

For reference the key files you’re interested in here are:

  • conf/login-config.xml
  • deploy/jmx-console.war/META-INF/web.xml
  • deploy/jmx-console.war/META-INF/jboss-web.xml

Securing the JMX Web Console

For reference the key files you’re interested in here are:

  • conf/login-config.xml
  • deploy/management/web-console.war/META-INF/web.xml
  • deploy/management/web-console.war/META-INF/jboss-web.xml

Stack

These instructions will apply broadly but for reference purposes the stack I have is:

  • JBoss 4.2.3.GA
  • Java jdk1.6.0_13
  • Windows XP or Solaris 10

References

http://www.jboss.org/community/wiki/Twiddle

http://www.jboss.org/community/wiki/jbossserver-aquicktour

https://jira.jboss.org/jira/secure/attachment/12313982/jboss-securejmx.pdf (PDF Document)

Regards,
Jon

🙂

Patch For Twiddle to Take Password from StdIn (no promises or guarantees)


Index: src/main/org/jboss/console/twiddle/Twiddle.java
===================================================================
--- src/main/org/jboss/console/twiddle/Twiddle.java    (revision 94201)
+++ src/main/org/jboss/console/twiddle/Twiddle.java    (working copy)
@@ -24,8 +24,10 @@
 import gnu.getopt.Getopt;
 import gnu.getopt.LongOpt;

+import java.io.BufferedReader;
 import java.io.File;
 import java.io.InputStream;
+import java.io.InputStreamReader;
 import java.io.PrintWriter;
 import java.net.MalformedURLException;
 import java.net.URL;
@@ -41,7 +43,6 @@
 import javax.naming.Context;
 import javax.naming.InitialContext;
 import javax.naming.NamingException;
-
 import org.jboss.console.twiddle.command.Command;
 import org.jboss.console.twiddle.command.CommandContext;
 import org.jboss.console.twiddle.command.CommandException;
@@ -148,7 +149,7 @@
 }
 };
 }
-
+
 public Command createCommand(final String name)
 throws NoSuchCommandException, Exception
 {
@@ -383,7 +384,7 @@

 out.println("A JMX client to 'twiddle' with a remote JBoss server.");
 out.println();
-      out.println("usage: " + PROGRAM_NAME + " [options] <command> [command_arguments]");
+      out.println("usage: [echo <password> | ] " + PROGRAM_NAME + " [options] <command> [command_arguments]");
 out.println();
 out.println("options:");
 out.println("    -h, --help                Show this help message");
@@ -397,6 +398,10 @@
 out.println("    -u, --user=<name>         Specify the username for authentication");
 out.println("    -p, --password=<name>     Specify the password for authentication");
 out.println("    -q, --quiet               Be somewhat more quiet");
+      out.println();
+      out.println("A password should be passed in by echoing it and piping it to the command. If you");
+      out.println("use the -p (--password) option your password may be visible in clear text in a ");
+      out.println("process listing such as `ps -ef`.");
 out.flush();
 }

@@ -421,6 +426,28 @@
 Getopt getopt = new Getopt(PROGRAM_NAME, args, sopts, lopts);
 int code;

+        /* Get standard in if it's there - assume it's a password. This is to allow a password to be passed and
+         * prevent it showing in a process listing (e.g. ps -ef in Unix). The -p argument will be ignored if
+         * the password is passed through Standard In.
+         */
+        boolean passwordRetrievedFromStdIn = false;
+        if (System.in.available() > 0) {
+            InputStreamReader inp = new InputStreamReader(System.in);
+            BufferedReader br = new BufferedReader(inp);
+            String stdin = br.readLine();
+
+            if (stdin != null &amp;&amp; stdin.trim().length() > 0) {
+                String password = stdin.trim();
+                SecurityAssociation.setCredential(password);
+
+                passwordRetrievedFromStdIn = true;
+
+                if (log.isDebugEnabled()) {
+                    log.debug("Password retrieved from standard in. Ignoring -p argument.");
+                }
+            }
+        }
+
 PROCESS_ARGUMENTS:

 while ((code = getopt.getopt()) != -1)
@@ -531,8 +558,13 @@
 SecurityAssociation.setPrincipal(new SimplePrincipal(username));
 break;
 case 'p':
-                 String password = getopt.getOptarg();
-                 SecurityAssociation.setCredential(password);
+                  if (!passwordRetrievedFromStdIn) {
+                     String password = getopt.getOptarg();
+                     SecurityAssociation.setCredential(password);
+
+                     log.warn("Password retrieved from -p argument. Your password may be visible in cleartext in a process listing during execution. " +
+                             "Consider using Standard In to enter the password instead (i.e. echo \"password\" | twiddle ...)");
+                  }
 break;

 // Enable quiet operations

 light="true"
Advertisements

Thought for Friday: What are we going to do about the crap code ?

Yesterday, my manager asked me “What are we going to do about the crap code?”

It’s a hard question to answer because before we can answer what to do about it; we must understand what crap code is and what causes it.

What is crap code? This is still a hard question, but I think we can at least attempt to answer it by defining what it isn’t:

  • Crap code isn’t maintainable
  • Crap code isn’t easily testable
  • Crap code isn’t as efficient as it could be
  • Crap code isn’t understandable without lots of comments

What causes crap code? In my experience it is some combination of:

  • Ignorance
  • Fear
  • Lack of experience, pride, oversight, accountability, time and analysis/understanding of the domain
  • Resume-building perhaps?

So what is the solution? There is no silver bullet, it requires vigilance and potentially each of these causes needs to be tackled individually.

Ignorance: One way of tackling this is encouraging learning, peer review and discussion in the organization. Time needs to be allocated to ensure that project work doesn’t stop these activities from flourishing.

Fear: One of the major differences between a good developer and an average one is the fear factor; good developers don’t have the same fear of changing something, or fixing something since they are confident they are not going to break something without knowing about it. Because of this, fear has the effect of destroying reuse – copy and paste code thrives in this environment. Besides learning how to study code, unit testing and other forms of automated testing are good for addressing this.

Experience: You need to install a mentoring system to ensure that the less experienced team members have someone to go to for advice. Team members need to be exposed to different types of development work. E.g. database, business-logic, front-end / web etc. They’ll never get enough experience doing the same thing day in, day out. Ensure every commit to the code base is emailed to the rest of the team, experienced developers should be reviewing the code and suggesting changes if required.

Pride: This is a hard one, and probably comes from the side effects of the other causes. One way is to allow team members to look after a particular module, service / sub-system.

Oversight / Accountability: Oversight is simple – review code and don’t compromise on quality – you’ll end up regretting it at some point. Generally speaking accountability decreases the more segregated the teams are.  I.e. if you have a team of testers that are separate to your developers, the less responsibility the developers will feel to getting something right – you’ll get a “throw it over the fence” mentality. Create a team with all disciplines (developers, testers, dbas, infrastructure etc.,,).

Time: This is about planning properly and ensuring that developers are aware of all the steps to produce bit of functionality. It’s also about being realistic when it comes to providing your estimates. Make sure you include time for producing your unit and integration tests, and don’t underestimate just so you get the job. Stress to developers that the job isn’t done until it is tested and works.

Resume-building: This is subjective but I’ve been on projects where the tech lead has read that this new piece of technology is the ducks and they wants to get in on it just so they can put it down on their resume. They manage to get all the newest and greatest stuff in there but at what price? The code under the covers becomes a maintenance nightmare because it is far to complicated for what it needs to do. Ensure that the technology you use is decided up front. Don’t deviate unless you get further sign-off from the stakeholders. Create a list of “guiding principles” (N.B. not coding standards) to decide how you’re going to go about writing the code – and enforce code reviews on these.

Understanding the domain:  When you start a new project it is really important, that at every step in the process you constantly to try to refactor based on your current understanding of the code. Data models are really hard to change once a system goes into production, so you’ve got to make an effort to get it right. If some aspect of the design doesn’t feel right, then it probably isn’t it. Don’t be in denial – fix it.  I’ve seen so much bad code and bugs caused because the new requirements or understanding about a system are applied to a broken domain.

The bottom line is, does crap code really matter? Who cares if it works, right? I think it does. And crap code is directly proportional to defects in my experience.  Defects will cost you money fixing them and cleaning up the consequences of them.

Fixing the crap code is no doubt a hard problem. The solutions involve constant vigilance and culture change – but it is worth it in the end.

EJB 3.0 Injecting The Correct Implementation

When using stateless and stateful session beans I often find that I’m creating one interface per one implementation, which will be enhanced with the release of the 3.1 spec and implicit interfaces. Anyway on one of the occasions I had actually had multiple implementations of an interface that I wanted to inject, this made me stop and think how do I that!! It turns out it was pretty simple with a few annotations.

On your class that is implementing the interface you need to use the name attribute: –


@Stateful(name="xxx.AssetTypeOverrideService")
public class AssetTypeOverrideService implements OverrideService{

@Stateful(name="xxxRuleTypeOverrideService")
public class RuleTypeOverrideService implements OverrideService {

Then on the bean you want inject these into you use:


@EJB(beanName="xxx.AssetTypeOverrideService")
private OverrideService assetOverrideService;
	
@EJB(beanName="xxx.RuleTypeOverrideService")
private OverrideService ruleTypeOverrideService;

The name can be anything you want I just think it’s logical to keep it the same as the class and the beanName obviously is the same value of the bean you want to inject that has the correct implementation.

It seems easy now but it did make me stop and think.

We only hire the best

I was recently having dinner with an acquaintance who works for a massive multinational IT company. He was telling me that at a recent company conference that, despite the economy, the CEO said the company was doing well because “we only hire the best”.

I almost choked on my steak, trying to contain myself. The sad thing was that he actually believed it. How come all companies that I’ve worked at only hires the best ? Surely all of them can’t have the best ?

Why do companies maintain this illusion ? Is it some sort of psychological trick so you don’t quit and join a company full of bozos ? Or do people at the top truly believe it, in some sort of paradoxical delusion ?

I don’t know, but it’s odd isn’t it ?

EJB 3.0 Interceptors

I wanted to carry out some processing when an entity was committed to the db, I could of used the lifecycle annotation @PostPersist in the bean itself but that that would go against leaky abstraction (see post Non-anaemic models). Anywho it was pretty simple to get this working all that I had to use were a couple of annotations.

The first @EntityListerners which accepts an array of classes that will become your event handlers:

@EntityListeners(ErrorListener.class)
public class SystemError implements Serializable {

The next is to annotate the methods in your listener class that will handle the various events. The annotations that can be used are:

    @PrePersist Executed before the entity manager persist operation is actually executed or cascaded. This call is synchronous with the persist operation.
    @PreRemove Executed before the entity manager remove operation is actually executed or cascaded. This call is synchronous with the remove operation.
    @PostPersist Executed after the entity manager persist operation is actually executed or cascaded. This call is invoked after the database INSERT is executed.
    @PostRemove Executed after the entity manager remove operation is actually executed or cascaded. This call is synchronous with the remove operation.
    @PreUpdate Executed before the database UPDATE operation.
    @PostUpdate Executed after the database UPDATE operation.
    @PostLoad Eexecuted after an entity has been loaded into the current persistence context or an entity has been refreshed.

Note that if you are using EJB dependency injection it won’t work in your listener as it’s not managed by your container buy you can use jndi: –

Context ctx = new InitialContext();
XXXService service = (XXXService) ctx.lookup("EARNAME/XXXServiceBean/local");

Routing around damaged processes

It is often said that “The Internet interprets censorship as damage and routes around it”. I don’t know how true this statement is, but I have seen this affect applied to organizational processes.

When you work at a sufficiently large organization, you’ll start to employ people who’s sole responsibility is creating process. The problem with these roles is that, if no new process is created, then those people are not seen to be doing their job.

Once they get started, there’ll be a process for each aspect of developing software, production releases, production data changes, process for creating a process, etc. etc. ad nauseam. (I’m not saying that there is anything wrong with processes, but more that it unsustainable to continue to create them indefinitely).

This usually has either two outcomes:

  1. good people who just want to do their jobs get fed up with the red tape and leave or;
  2. people “route around the damage“d process.

I see #2 happen all the time. The problem is unnecessary risks are taken because it has just gotten too hard to do anything.

How do you solve this ? Ideally, the people that author the process should also have to follow it themselves. At the very least, you should involve the people that have to follow the process in its creation.

The most important thing is, you have to make following the process easy and fast. If you slow people down, they will just route around the damage (you!).

Joda with JPA

I have to work with a few legacy databases that are used by several different apps (apparently the database is a perfect integration point), and I want to use the Joda api in my persistence. In this example the date is actually stored as a string, genius! I’m going to use LocalDate as the string doesn’t store the time element. If you are lucky enough and the database is actually storing date time check this post out Joda With DateTime

To start with a create a simple helper class that converts String to LocalDate and LocatDate to String

public static LocalDate createLocalDateForString(String dateString){
  DateTimeFormatter fmt = DateTimeFormat.forPattern("yyyyMMdd");
  return StringUtils.isEmpty(value) ? null :  fmt.parseDateTime(dateString).toLocalDate();	
}
    
    public static String createStringForLocalDate(LocalDate date){
    	return date.toString("yyyyMMdd"); 
    }

Next is to create your own user type. This is done by implementing org.hibernate.usertype.UserType. The main methods I’m implement are

public Object nullSafeGet(ResultSet rs, String[] names, Object owner)
			throws HibernateException, SQLException {
		LocalDate result = null;
		String dateAsString = rs.getString(names[0]);
		if(!rs.wasNull()){
			result = StringUtils.isNotEmpty(dateAsString) ? DateHelper.createLocalDateForString(dateAsString) : null;
		}
		return result;
	}

public void nullSafeSet(PreparedStatement statement, Object value, int index)
			throws HibernateException, SQLException {
		if(value == null){
			statement.setNull(index, SQL_TYPES[0]);
		}else{
			String dateAsString = DateHelper.createStringForLocalDate((LocalDate)value);
			statement.setString(index, dateAsString);
		}
}

All that is left is to add the annotation to each of your entity properties that you want to use as LocalDate.

@org.hibernate.annotations.Type(type = "xxx.LocalDateUserType")