Scrum and Java projects. Why they tend to fail?

October 11, 2016 agile development, java, Project management, scrum No comments , ,

1475832328_featured.jpeg

As consultant I saw a lot of client passionate about scrum:

‚Scrum works very well!‘ ‚Scrum sucks!‘ ‚We partially (?!) adopt Scrum‘ and so on.

Many people think that scrum is a methodology but in reality is only a framework.
Spring is a framework too. It sucks? It’s productive? Well it depends how you use it, how you know it. The same is true for every framework.

When the motivation goes away Scrum is counterproductive

I have seen teams that were simply not motivated to use it or simply bored after few weeks. The 15 (or more) minutes of daily meeting was a simple routine that required a simple answer for the greatest happiness of all the participants :

‚I worked on that feature. No blocking issues.‘

During those meeting I saw participants continuing to work at their laptop, eating, watching TV series (!), chatting with the neighbour, reading the news on the smartphone … but everybody was certainly multitasking and listening with attention the other talking.

The notion of (almost) done

What about the notion of done? ‘The task is done!‘ (Developer 1) ‚Cool! Update excel, if we have time somebody will do a peer review‘ (Developer 2). Result: the task was more or less ‚done’ and the peer review will be realized by the client on their validation system or, probably, on their production environment.

The issues in a Java project?

The more frequent issues in my experience are:

Communication between developers

most of the wonderful talent in IT are not the best in communication, scrum try to help but the human nature is hard to change.

Specifications and users

they change all the time, they are incomplete most of the time, they are often impossible to implement. Agile methodologies try to help. The notion of product owner helps a lot but the artifacts will continue to change. The users are becoming more and more client oriented consultants and are losing their specific business skills. The developer need a lot more business information. Every exception has to be managed, in development we often spend more time implementing solutions to problems that the user will never see. Invite a user in a technical meeting … he will get a new perspective on his own activity. The analysts? If you are lucky they were skilled developers, otherwise … good luck!

Time and budget

a sales guy sold a project based on an estimation of the total cost of the project based on a draft of business concept. A developer (that is not currently involved in the project) initially estimated the effort at 100 (‚it’s easy‘), the project manager added a risk margin of 50% (‚150 should be ok‘).
As we know the initial estimation is in 90% of the projects completely wrong. This issue is not solved by the common project frameworks that assume endless time or feature stop at the end of the time. The pure agile is good for internal and continuous development. When you face clients that pay for your solution endless budget and feature stop are simply not realistic.

Technical decisions

today there are hundreds of frameworks, every year a new trend impact the new development projects for few years. Those frameworks survive only a couple of years before to ‚disappear‘ but their cost is immense for many years : Flex, Grails, GWT, ExtJS, JSF, AngularJS (1.x) are a few example. The result is that the projects today are a toxic mix of old and new technologies glued together.

Technical skills

many developers know the syntax (Java, .Net, JS) and they are correctly qualified as ‚developers‘. The problem is that to build quality the syntax is not enough. You need to know the grammar and the rhetoric too. Is this enough? Not yet, ask 5 author to write a chapter of the same book. The result is inconsistency, repetition and so on. The architecture and the lead are very important in this context.

All developers equals? The team is not the sum

some (agile) theories say that the developers will find the best solution themselves in an agile context. Is it true? Not according to my experience. The alpha guys will try to impose their view, arguing if necessary. Without a lead everybody will fill his tasks without caring how the others are working. A structure in the code has to be implemented and somebody has to take the lead on this. The structure can be challenged to improve the quality and productivity. To improve the maintanability everybody in the team has to be able to read the code of the others.

Individualities

often the management think that all the developers are the same, same skills, same productivity. ‚We are late? We add 5 new developer for 2 weeks and the project is finished‘. The new Fordism applied to the development. In reality the situation is more similar to a football (or soccer or any other sport) team. Some people are excellent defenders but unskilled strikers. Soft skills, technical skills, personal constraints are important factors that can define the success or the failure of a project. If you have Messi and Ronaldo in your team you are lucky. If Messi will play as goalkeeper and Ronaldo as central defender … well … disaster announced.

Solutions?

‘Ok, thanks! We already know it. Do you have any practical solution?‘

Well scrum is a generic framework that can be used in a Java project as well in a house building project or prepare a new recipe.
A complete solution require time to be elaborated. There are some techniques that I can suggest to improve the work for/in a team. My question is the following:

To increase the quality of our Java (or language X) project why we don’t use a Java-oriented (or language X) dedicated framework?

You can leave any comment about your direct experience.

Spring Boot, AngularJS 2, TypeScript: Hello World Tutorial

February 23, 2016 Angular 2, AngularJS, java, Java EE, REST, Spring, Spring Boot, TypeScript 26 comments , , ,

GitHub: https://github.com/marco76/SpringAngular2TypeScript

[update Sept. 19, 2016: the new blog ngjava.com will contain further information about the integration of java / spring with Angular 2, you can follow the updates using twitter : ]

[update Sept. 20, 2016: the first docker version is ready, still a prototype but the source is dowloaded, compiled and started correctly. More details here: ngjava.com. The dockerfile is here: Dockerfile]

[update Sept. 21, 2016: the example has been updated to the final version of Angular 2!!! A lot of changes in particular in the services (RC5 changed the Http injection). I will continue to update the next days!]

[Post not completed yet … it will be updated on ngjava.com]
I created a small project (the sources are not cleaned yet) that integrates:

  • Spring Boot
  • AngularJS 2 (Beta)
  • TypeScript

The goal of this example are the following:

  • Evaluate the maturity of AngularJS 2 that is beta
  • Evaluate the advantages of TypeScript over JavaScript
  • Evaluate the complexity of integration of a Java back-end with a Javascript front-end compared to a full Java solution (JavaEE + JSF)
  • Integration of Maven with Grunt
  • Integration of the security for REST

At the moment you can find the sources, I will explain soon how to build it.

The result is like this:

davis

Java EE (or Spring) + Angular = New blog?

August 24, 2016 Angular 2, java, Java 6, Java EE 7 comments

Dear All 🙂

  • I see that most of the views of my blog in the last months (btw thanks for the visit!) are connected to the articles about Angular 2 and Spring.
  • I see that on the market Java developers are required to know more and more Angular for the new project, that JSF and Struts are replaced by Angular.
  • I know that is not easy to match together the 2 different approaches and technologies.

I was thinking about the opportunity to create a new website or blog with the goal to collect the best practices to develop with Java and Angular together from a Java developer perspective. What do you think?

I would limit it to Angular 2 and not AngularJS (1.x). The time is limited.

Any comment or idea is welcome!

New Java project : checklist

May 3, 2016 java, Java EE, Project management, Uncategorized No comments

When we have to start a new project in Java or we have to join an existing project there are many points to be clarified and choices to do. You can find an updated list here: https://github.com/marco76/java-project-checklist (you can add your suggestions)

Example:

IDE – choice

  • Eclipse – Intellij – Netbeans
  • Free choice?

IDE – configuration

  • Is it possible to use custom plugins?
  • Is it possible to use custom configuration?
  • Are there existing profiles to be imported?

Java SE version

  • Which version?

Java enterprise framework

  • Spring – JavaEE – Other

JavaEE – injection preference

  • CDI – EJB
  • No preferences (mixing allowed)

[… continue on GitHub …]

Java Threads meaning

April 21, 2016 java No comments ,

You started your ‘Hello World’ Java application and Visual VM … what are all those threads???

  • main: the application you started with the main method
  • Signal Dispatcher: it handles the signals sent from the OS to the JVM
  • Attach Listener: it allows external process to start a thread in the JVM and send back information to the external process
  • Finalizer: pull object from the finalization queue and call the finalization method (it is created by Finaliser.java)
  • Reference Handler: it manages the reference queue (v. Reference.java)
  • RMI TCP Accept-0, RMI TCP Connection …, RMI Scheduler, RMI Reaper: it handles the access to remote objects
  • JMX server connection: it allows the connections of JMX modules. It is created by com.sun.jmx.remote.internal.ServerCommunicationAdmin
  • DestroyJavaVM: the application is shutting down, wait for all non-daemon threads to end, then destroy the VM. It’s called in the java.c file
  • GC Daemon:Garbage collection thread

Change the JRE / JDK in Eclipse on Mac OS X and access the sources

April 20, 2016 Eclipse, java, tutorial No comments , , , ,

Sometimes it happen that we have to change or add the JDK/JRE version used by eclipse and/or we want to access the correct source code of the JDK.

Here the step by step procedure under OS X:
1. Open File -> Preferences
2016-04-20_13-53-18
2. Search for the Installed JRE (currently the 1.8.0_73) and click ‘Add…’

2016-04-20_11-57-153. Select the JRE Home

If you installed the SDK using the installer it should be in:

/Library/Java/JavaVirtualMachines/jdk1.[version].jdk/Contents/Home/2016-04-20_11-58-59

4. To access the sources select rt in the System Libraries and click Source Attachment …2016-04-20_11-58-59_2

5. Choose the src.zip file from the directory containing the JDK

2016-04-20_11-59-27

6. Confirm and set the new JRE as the default one clicking on the checkbox near the name 2016-04-20_12-00-23_2

 

The change is effective only for the new projects, if you want to change the JDK/JRE for an existing project:

7. Click with the right button on the project and choose ‘Properties’. Search for ‘Java Build Path’ and select the JRE under Libraries. Click te ‘Edit’ button.
2016-04-20_12-01-12

8. Choose the new JRE

2016-04-20_12-01-33

9. (Optional) Restart Eclipse

Without a restarting if you try to call the JRE sources Eclipse will throw an error

2016-04-20_12-02-29

 

How to validate a Date in DB2

April 20, 2016 SQL Server No comments ,

This function verify if a date is valid in DB2 according to a custom date format.

-- this function return 1 if the varchar date and 0 if the date is not valid
-- the format of the date is passed in the second parameter (ex. mm/dd/yyyy)

CREATE OR REPLACE FUNCTION DB.FUNC_IS_VALID_DATE(dateToValidate VARCHAR(20), dateFormat VARCHAR(20)) RETURNS INTEGER

  BEGIN
     DECLARE functionResult INT;
     DECLARE resultingDate DATE;
     DECLARE SQLCODE INTEGER;

     DECLARE CONTINUE HANDLER FOR SQLEXCEPTION BEGIN
         RETURN 0; -- the data is not valid
     END; -- END DECLARE

     SELECT to_date(dateToValidate, dateFormat) INTO resultingDate FROM SYSIBM.SYSDUMMY1;
     RETURN 1; -- the data is valid
  END;

Java Server Sent Event – Automatically update web pages

April 17, 2016 Glassfish, java, Java EE, JSON, REST, tutorial, Web Services No comments

Code source here: https://github.com/marco76/javaSSE
How a client (ex. web browser) can get updates from a server? Here some options:

1. Polling

The client regularly request to the server new data using a simple request/response via HTTP.

sse_1_2016-04-17_11-15-19

2. Websocket

Java EE 7 and Spring 4 implemented the WebSocket protocol that allows a bi-directional communication between a client and a server using TCP. The HTTP is used only for the handshake. WebSockets are the appropriate solution for application that need frequent exchange of small chunks of data at high speed (ex. trading, videogames, …).

sse_2_2016-04-17_11-15-29

3. Server-side events

HTML5 allows an unidirectional communication similar to the publish/subscribe model in JMS. The protocol used is HTTP and is described in the W3C documentation (note than IE is not compatible).

sse_3_2016-04-17_11-15-43

The events can be broadcasted multiple clients:

sse_4_2016-04-17_14-09-19

This solution is appropriate for application that require to be notified about new events (newsfeed, twitter-like, stock market etc.)
In Java the support for SSE is not yet a standard (it should be in Java EE 8). You can implement it using a Servlet or use some of the libraries that already support it : Jersey and Spring.

The following example uses Jersey in a Java EE environment (Weblogic).

Example

In this example we simulate the ‘live’ visualisation of a simple running competition. The result time (and the time of appearance) is randomly defined.

The class that produces the result is a standard REST java class. The resource must produce a SERVER_SENT_EVENTS type. The method return an EventOutput class. This object maintain the connection with the client and send the results with the write() method

@Path("competition")
public class SseEventOutput {

	static Logger logger = Logger.getLogger("HelloSse.class");

	// EventOutput is created when the client open this resource
	// the method return the object EventOutput to the client (http)
	// until when the EventOutput is not closed it can send data to the client using write()
	
	EventOutput eventOutput = new EventOutput();
	
	@Path("results")
	@GET
	@Produces(SseFeature.SERVER_SENT_EVENTS)
	public EventOutput getServerSentEvents() {

In the next fragment of code we show when the new data is sent to the client browser using the write method:

// prepare the OutboundEvent to send to the client browser
OutboundEvent event = buildOutboundEvent(runner);
						
logger.log(Level.INFO, "writing the result for the participant in position: " + position);

// the event is sent to the client
// the EventOutput is already returned to the client but until when is not closed it can send messages to the client
eventOutput.write(event);
												// waiting for the next runner
position++;

The data to send is created and stored in the OutboundEvent object. In the object is defined in which format the data will be sent (JSON).

private OutboundEvent buildOutboundEvent(final Runner runner){
    OutboundEvent.Builder eventBuilder = new OutboundEvent.Builder();
    eventBuilder.name(LocalTime.now() + " - runner at the finish ... ");
		
    // the runner object will be converted to JSON
    eventBuilder.data(Runner.class, runner);
    eventBuilder.mediaType(MediaType.APPLICATION_JSON_TYPE);
	    
return eventBuilder.build();
}

Here the result of the execution:
2016-04-17_16-07-34_chrome
and the log:
2016-04-17_16-07-58_log
While the connection was open Chrome showed the ‘working in progress’ icon:
2016-04-17_16-06-05_chrome

Broadcast

In the class SSEBroadcast you can find an example of how the broadcast implementation works.

In the image you can see the result of sending 3 messages using post (green terminal) to the resource /api/hello_sse_broadcast/send.
The black terminal and chrome opened the page /api/hello_sse_broadcast/listen and they received the messages (chrome connected later and received only 2 messages).

2016-04-17_16-25-22_clients

The implementation of the broadcast require a Singleton:

@Path("hello_sse_broadcast")
@Singleton // singleton, one producer and multiple listener
public class SseBroadcast {	
	
	SseBroadcaster broadcaster = new SseBroadcaster();

We implemented one resource that return an EventOutput object. This object allows the communication between the server and the client. This resource is called by the client that subscribe the notifications/events :

	
	/**
	 * When a client connect to this resource it opens a communication channel.
	 * @return
	 */
	@Path("listen")
	@GET
	@Produces(SseFeature.SERVER_SENT_EVENTS)
	public EventOutput listenData() {
		 final EventOutput eventOutput = new EventOutput();
	        this.broadcaster.add(eventOutput);
	       
	        return eventOutput;
	}

Another resource receive the POST messages and create an event to send to the subscribers:

	/**
	 * For each call to this method some data is broadcasted to the listeners
	 * @return
	 */
	
	@POST
	@Path("send")
	public String sendData(@FormParam(value = "text") String text) {
		OutboundEvent.Builder builder = new OutboundEvent.Builder();
		builder.comment("optional comment: " + text);
		builder.data(Calendar.getInstance().getTime().toString());
		builder.mediaType(MediaType.APPLICATION_JSON_TYPE);
		
		broadcaster.broadcast(builder.build());
		
		return "date sent";

	}
}

Interesting references:
http://streamdata.io/blog/push-sse-vs-websockets/

OpenShift + Keycloak + AngularJS + JavaEE + MongoDB + CSS = MyCV

March 14, 2016 AngularJS, EJB, Java EE, JSON, mongodb, REST, tutorial, Uncategorized, Web Services No comments , , , , ,

—— Update Jul 25, 2016 ——
OpenShift is a great enterprise service. Unfortunately the resources needed by MongoDB and WildFly require some medium gears for an expense of $30 per month and the expense is not justified for a simple example website.
For this reason I rented a VPS, I installed everything from scrath and I moved to the  application to the new server. I will explain how to configure the server in a new post.
I used OVH as server provider but the procedure is the same with DigitalOcean. The goal was to spend less than $5/month and learn how to do it.
Preview: CentOS, WildFly, Java 8, MongoDB, Apache, Multiple websites.
—— END ——

Website: http://mycv.host
GitHub: https://github.com/marco76/myCv

I’ve a bit of time in this period and I’m updating (or, better, upgrading) my knowledge with the new trends on the market (recently I had to work with jsp and db2 ;))

I published a website that uses the following technologies:stack

  • OpenShift as PaaS (it manages all this architecture for … free …)
  • WildFly 10 + RESTful (RESTeasy)
  • MongoDB for the data
  • Keycloak for the Oauth2 security (google, github, facebook access)
  • AngularJS as JavaScript framework
  • HTML5 + CSS for the pages views

 

The goal of the website is to glue all these technologies together and try to shrink my CV to only one page (frscreencapture-www-mycv-host-1457966794473om the original 5 pages).

The formatting of the CV is done using almost only ‘<div>’ combined with CSS (the original template is not mine).

The data that fill the CV comes from a MongoDB server that contains the values. The data is in JSON format, a Java EE rest service send the data to AngularJS that fill the template.
2016-03-14_16-06-01_mongo

For each request to the CV a document with the information of the visitor is created in MongoDB an @Asynchronous method retrieve the visitor geographic information from freegeoip.net.

Using only CSS is possible to change the CV if the recruiter decide to print it. In this case the border are eliminated from the document and the social icon (links) are replaced by the email address.

2016-03-14_16-18-19_print

I don’t add code in this Post, you can find it on github. The code is changing frequently. I just show how easy is to connect to the DB on OpenShift:

 private MongoClient mongoClient = null;
 if (System.getenv("OPENSHIFT_MONGODB_DB_URL") == null){
    //localhost
     mongoClient = new MongoClient();
} else {
     // Openshift URL
     mongoClient = new MongoClient(new MongoClientURI(System.getenv("OPENSHIFT_MONGODB_DB_URL")));
 };

Security with Keycloak (and Google Oauth2)
I initially started implementing the security using Oauth2 (you can find it at this endpoint http://mycv.host/rest/login) but I found a better solution : Keycloak. It handles almost everything (email verification, social auth, password policies etc). The feature are impressive and allow to avoid a lot of risky plumbing on the security side.
Email verification request:
2016-03-17_10-46-32

Supported providers:
2016-03-17_11-32-48

Next steps:

  • Angular 2
  • Bootstrap or Material Design
  • Multiple users
  • Min js

RESTful web service with Java EE (JAX-RS) and JSON: Hello World tutorial

February 5, 2016 Glassfish, java, Java EE, JSON, Netbeans, REST, Web Services 1 comment , , , , , ,

You can find the java code here: https://github.com/marco76/simpleRestExample

For this example we use Netbeans 8.1 (with the new and nice darkula theme :)) that comes with Glassfish Server out-of-the-box.
The goal is to create and show the structure of a basic REST service in Java.
In the tutorial:
1. basic REST application with Java EE (no Spring)
2. response to a get with : text, list on JSON objects
3. simple integration test using Client

File structure

For this example we create 4 java classes (2 for the code, 1 for the unit test, 1 for the integration test).

2016-02-05_17-48-28

Configuration

In RestConfig.java we tell to the server that there is a REST service and his address (endpoint).

import javax.ws.rs.ApplicationPath;
import javax.ws.rs.core.Application;

/**
 *
 * This class is used to tell to the server that there are REST services.
 * An alternative is to declare a servlet in the web.xml.
 * 
 * @author marco
 */

@ApplicationPath("rest") // the 'rest' adress is mapped to the REST services
public class RestConfig extends Application{ // a javax.ws.rs.core.Application must be extended
    
}

We can configure the service with an annotation like in our example or declaring a Servlet in web.xml. In this example we don’t even create web.xml.
We leave the code of the class empty because we implements the methods in a second class: RestHelloWorld.java

Service class

Simple text answer

// @Provider tell the server that this is a REST class
@Provider  
// @Path defines the path for this class: [server]/rest/examples
@Path("examples") 
public class RestHelloWorld {

@Provider: tells to the server that this is a service class
@Path: part of the URL used to call the service

In this class we declare 1 method that return a simple text:

    // path used to call the method: [server]/rest/examples/examples/helloWorld
    @Path("helloWorld") 
    // answer only to a http get request
    @GET 
    // return a simple string (text/plain by default)
    public String hello(){  
        return "Hello World"; // string to be returned
    }
 

@Path: is the url to be used to call the service. It is added to the server URL + @ApplicationPath + @Path of the class. The name of the method is not important.
@GET : reads the calls with the http get method (ex. url written directly in the address bar of the browser)

This method return a simple text (no HTML) that is shown in the browser:

2016-02-05_17-01-34

Java List to JSON Array answer

The second method is more interesting because it return a list of String in JSON format


@Path("helloJSON")
@GET
@Produces("application/json")
public List<String> helloJSONList(){
    List<String> jsonList = new ArrayList<String>();
    jsonList.add("Hello World");
    jsonList.add("Bonjour monde");
        
    return jsonList;           
}
 

@Produces: defines in which format the answer should be returned. It automatically transform our object (List of strings) in JSON format.

2016-02-05_17-05-36

Netbeans recognize the services declared in this class and show them in the project structure:

2016-02-05_17-29-33

To transform Java Objects in JSON and vice versa we need to import some converters in our project to avoid some MessageBodyWriter or MessageBodyReader media type error:

 <!-- JSON support -->
        <!-- List to JSON -->
        <dependency>
            <groupId>com.sun.jersey</groupId>
            <artifactId>jersey-json</artifactId>
            <version>1.19</version>
        </dependency>
        <!-- JSON to List - MessageBodyReader media type -->
        <dependency>
            <groupId>org.glassfish.jersey.media</groupId>
            <artifactId>jersey-media-json-jackson</artifactId>
            <version>2.22.1</version>
        </dependency>

Integration Test

We can test our services directly using an explorer or, better, with some integration tests.

The class RestHelloWorldTestIT.java is used to test the REST service if it is up and running.
In the following code the test try to read the JSON list from the server converting it in a java String:


 @Test
 public void testIntegrationHelloJSON(){
     // the client connect to the REST service
     Client client = ClientBuilder.newClient();
        
     List<String> helloWorldString = client.target(helloWorldURL+"helloJSON") // connection to the pre-defined URL
                 .request()
                 .get(ArrayList.class); // we call the 'get' method and we transform the answer in a String
        assertEquals(2, helloWorldString.size());
 }
 

Because of the name convention Maven recognize that this is an integration test and it doesn’t execute it with the unit tests.
You need the failsafe plugin to profit from this feature:

   <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>
      ...