Skip to content

Commit

Permalink
Merge pull request #220 from telefonicaid/task/fix_rtd_images__releas…
Browse files Browse the repository at this point in the history
…e/1.0.0

Task/fix rtd images  release/1.0.0
  • Loading branch information
frbattid authored Oct 6, 2016
2 parents 71e37a6 + 2470a0e commit fb9dd95
Show file tree
Hide file tree
Showing 20 changed files with 13 additions and 234 deletions.
File renamed without changes
File renamed without changes
File renamed without changes
12 changes: 6 additions & 6 deletions doc/manuals/installation_and_administration_manual/gui.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,23 +176,23 @@ Please observe the user management for accessing the GUI or any other FIWARE com

Registration must be done one and only once. As an already registered user in the Identity Manager, login (this user will be the admin user). You should be able to see an applications panel, in addition to an organizations panel, in your home tab:

![](doc/images/register_cosmos_gui__apps_panel.png)
![](../images/register_cosmos_gui__apps_panel.png)

Click in the <i>register</i> button of the applications panel and give a name, a description, a URL and a callback URL for the new application. For instance:

![](doc/images/register_cosmos_gui__data.png)
![](../images/register_cosmos_gui__data.png)

Then choose an image for the application, this will be shown as an icon for future users:

![](doc/images/register_cosmos_gui__icon.png)
![](../images/register_cosmos_gui__icon.png)

Finally, manage the roles for this application. If you do not expect to add more roles than the default ones, or you simply do not know about roles, skip this step and finish the user registration:

![](doc/images/register_cosmos_gui__roles.png)
![](../images/register_cosmos_gui__roles.png)

cosmos-gui is now registered:

![](doc/images/register_cosmos_gui__result.png)
![](../images/register_cosmos_gui__result.png)

An important result of the registration process are the OAuth2 credentials that can be inspected by clicking on the appropriate button. These credentials must be configured in cosmos-gui as shown later.

Expand Down Expand Up @@ -267,7 +267,7 @@ Please observe the usage of `sudo`. This is because the GUI must be able to exec

If everything goes well, you should be able to see in a web browser the login page (`http://<node_hosting_cosmos_gui>:<port>`):

![](doc/images/cosmos_gui__init.png)
![](../images/cosmos_gui__init.png)

cosmos-gui typically listens in the TCP/443 port (TLS encryption), but you can change it by editing `conf/cosmos-gui.conf`.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ If you don't relay on [FIWARE LAB](http://account.lab.fiware.org/) global instan

If you still have doubts, we have built the flow diagram below in order to help you identifying which kind of Big Data user you are (if any):

![Figure 1 - Which kind of Big Data user am I?](./big_data_installation_guide_figure_1.png)
![Figure 1 - Which kind of Big Data user am I?](../images/big_data_installation_guide_figure_1.png)

[Top](#top)

Expand Down Expand Up @@ -57,4 +57,3 @@ There are several channels suited for reporting issues and asking for doubts in
**NOTE**: Please try to avoid personally emailing the GE owners unless they ask for it. In fact, if you send a private email you will probably receive an automatic response enforcing you to use [stackoverflow.com](http://stackoverflow.com) or [ask.fiware.org](http://ask.fiware.org/questions/). This is because using the mentioned methods will create a public database of knowledge that can be useful for future users; private email is just private and cannot be shared.

[Top](#top)

Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ The ecosystem raises when Cosmos is completed/complemented with certain manageme
* [Hadoop extensions](http://github.com/telefonicaid/fiware-tidoop/tree/master/tidoop-hadoop-ext) for using non HDFS data.
* [Wilma PEP Proxy](http://github.com/ging/fi-ware-pep-proxy) together with [OAuth2 Tokens Generator](http://github.com/telefonicaid/fiware-cosmos/tree/develop/cosmos-auth) if wanting to implement OAuth2-based authentication and authorization in your REST APIs.

![Figure 2 - Big Data architecture reminder](big_data_installation_guide_figure_2.png "Figure 2 - Big Data architecture reminder")
![Figure 2 - Big Data architecture reminder](../images/big_data_installation_guide_figure_2.png "Figure 2 - Big Data architecture reminder")

[Top](#top)

Expand All @@ -45,7 +45,7 @@ Since the above solution may require a very large infrastructure (for instance,

Indeed, this is the version currently deployed in FIWARE Lab.

![Figure 3 - Light-version Big Data architecture](big_data_installation_guide_figure_3.png "Figure 3 - Light-version Big Data architecture")
![Figure 3 - Light-version Big Data architecture](../images/big_data_installation_guide_figure_3.png "Figure 3 - Light-version Big Data architecture")

##<a name="section3"></a>What is mandatory and optional

Expand Down
8 changes: 4 additions & 4 deletions doc/manuals/user_and_programmer_manual/using_gui.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@ Content:
##<a name="section1"></a>Login
Once installed and run, you can visit `http://<node_hosting_cosmos_gui>:<port>` (adapt the port if you changed it). This page basically prompts the user to login.

![](doc/images/cosmos_gui__init.png)
![](../images/cosmos_gui__init.png)

The login procedure delegates in FIWARE Identity Manager. This means cosmos-gui does not perform <i>any propietary user management</i> from the GUI point of view (as it will be seen, cosmos-gui performs a propietary user management for accessing the Hadoop cluster; that is, particularly, its purpose). Thus, once clicked the login button, we are redirected to `https://account.lab.fiware.org`:

![](doc/images/cosmos_gui__auth.png)
![](../images/cosmos_gui__auth.png)

[Top](#top)

Expand All @@ -36,7 +36,7 @@ Next coming versions of the GUI will allow the users to explore their HDFS space

The only option for the time being is to access to the profile page (see next section).

![](doc/images/cosmos_gui__dashboard.png)
![](../images/cosmos_gui__dashboard.png)

[Top](#top)

Expand All @@ -45,6 +45,6 @@ The profile section shows the user account details and certain statistics, such

This is useful in order to know the credentials the user has in the Cosmos platform.

![](doc/images/cosmos_gui__profile.png)
![](../images/cosmos_gui__profile.png)

[Top](#top)
220 changes: 0 additions & 220 deletions doc/manuals/user_and_programmer_manual/using_hadoop_and_ecosystem.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,6 @@ Content:<br>
* [Programming a custom Hive client](#section2.2)
* [Java](#section2.2.1)
* [Python](#section2.2.2)
* [Oozie](#section3)
* [Oozie CLI](#section3.1)
* [Oozie workflow](#section3.2)
* [Programming a custom Oozie client](#section3.3)
* [Java](#section3.3.1)

##<a name="section1"></a>MapReduce

Expand Down Expand Up @@ -662,218 +657,3 @@ The following code implements a basic Hive client using Python:
print i`

[Top](#top)

##<a name="section3"></a>Using Oozie

Oozie is a workflow scheduler system for Apache Hadoop jobs. It allows designing Oozie Workflows, i.e. Directed Acyclical Graphs (DAGs) of actions, which in the end coordinate the execution of the jobs.

An action can be a MapReduce job, a Pig application, a file system task, or a Java application. Flow control in the DAGs is performed by node
elements providing a certain logic based on the input of the preceding task in the graph (e.g. forks, joins, decision nodes), or when an event (time, whatever) triggers. encies have been met.

An example of DAG is the following one:

![Figure 1 - Example of Oozie DAG](oozie_dag.png "Figure 1 - Example of Oozie DAG")

Oozie workflows definitions are written in hPDL (a XML Process Definition Language similar to JBOSS JBPM jPDL). Oozie workflows can be parameterized (using variables like \${inputDir} within the workflow definition). When submitting a workflow job values for the parameters must be provided. If properly parameterized (i.e. using different output directories) several identical workflow jobs can concurrently.

Oozie can be used in three ways, command line, Java client API and API REST.

[Top](#top)

###<a name="section3.1"></a>Oozie CLI

In order to use Oozie through commands, it is necessary to install the Oozie client in a remote machine. This client will be able to talk with the Oozie server already installed in the cluster (TCP/11000 port). This [official guidelines](http://oozie.apache.org/docs/4.0.0/DG_QuickStart.html#Client_Installation) will help you to install it.

Once the client has been setup, you can schedule your jobs by following this other [guidelines](http://oozie.apache.org/docs/4.0.0/DG_Examples.html#Command_Line_Examples). Basically, you have to type a command like this one:

$ oozie job -oozie http://<master-node>:11000/oozie -config examples/apps/map-reduce/job.properties -run

If everything goes well, a job identifier is printed. As can be seen, you must specify the Master Node IP address or hostname, and the path to the Oozie application you want to run (in the above command, we are using the examples given with the Oozie distribution).

The oozie command can provide per job status information as well:

$ oozie job -oozie http://<master-node>:11000/oozie -info <job-identifier>
Job ID : 0000014-140116081225611-oozie-oozi-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : map-reduce-wf
App Path : hdfs://<name-node>:8020/user/<user>/examples/apps/map-reduce/workflow.xml
Status : RUNNING
Run : 0
User : <user>
Group : users
Created : 2014-01-16 16:53
Started : 2014-01-16 16:53
Last Modified : 2014-01-16 16:53
Ended : -

Actions
------------------------------------------------------------------------------------------------------------------------------------
ID Status Ext ID Ext Status Err Code
------------------------------------------------------------------------------------------------------------------------------------
0000014-140116081225611-oozie-oozi-W@mr-node RUNNING job_201401151554_0032 RUNNING -
------------------------------------------------------------------------------------------------------------------------------------`

Oozie applications consists of a well-known structured directories containing:

* **lib/**, a directory containing the MR job jar, needed libraries, etc.
* **workflow.xml**, the workflow definition written in hPDL.
* **job.properties**, the values for the parameters used in the workflow definition.

The programming guide will provide guidelines about how to create your own Oozie applications.

[Top](#top)

###<a name="section3.2"></a>Oozie workflow

Writing Oozie workflows can be a hard task, but in a few words it is necessary to generate a XML document as the one below, containing a list of flow control nodes (e.g. start, kill, end) and actions to be executed. Flow control nodes and actions may be parameterized in the
`job.properties` file. This file must be called `workflow.xml` and must be included in the application folder.

<workflow-app name="wordcount-wf" xmlns="uri:oozie:workflow:0.1">
<start to="wordcount"/>
<action name="wordcount">
<map-reduce>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.mapper.class</name>
<value>org.myorg.WordCount.Map</value>
</property>
<property>
<name>mapred.reducer.class</name>
<value>org.myorg.WordCount.Reduce</value>
</property>
<property>
<name>mapred.input.dir</name>
<value>${inputDir}</value>
</property>
<property>
<name>mapred.output.dir</name>
<value>${outputDir}</value>
</property>
</configuration>
</map-reduce>
<ok to="end"/>
<error to="end"/>
</action>
<kill name="kill">
<message>Something went wrong: ${wf:errorCode('wordcount')}</message>
</kill/>
<end name="end"/>
</workflow-app>

All the detailed documentation can be found [here](http://archive.cloudera.com/cdh/3/oozie/WorkflowFunctionalSpec.html).

Once a workflow is written, it is ready to be executed with the Oozie client as described in the *Using Oozie* section. Only if you a need a custom mechanism to run the workflows, the Java client API and the API REST are recommended (see next sections).

[Top](#top)

###<a name="section3.3"></a>Programming a custom Oozie client

Oozie provides a Java API for custom Java clients development. There is a step-by-step example in this [official link](http://oozie.apache.org/docs/4.0.0/DG_Examples.html#Java_API_Example).

Additionally, the [API REST](http://oozie.apache.org/docs/4.0.0/WebServicesAPI.html) for Oozie allows you for submitting and running workflows in a REST fashion.

The programming section of this document will explain you how to create an application taking advantage both of the Oozie API and the Oozie REST API.

[Top](#top)

####<a name="section3.3.1"></a>Java

Oozie provides a Java API for custom Java clients development. There is a step-by-step example in this [official link](http://oozie.apache.org/docs/4.0.0/DG_Examples.html#Java_API_Example) which is summarized here.

Add the following lines to your Maven based pom.xml in order to solve the dependencies:

<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>

<dependencies>
...
<dependency>
<groupId>com.yahoo.oozie</groupId>
<artifactId>oozie-client</artifactId>
<version>2.3.2-cdh3u6</version>
</dependency>
</dependencies>

This is the minimum code:

package com.mycompany.oozieclienttest;

import java.util.logging.Level;
import java.util.logging.Logger;
import org.apache.oozie.client.OozieClient;
import org.apache.oozie.client.OozieClientException;
import org.apache.oozie.client.WorkflowJob;
import java.util.Properties;

/**
* Oozie client test.
*
*/
public final class OozieClientTest {

/**
*
*/
private OozieClientTest() {
} // OozieClientTest

/**
*
* @param args
*/
public static void main(String[] args) {
// get a OozieClient for local Oozie
OozieClient client = new OozieClient("`[`http://130.206.80.46:11000/oozie/`](http://130.206.80.46:11000/oozie/)`");

// create a workflow job configuration and set the workflow application path
Properties conf = client.createConfiguration();
conf.setProperty(OozieClient.APP_PATH, "hdfs://cosmosmaster-gi:8020/user/frb/examples/apps/map-reduce");

// setting workflow parameters
conf.setProperty("nameNode", "hdfs://cosmosmaster-gi:8020");
conf.setProperty("jobTracker", "cosmosmaster-gi:8021");
conf.setProperty("outputDir", "output-data");
conf.setProperty("examplesRoot", "examples");
conf.setProperty("queueName", "default");

// submit and start the workflow job
String jobId = null;

try {
jobId = client.run(conf);
} catch (OozieClientException ex) {
Logger.getLogger(OozieClientTest.class.getName()).log(Level.SEVERE, null, ex);
} // try catch

System.out.println("Workflow job submitted");

try {
// wait until the workflow job finishes printing the status every 10 seconds
while (client.getJobInfo(jobId).getStatus() == WorkflowJob.Status.RUNNING) {
System.out.println("Workflow job running ...");
Thread.sleep(10 * 1000);
} // while
} catch (OozieClientException ex) {
Logger.getLogger(OozieClientTest.class.getName()).log(Level.SEVERE, null, ex);
} catch (java.lang.InterruptedException ex) {
Logger.getLogger(OozieClientTest.class.getName()).log(Level.SEVERE, null, ex);
} // try catch catch

// print the final status o the workflow job
System.out.println("Workflow job completed ...");

try {
System.out.println(client.getJobInfo(jobId));
} catch (OozieClientException ex) {
Logger.getLogger(OozieClientTest.class.getName()).log(Level.SEVERE, null, ex);
} // try catch
} // main
} // OozieClientTest

[Top](#top)

0 comments on commit fb9dd95

Please sign in to comment.