Merge pull request #24 from telefonicaid/feature/23_admin_commands_ov…

…er_ssh

hardening/23_admin_commands_over_ssh

CHANGES_NEXT_RELEASE

@@ -3,5 +3,6 @@
 - [cosmos] [HARDENING] Add initial content to the general README (#1)
 - [cosmos-gui] [HARDENING] Set 740 permissions to new created accounts (#17)
 - [cosmos-gui] [HARDENING] Add the HDFS superuser as a configuration parameter (#20)
+- [cosmos-gui] [HARDENING] Run administration commands remotely through ssh (#23)
 - [cosmos-gui] [HARDENING] Add detailed information about reporting issues and contact information (#5)
 - [cosmos-gui] [HARDENING] Running as sudo is not required anymore (#28)

cosmos-gui/README.md

@@ -16,14 +16,14 @@
 * [Reporting issues and contact information](#contact)
 
 ##<a name="whatis"></a>What is cosmos-gui
-This is one of the pieces of the called "Cosmos ecosystem". Cosmos is the code name for a [Hadoop](http://hadoop.apache.org/)-based solution to FIWARE's BigData Analysis Generic Enabler; such a solution is based on the split of storage and computing:
+This is one of the pieces of the named "Cosmos Ecosystem". Within such an ecosystem there is a [Hadoop](http://hadoop.apache.org/)-based implementation of FIWARE's BigData Analysis Generic Enabler; such a solution is based on the split of storage and computing capabilities:
 
 * A only-[HDFS](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html) cluster for permanently storing the user data.
 * Depending on the available resources and the goals pursued by your deployment, there are two flavours for the computing side:
     * Another Hadoop cluster, shared among all the users, addressing data processing and only allowing for temporal storage.
     * A [Sahara](https://wiki.openstack.org/wiki/Sahara)-based platform for on-demand private temporal Hadoop clusters.
 
-As seen, the storage cluster is always shared, thus a provisioning procedure is require in order to create specific Unix users and HDFS user spaces; this is also know as creating a <i>Cosmos account</i>. This procedure is automated by this cosmos-gui, a Node.js application rendering a set of web pages mainly in charge of guiding the user through this provisioning step.
+As seen, the storage cluster is always shared, and depending on the chosen flavour, the computing cluster is shared as well. Thus a provisioning procedure is require in order to create specific Unix users and HDFS user spaces within both clusters; this is also know as creating a <i>Cosmos account</i>. This procedure is automated by this cosmos-gui, a Node.js application rendering a set of web pages mainly in charge of guiding the user through this provisioning step.
 
 In addition, the cosmos-gui can be used as a centralized dashboard where a user can explore its HDFS space and run [predefined MapReduce](https://github.com/telefonicaid/fiware-tidoop/tree/develop/tidoop-mr-lib-api) jobs, once his/her Cosmos account has been provisioned.
 
@@ -35,7 +35,9 @@ This is a software written in JavaScript, specifically suited for [Node.js](http
 [Top](#top)
 
 ###<a name="prerequisites"></a>Prerequisites
-This GUI has no sense if there is no [Hadoop](http://hadoop.apache.org/)-based storage cluster to be managed.
+This GUI has no sense if there is no storage and computing clusters to be managed.
+
+A couple of sudoer users, one within the storage cluster and another one wihtin the computing clusters, are required. Through these users the cosmos-gui will remotely run certain administration commands such as new users creation, HDFS userspaces provision, etc. The access through these sudoer users will be authenticated by means of private keys.
 
 The Cosmos users management is done by means of a [MySQL](https://www.mysql.com/) database, thus install it in the same node the GUI runs, or a remote but accessible machine.
 
@@ -46,20 +48,27 @@ Of course, common Unix tools such as `git` and `curl` are needed.
 [Top](#top)
 
 ###<a name="gui"></a>Installating the GUI
-cosmos-gui must be installed in a machine being part of the storage cluster to be managed. This can be done in the Namenode, or in a dedicated node for services related to the cluster, such as HttpFS.
+cosmos-gui must be installed in a machine having ssh access both to the storage and computing clusters the GUI is going to manage. This ssh access may be limited to the Namenode (or Namenodes, if HA is enabled) of each cluster, and it is necessary since certain administration commands are remotely run through ssh.
+
+Start by creating, if not yet created, a Unix user named `cosmos-gui`; it is needed for installing and running the application. You can only do this as root, or as another sudoer user:
 
-Once logged into the node, start by creating, if not yet created, a sudoer Unix user named `cosmos`; it is needed for installing and running the application. You can only do this as root, or as another sudoer user:
+    $ sudo useradd cosmos-gui
+    $ sudo passwd cosmos-gui <choose_a_password>
+
+Now, change to the new fresh `cosmos-gui` user:
 
-    $ sudo useradd cosmos sudo   # if the 'sudo' group is within /etc/sudoers
-    $ sudo useradd cosmos        # if the 'sudo' group is not within /etc/sudoers
-    $ sudo passwd cosmos <choose_a_password>
+    $ su - cosmos-gui
 
-If the `sudo` group is not within `/etc/sudoers` you can add such a group or add specifically the `cosmos` user, as you want. This is done by invoking the `sudo visudo` command.
+Before continuing, remember to add the RSA key fingerprints of the Namenodes accessed by the GUI. This fingerprints are automatically added to `/home/cosmos-gui/.ssh/known_hosts` if you try a ssh access to the Namenodes for the first time. 
+
+    $ ssh [email protected]
+    The authenticity of host 'my.storage.namenode.com (192.168.12.1)' can't be established.
+    RSA key fingerprint is 96:c4:0b:8c:09:ce:d4:09:91:a2:b2:9c:40:71:9b:c6.
+    Are you sure you want to continue connecting (yes/no)? yes
+    Warning: Permanently added 'my.storage.namenode.com,192.168.12.1' (RSA) to the list of known hosts.
 
-Now, change to the new fresh `cosmos` user:
+Please observe `somesudoeruser` is the (ficticious) sudoer user required for the storage cluster, as stated in the [Prerequisites](#prerequisites) section. Do the same for the computing cluster.
 
-    $ su - cosmos
-
 Then, clone the Cosmos repository somewhere of your ownership:
 
     $ git clone https://github.com/telefonicaid/fiware-cosmos.git
@@ -135,10 +144,19 @@ To be done.
 [Top](#top)
 
 ##<a name="configuration"></a>Configuration
-cosmos-gui is configured through `conf/cosmos-gui.json`. There you will find a JSON document with four main *sections*:
+cosmos-gui is configured through `conf/cosmos-gui.json`. There you will find a JSON document with five main *sections*:
 
 * **gui**:
     * **port**: specifies the listening port for the application. By default it is 80, but can be changed if such a port is being used in your deployment.
+* **clusters**:
+    * **storage**
+        * **endpoint**: IP address or FQDN of the Namenode/HttpFS server of the storage cluster.
+        * **user**: Unix user within the Namenode/HttpFS server having sudo permissions.
+        * **private_key**: user's private key used to ssh into the Namenode/HttpFS server.
+    * **computing**
+        * **endpoint**: IP address or FQDN of the Namenode/HttpFS server of the computing cluster.
+        * **user**: Unix user within the Namenode/HttpFS server having sudo permissions.
+        * **private_key**: user's private key used to ssh into the Namenode/HttpFS server.
 * **hdfs**:
     * **quota**: measured in gigabytes, defines the size of the HDFS space assigned to each Cosmos user.
     * **superuser**: HDFS superuser, typically `hdfs`.

cosmos-gui/conf/cosmos-gui.json

@@ -2,15 +2,27 @@
   "gui": {
     "port": 80
   },
+  "clusters": {
+    "storage": {
+      "endpoint": "",
+      "user": "",
+      "private_key": ""
+    },
+    "computing": {
+      "endpoint": "",
+      "user": "",
+      "private_key": ""
+    }
+  },
   "hdfs": {
     "quota": 5,
     "superuser": "hdfs"
   },
   "oauth2": {
-    "idmURL": "",
+    "idmURL": "https://account.lab.fiware.org",
     "client_id": "",
     "client_secret": "",
-    "callbackURL": "http://localhost:80/auth",
+    "callbackURL": "",
     "response_type": "code"
   },
   "mysql": {

cosmos-gui/public/stylesheets/style.css

@@ -105,38 +105,28 @@ a img {
  */
 body {
   font-family: 'Georgia';
-  background-color: #faf9f0;
+  background-color: #fff;
   color: #444;
 }
 header {
-  font-family: 'Quicksand';
-  padding: 50px 10px;
-  color: #fff;
-  font-size: 25px;
   text-align: center;
-/*
-   * Note the use of the `main-color`
-   * variable and the `darken` function
-   */
-  background-color: #fa5b4d;
-  border-bottom: 1px solid #df1806;
-  text-shadow: 0px -1px 0px #df1806;
+  background-color: #fff;
 }
 .container {
-  width: 500px;
+  width: 700px;
   margin: 50px auto;
   overflow: hidden;
 }
 .main-content {
   float: left;
-  width: 300px;
+  width: 400px;
 }
 .main-content p {
   margin-bottom: 20px;
 }
 .sidebar {
   float: left;
-  width: 200px;
+  width: 300px;
 }
 .sidebar .widget {
 /*
@@ -148,7 +138,7 @@ header {
   border-radius: 3px;
   border: 1px solid #ccc;
   margin-left: 20px;
-  background-color: #faf9f0;
+  background-color: #fff;
   -webkit-box-shadow: inset 0px 0px 0px 3px rgba(255,255,255,0.8), 0px 3px 0px -2px rgba(0,0,0,0.1);
   box-shadow: inset 0px 0px 0px 3px rgba(255,255,255,0.8), 0px 3px 0px -2px rgba(0,0,0,0.1);
 }
@@ -170,9 +160,24 @@ p {
   line-height: 1.8;
 }
 footer {
-  width: 500px;
-  margin: 50px auto;
+  margin: 20px auto;
   border-top: 1px dotted #ccc;
   padding-top: 5px;
   font-size: 13px;
+  text-align: center;
+  bottom: 0;
+  position: absolute;
+  width: 100%;
+}
+#submit {
+  background-color: #ccc;
+  padding: 5px;
+  margin: 10px;
+}
+#submit2 {
+  background-color: #ccc;
+  padding: 5px;
+}
+#password {
+  width: 400px;
 }

cosmos-gui/src/app.js

@@ -69,6 +69,12 @@ var response_type = config.oauth2.response_type;
 var callbackURL = config.oauth2.callbackURL;
 var hdfsQuota = config.hdfs.quota;
 var hdfsSuperuser = config.hdfs.superuser;
+var scPrivKey = config.clusters.storage.private_key;
+var scUser = config.clusters.storage.user;
+var scEndpoint = config.clusters.storage.endpoint;
+var ccPrivKey = config.clusters.computing.private_key;
+var ccUser = config.clusters.computing.user;
+var ccEndpoint = config.clusters.computing.endpoint;
 
 // Creates oauth library object with the config data
 var oa = new OAuth2(client_id,
@@ -142,52 +148,52 @@ app.post('/new_account', function(req, res) {
     if (password1 === password2) {
         mysqlDriver.addUser(idm_username, username, password1, function(error, result) {
             if (error) {
-                res.boom.badData('There was some error when adding information in the database for user ' + username, error);
+                res.boom.badData('There was some error when adding information in the database for user '+ username, error);
                 return;
             } // if
 
-            cmdRunner.run('bash', ['-c', 'useradd ' + username], function(error, result) {
+            cmdRunner.run('bash', ['-c', 'echo "sudo useradd ' + username + '" | ssh -i ' + scPrivKey + ' ' + scUser + '@' + scEndpoint], function(error, result) {
                 if (error) {
                     res.boom.badData('There was an error while adding the Unix user ' + username, error);
                     return;
                 } // if
 
-                console.log('Successful command executed: \'bash -c useradd ' + username + '\'');
-                cmdRunner.run('bash', ['-c', 'echo ' + password1 + ' | passwd ' + username + ' --stdin'], function(error, result) {
+                console.log('Successful command executed: \'bash -c echo "sudo useradd ' + username + '" | ssh -i ' + scPrivKey + ' ' + scUser + '@' + scEndpoint + '\'');
+                cmdRunner.run('bash', ['-c', 'echo "echo ' + password1 + ' | sudo passwd ' + username + ' --stdin' + '" | ssh -i ' + scPrivKey + ' ' + scUser + '@' + scEndpoint], function(error, result) {
                     if (error) {
                         res.boom.badData('There was an error while setting the password for user ' + username, error);
                         return;
                     } // if
 
-                    console.log('Successful command executed: \'bash -c echo ' + password1 + ' | passwd ' + username + ' --stdin\'');
-                    cmdRunner.run('bash', ['-c', 'sudo -u ' + hdfsSuperuser + ' hadoop fs -mkdir /user/' + username], function(error, result) {
+                    console.log('Successful command executed: \'bash -c echo "echo ' + password1 + ' | sudo passwd ' + username + ' --stdin' + '" | ssh -i ' + scPrivKey + ' ' + scUser + '@' + scEndpoint + '\'');
+                    cmdRunner.run('bash', ['-c', 'echo "sudo -u ' + hdfsSuperuser + ' hadoop fs -mkdir /user/' + username + '" | ssh -i ' + scPrivKey + ' ' + scUser + '@' + scEndpoint], function(error, result) {
                         if (error) {
                             res.boom.badData('There was an error while creating the HDFS folder for user ' + username, error);
                             return;
                         } // if
 
-                        console.log('Successful command executed: \'bash -c sudo -u ' + hdfsSuperuser + ' hadoop fs -mkdir /user/' + username + '\'');
-                        cmdRunner.run('bash', ['-c', 'sudo -u ' + hdfsSuperuser + ' hadoop fs -chown -R ' + username + ':' + username + ' /user/' + username], function(error, result) {
+                        console.log('Successful command executed: \'bash -c echo "sudo -u ' + hdfsSuperuser + ' hadoop fs -mkdir /user/' + username + '" | ssh -i ' + scPrivKey + ' ' + scUser + '@' + scEndpoint + '\'');
+                        cmdRunner.run('bash', ['-c', 'echo "sudo -u ' + hdfsSuperuser + ' hadoop fs -chown -R ' + username + ':' + username + ' /user/' + username + '" | ssh -i ' + scPrivKey + ' ' + scUser + '@' + scEndpoint], function(error, result) {
                             if (error) {
                                 res.boom.badData('There was an error while changing the ownership of /user/' + username, error);
                                 return;
                             } // if
 
-                            console.log('Successful command executed: \'bash -c sudo -u ' + hdfsSuperuser + ' hadoop fs -chown -R ' + username + ':' + username + ' /user/' + username + '\'');
-                            cmdRunner.run('bash', ['-c', 'sudo -u ' + hdfsSuperuser + ' hadoop fs -chmod -R 740 /user/' + username], function(error, result) {
+                            console.log('Successful command executed: \'bash -c echo "sudo -u ' + hdfsSuperuser + ' hadoop fs -chown -R ' + username + ':' + username + ' /user/' + username + '" | ssh -i ' + scPrivKey + ' ' + scUser + '@' + scEndpoint + '\'');
+                            cmdRunner.run('bash', ['-c', 'echo "sudo -u ' + hdfsSuperuser + ' hadoop fs -chmod -R 740 /user/' + username + '" | ssh -i ' + scPrivKey + ' ' + scUser + '@' + scEndpoint], function(error, result) {
                                 if (error) {
                                     res.boom.badData('There was an error while changing the permissions to /user/' + username, error);
                                     return;
                                 } // if
 
-                                console.log('Successful command executed: \'bash -c sudo -u ' + hdfsSuperuser + ' hadoop fs -chmod -R 740 /user/' + username + '\'');
-                                cmdRunner.run('bash', ['-c', 'sudo -u ' + hdfsSuperuser + ' hadoop dfsadmin -setSpaceQuota ' + hdfsQuota + 'g /user/' + username], function(error, result) {
+                                console.log('Successful command executed: \'bash -c echo "sudo -u ' + hdfsSuperuser + ' hadoop fs -chmod -R 740 /user/' + username + '" | ssh -i ' + scPrivKey + ' ' + scUser + '@' + scEndpoint + '\'');
+                                cmdRunner.run('bash', ['-c', 'echo "sudo -u ' + hdfsSuperuser + ' hadoop dfsadmin -setSpaceQuota ' + hdfsQuota + 'g /user/' + username + '" | ssh -i ' + scPrivKey + ' ' + scUser + '@' + scEndpoint], function(error, result) {
                                     if (error) {
                                         res.boom.badData('There was an error while setting the quota to /user/' + username, error);
                                         return;
                                     } // if
 
-                                    console.log('Successful command executed: \'bash -c sudo -u ' + hdfsSuperuser + ' hadoop dfsadmin -setSpaceQuota ' + hdfsQuota + 'g /user/' + username + '\'');
+                                    console.log('Successful command executed: \'bash -c echo "sudo -u ' + hdfsSuperuser + ' hadoop dfsadmin -setSpaceQuota ' + hdfsQuota + 'g /user/' + username + '" | ssh -i ' + scPrivKey + ' ' + scUser + '@' + scEndpoint + '\'');
                                     res.redirect('/');
                                 })
                             })