R: functions

Defining a function in R, looks like

 mynewsumfunction <- function(x,y = 34){
  x + y
 }

Parameters can have a default value.
If parameters are strings  you can use function paste

paste(c("The", "quick", "brown", "fox"))
[1] "The"  "quick" "brown" "fox"

Ellipsis allows to declare a variable number of parameters in a function.

> myhist <- function(x, border="blue", ...){
+ 	hist(x, border=border, ...)
+ }
> set.seed(5)
> myhist(rnorm(1000), breaks=30)
> set.seed(5)
> myhist(rnorm(1000))  

Function prototypes can be checked with args function:

args(mean)
function (x, ...)
Environments:

Every function and symbol has to be defined in an environment:

  • global
  • packages loaded
  • package base
search (myfunction)

Functions can be defined within other functions.

environment(myfunction)

Getting started with gitlab

Installation for Debian 8

curl -sS https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash
sudo apt-get install gitlab-ce

sudo gitlab-ctl reconfigure

https://about.gitlab.com/downloads/#debian8

Restarting gitlab

To restart web service

gitlab-ctl restart
ok: run: gitlab-workhorse: (pid 29291) 0s
ok: run: logrotate: (pid 29299) 1s
ok: run: nginx: (pid 29305) 0s
ok: run: postgresql: (pid 29313) 0s
ok: run: redis: (pid 29327) 1s
ok: run: sidekiq: (pid 29331) 0s
ok: run: unicorn: (pid 29336) 1s

Reconfigure gitlab settings

La configuración de gitlab se guarda en un archivo .yml, que se genera a través de :

emacs /etc/gitlab/gitlab.rb
sudo gitlab-ctl reconfigure

To locate configuration files in our system

find / -name gitlab.rb
find / -name gitlab.yml

Logging

Gitlab logs are separated in three files: application, sidekiq and production

tail -f /var/log/gitlab/gitlab-rails/production.log
tail -f /var/log/gitlab/gitlab-rails/application.log
tail -f /var/log/gitlab/gitlab-rails/sidekiq.log

Code highlighting

For highlighting source code, Gitlab uses Rouge module.

To know whether a language will be coloured or not, they have this awesome website where you can actually test colouring different code snippets.

Available grammars are in Github.

Backup de correo pst

Los archivos pst de correo en outlook se ubican en

/cygdrive/c/Users/myuser/AppData/Local/Microsoft/Outlook/archive.pst

En mi caso, al tener más de 4 GB el archivo, no es posible hacer un backup a un disco duro Fat32.

Vamos a usar cygwin y tar, para hacer una copia del correo, separada en archivos de 1 GB:

CORREO=/cygdrive/c/Users/myuser/AppData/Local/Microsoft/Outlook/archive.pst

tar cz $CORREO  | split -b 1024MiB - $CORREO_bkp

Para recomponer:

cat myfiles_split.tgz_* | tar xz

R: Plots and graphs

Plotting in R is extremely easy.

plot (1:21)

rstudio_plotting_000

plot (1:8, main="TESTING",xlab="x",ylab="y")

rstudio_plotting_001

points(c(1,5),c(4,2), col="green")

rstudio_plotting_002

abline(h=3,col="red",lty=2)

rstudio_plotting_003

abline(h=1,col="yellow",lty=2)

rstudio_plotting_004

grid(col="purple")

rstudio_plotting_005

hist (1:21,col="orange")

rstudio_plotting_006

lines(c(2,4),c(5,3),col="blue")

rstudio_plotting_007

abline(h=3,col="red",lty=2)

Exporting and importing graphs

dev.copy(png,'myplot.png')
dev.off()

To list existing colors

> colors()
  [1] "white"                "aliceblue"            "antiquewhite"         "antiquewhite1"       
  [5] "antiquewhite2"        "antiquewhite3"        "antiquewhite4"        "aquamarine"          
  [9] "aquamarine1"          "aquamarine2"          "aquamarine3"          "aquamarine4"         
 [13] "azure"                "azure1"               "azure2"               "azure3"              
(...)
"violetred2"           "violetred3"          
[645] "violetred4"           "wheat"                "wheat1"               "wheat2"              
[649] "wheat3"               "wheat4"               "whitesmoke"           "yellow"              
[653] "yellow1"              "yellow2"              "yellow3"              "yellow4"             
[657] "yellowgreen"

Global changes for styles and colors

Adding texts to plots

text(11,11,"Regular text")

text(11,11, "Bigger text", cex=2)

text(5,5, "serif text", family="serif")

text(3,3 "mono text", family="mono")

text(7,7 "Sans text", family="sans")

 

Mathematic notation

demo(plotmath)

draw.plotmath.cell(expression(x != y), i, nr); i <- i + 1

> draw.plotmath.cell(expression(x < y), i, nr); i <- i + 1

> draw.plotmath.cell(expression(x <= y), i, nr); i <- i + 1

> draw.plotmath.cell(expression(x > y), i, nr); i <- i + 1

> draw.plotmath.cell(expression(x >= y), i, nr); i <- i + 1

> draw.plotmath.cell(expression(x %~~% y), i, nr); i <- i + 1

> draw.plotmath.cell(expression(x %=~% y), i, nr); i <- i + 1

> draw.plotmath.cell(expression(x %==% y), i, nr); i <- i + 1

> draw.plotmath.cell(expression(x %prop% y), i, nr); i <- i + 1

> draw.plotmath.cell(expression(x %~% y), i, nr); i <- i + 1

 

Installing Dataprotector 9 Agents on Debian 8

For some reason, dataprotector agents are released as an ISO image, with a +3000 lines shell script omnisetup.sh.

sudo ./omnisetup.sh -server myserver.myorg.es -install da

This lengthy script install dataprotector clients or server in several architectures:

  • osf1

  • hp-ux

  • SunOs and Solaris

  • AIX,

  • Linux x86, linux ia64

  • sco_sv

  • Darwin

  • PowerPC

For linux, it relays con rpm configuration. For Debian, the way to go would be alien.

Packages needed for Dataprotector agents (DA):

The only packages that are needed for the agents are:

  • OB2-TS-CORE-A.09.00-1.x86_64

  • OB2-CORE-A.09.00-1.x86_64

  • OB2-DA-A.09.00-1.x86_64

So, our first and naive aproach was a conversion with Alien and install them via dpkg -i.

for RPM in *.rpm; do sudo alien -c -k -d --fixperms $RPM; done

This led to an error as this three packages couldn’t be installed because we lacked some util called omnicc.

At this point, I tried running omnicc with bash -x, so that I could see what the long omnisetup was trying to do in my Debian machine.

bash -x ./omnisetup.sh -server myserver.myorg.es -install da

And then I noticed that, though only the above packages were installed, some others were uncompressed and used:

rpm2cpio /usr/local/share/Software_HP_DP_9.00_for_Linux_TD586-15021/linux_x86_64/DP_DEPOT/OB2-CORE-IS-A.09.00-1.x86_64.rpm 

cpio -id ./opt/omni/databases/utils/gpl/x86_64/linux-x86-64/utils.tar

cp ./opt/omni/databases/utils/gpl/x86_64/linux-x86-64/utils.tar .
(...)
rpm2cpio /usr/local/share/Software_HP_DP_9.00_for_Linux_TD586-15021/linux_x86_64/DP_DEPOT/OB2-TS-CFP-A.09.00-1.x86_64.rpm

cpio -id ./opt/omni/databases/vendor/ts_core/gpl/x86_64/linux-x86-64/A.09.00/packet.Z

PacketFullPath=/tmp/omni_tmp/opt/omni/databases/vendor/ts_core
/gpl/x86_64/linux-x86-64/A.09.00/packet.Z

(...)

rpm2cpio /usr/local/share/Software_HP_DP_9.00_for_Linux_TD586-15021/linux_x86_64/DP_DEPOT/OB2-DAP-A.09.00-1.x86_64.rpm

cpio -id ./opt/omni/databases/vendor/da/gpl/x86_64/linux-x86-64/A.09.00/packet.Z

PacketFullPath=/tmp/omni_tmp/opt/omni/databases/vendor/da/gpl/x86_64/linux-x86-64/A.09.00/packet.Z

(...)

At this point, I decided to stick to rpm installation, and use the omnisetup.sh script.

These are the files from the ISO image that we need to distribute to our Debian machines:

linux_x86_64 
│   ├── DP_DEPOT
│   │   ├── gpg-hpPublicKey.pub
│   │   ├── OB2-CFP-A.09.00-1.x86_64.rpm
│   │   ├── OB2-CORE-IS-A.09.00-1.x86_64.rpm
│   │   ├── OB2-DA-A.09.00-1.x86_64.rpm
│   │   ├── OB2-DAP-A.09.00-1.x86_64.rpm
│   │   ├── OB2-TS-CFP-A.09.00-1.x86_64.rpm
│   │   ├── OB2-TS-CORE-A.09.00-1.x86_64.rpm
│   │   └── OB2-TS-CS-A.09.00-1.x86_64.rpm
│   ├── scripts_linux_x86_64
│   │   ├── verify_gpg_lnx.sh
│   │   └── verify_gpg_lnx.sh.sig
│   └── SIG
│   └── DP_DEPOT
│   ├── OB2-CFP-A.09.00-1.x86_64.rpm.sig
│   ├── OB2-CORE-IS-A.09.00-1.x86_64.rpm.sig
│   ├── OB2-DA-A.09.00-1.x86_64.rpm.sig
│   ├── OB2-DAP-A.09.00-1.x86_64.rpm.sig
│   ├── OB2-TS-CFP-A.09.00-1.x86_64.rpm.sig
│   ├── OB2-TS-CORE-A.09.00-1.x86_64.rpm.sig
│   └── OB2-TS-CS-A.09.00-1.x86_64.rpm.sig
├── LOCAL_INSTALL
│   └── omnisetup.sh
├── Readme.txt

Manual installation:

apt-get install -y liblua5.1-0 xinetd rpm2cpio rpm 
sudo ./omnisetup.sh -server myserver.myorg.es -install da

Reconfigure xinetd to serve omni

cat /etc/xinetd.d/dataprotector 

omni stream tcp nowait root /opt/omni/lbin/inet inet -log /var/opt/omni/log/inet.log

Puppet installation

I am proud to present my own dataprotectoragent puppet module to avoid manual installation of this agent

puppet module install esterniclos-dataprotector agent

class {"dataprotectoragent":
      dataprotectorserver => "dataprotector.my.com",
    }

Available from

Troubleshooting

During this project, I encountered The package needs to be reinstalled, but I can’t find an archive for it error, and, luckily,  IhaveaPC awesome tutorial to get rid of it.

Instalación de host vmware esxi 6

Nuestra infraestructura está montada por blades HP con ilo. A traves de la ILO, montamos la imagen de ESX.

esx6.install004

El arranque desde CD:

 

esx6.install005

F11 para aceptar la licencia

esx6.install002

Hemos optado por unos blades sin discos, así que usamos las opciones de san-boot con una lun de 6GB para el propio sistema operativo

esx6.install006

esx6.install001

Configuración de red

Los esxi salen con una configuración dhcp, que, si no tenemos en nuestra infraestructura, tendremos que reconfigurar a Ips estáticas

esx6.config001

esx6.config002

VLAN:

esx6.config004Ipv4

esx6.config005

Servidores DNS

esx6.config006

Sufijo DNS

esx6.config007

Añadir al inventario vcenter

En nuestro caso, es una reinstalación de un servidor. Vamos a mantener su nombre, así que en la reconexión con el inventario, tendremos un error ssl.

esx6.config003

Stage patches

Usamos  vmware update manager

esx6.vum.001Otro reinicio…

Drivers

Últimamente, tras un cambio en cabinas, estamos teniendo muchos problemas con los drivers de los blades.

A pesar de que usamos la versión de ESX que paquetiza HP, tenemos que instalar manualmente, vía ssh los drivers:

esxcli software vib install -v  /vmfs/volumes/..../lpfc-10.5.70.0-1OEM.600.0.0.21
59203.x86_64.vib

Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: EMU_bootbank_lpfc_10.5.70.0-1OEM.600.0.0.2159203
   VIBs Removed: Emulex_bootbank_lpfc_10.2.340.18-1OEM.550.0.0.1331820
   VIBs Skipped:

Host profile

Aplicamos el host profile que guarda toda la información de redes, configuraciones ntp, snmp…

Este es ya el último reinicio y el servidor está preparado para coger carga.

Vmware 6 ESXi host installation

Mount Iso image through Ilo. In this case, we are installing on a HP blade, so we are using ILO device to mount virtual CD.

esx6.install004

Boot from CD and run installer.

 

esx6.install005

F11 to accept license agreement

esx6.install002

Select disk to install. I our case, we use a 6 GB datastore to boot from disk. This allows us to change easily blades and keep server configuration untouched.

BTW, this blades were cheaper to buy as they have no disks.
esx6.install006

Wait for installer to end

esx6.install001

Boot.

First boot configuration

Vmware Esxi 6.0 tries to auto configure network.

esx6.config001

If DHCP is not present, static network settings have to be configured manually.

esx6.config002

VLAN:

esx6.config004Ipv4

esx6.config005

DNS server

esx6.config006

DNS suffix

esx6.config007

Add to vcenter inventory

If its a reinstallation, ssl certificates will have changes and a warning will show up.

esx6.config003

Stage patches

Using vmware update manager

esx6.vum.001This will lead to a server reboot.

Extra Drivers

For some reason, we have still to add HP drivers for our servers. Via ssh

esxcli software vib install -v  /vmfs/volumes/..../lpfc-10.5.70.0-1OEM.600.0.0.21
59203.x86_64.vib

Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: EMU_bootbank_lpfc_10.5.70.0-1OEM.600.0.0.2159203
   VIBs Removed: Emulex_bootbank_lpfc_10.2.340.18-1OEM.550.0.0.1331820
   VIBs Skipped:

Apply host profile

Last reboot and host is ready to go!

Replace Github-linguist with source github version

First of all, we need to locate our github-linguist files which will be replaced. We will also need ruby-build and gcc.

find / -name languages.yml
apt-get install ruby-build gcc
ruby-build 2.1.0 /opt/gitlab/embedded/service/gem/ruby/2.1.0
cd /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/
wget https://github.com/github/linguist/archive/v4.7.5.tar.gz
tar -xvzf v4.7.5.tar.gz
mv linguist-4.7.5/ github-linguist-4.7.5
rm -f v4.7.5.tar.gz
# Install all ruby dependencies
bundle install

Generate samples.json:

After download from github,

bundle exec rake check_samples

Otherwise, we will face this error in logs

2016-03-30_07:49:38.08294 No such file or directory @ rb_sysopen - /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/github-linguist-4.7.5/lib/linguist/samples.json

Fetch all ace modes

bundle exec rake fetch_ace_modes

Problems with bundle install

gem install byebug -v '8.2.2'
mkmf.rb can't find header files for ruby at /usr/lib/ruby/include/ruby.h

Solved

apt-get install ruby-dev

When running bundle install, charlock_holmes would not be installed

gem install charlock_holmes -v '0.7.3'
(..)
checking for main() in -licui18n... no

Solved

apt-get install libicu-dev

We also had problems with rugged

gem install rugged -v '0.24.0'
(..)
checking for gmake... no
checking for cmake... no
checking for pkg-config... no
ERROR: CMake is required to build Rugged.

Solved

apt-get install cmake pkg-config
ln -s /usr/bin/make /usr/bin/gmake

 

 

Starting with R

Installation of R and R-studio in windows
  • https://cran.rediris.es/bin/windows/base/R-3.2.4revised-win.exe
  • https://download1.rstudio.org/RStudio-0.99.893.exe

Rconsole:

r.console.001

Proxy settings are configured at invocation command. I will use cygwin to start it:

 /cygdrive/c/Program\ Files/R/R-3.2.4revised/bin/x64/Rgui.exe  http_proxy=http:/localhost:8888/

You can also load values from internet explorer

 /cygdrive/c/Program\ Files/R/R-3.2.4revised/bin/x64/Rgui.exe  --internet2

Rconsole in linux

apt-get install r-recommended libxml2-dev  libcurl4-openssl-dev libssl-dev

wget https://download1.rstudio.org/rstudio-0.99.893-amd64.deb
sudo dpkg -i rstudio-0.99.893-amd64.deb

To Find which session file is freezing rstudio:

strace -f rstudio 2>&1 | grep open | grep home

Help. Sistema de ayuda

Online documentation for every package and function: http://www.rdocumentation.org/

help ("read.table")
?

Vignette the command used to open pdf help docs.

Installing new packages. Instalación de paquetes

To install new packages and dependencies

install.packages("curl", dependencies = TRUE)

To install more than one package at the same time, we can use C to make a vector of packages:

install.packages(c("curl","swirl","httr"), dependencies = TRUE)

Swirl package

Swirl()
Select_language()

Selección_001

Using and declaring variables.

myval <- 12

R simplest object  is a vector, it can be declared using c (combine)

myval <- c(12,21,2)

Variables value can be printed using print function or just their names:

print(myval) 
myval

To check the objects declared in a session

ls ()

Vectors

Defining a vector with a certain length and type

vector ("numeric", length = 23)

Create a vector with a repeating pattern

(rep(c(TRUE,FALSE),10))
 [1]  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE
[20] FALSE

Using a vector to index another vector.Return positions that are multiple of 4:

my_vector[c(FALSE,FALSE,FALSE,TRUE)]

Dates. Fechas

Great tutorial to work with dates: http://www.stat.berkeley.edu/~s133/dates.html

today <- Sys.Date()

Load data from a file

mydata <- read.table("mycommaseparatedvaluefile.csv", header=TRUE, sep=",", fileEncoding = "latin1")
view (Data)

 

sdf

Introducción a R

Instalación de R y R-studio en windows
  • https://cran.rediris.es/bin/windows/base/R-3.2.4revised-win.exe
  • https://download1.rstudio.org/RStudio-0.99.893.exe

 

r.console.001

Para la instalación de paquetes, es necesario configurar el proxy. En el ejemplo, se redirige el proxy a un puerto local con fiddler.

 /cygdrive/c/Program\ Files/R/R-3.2.4revised/bin/x64/Rgui.exe  http_proxy=http:/localhost:8888/

También se pueden recoger los valores desde Internet Explorer.

 /cygdrive/c/Program\ Files/R/R-3.2.4revised/bin/x64/Rgui.exe  --internet2

Instalación en linux. Debian 8.

apt-get install r-recommended libxml2-dev  libcurl4-openssl-dev libssl-dev

wget https://download1.rstudio.org/rstudio-0.99.893-amd64.deb
sudo dpkg -i rstudio-0.99.893-amd64.deb

Sistema de ayuda

http://www.rdocumentation.org/

help ("read.table")
?

También existe el comando Vignette para abrir los pdfs de ayuda que vienen integrados.

Instalación de paquetes nuevos

Instalación de nuevos paquetes con dependencias:

install.packages("curl", dependencies = TRUE)

Para instalar más de un paquete al mismo tiempo, se puede usar c (Combinar) con una lista de nombres de paquetes:

install.packages(c("curl","swirl","httr"), dependencies = TRUE)

Swirl package

Swirl()
Select_language()

Selección_001

Declaración de variables.

myval <- 12

Casi todos los datos simples que se van a usar, serán vectores.

myval <- c(12,21,2)

Para ver el valor, basta con usar print o poner el nombre del valor.

print(myval) 
myval

Los objetos existen en una memoria de sesión que se pueden consultar mediante el uso de ls.

ls ()

Vectores

Para definir un vector de tipo numérico con longitud determinada:

vector ("numeric", length = 23)

A los vectores se les puede asignar valores de inicio, con patrones repetitivos

(rep(c(TRUE,FALSE),10))
 [1]  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE
[20] FALSE

También se puede usar un vector para acceder a los valores de dentro del propio vector. En el ejemplo, se imprimirán por pantalla los valores que estén en posiciones múltiplos de 4.

my_vector[c(FALSE,FALSE,FALSE,TRUE)]

Fechas

http://www.stat.berkeley.edu/~s133/dates.html

today <- Sys.Date()

Leer un archivo:

mydata <- read.table("mycommaseparatedvaluefile.csv", header=TRUE, sep=",", fileEncoding = "latin1")
view (Data)