Notes on MonetDB

MonetDB download

(Below I used Ubuntu 13.04 and MonetDB v11.15.7 (Feb2013-SP2)

Installation

  1. Create file /etc/apt/sources.list.d/monetdb.list with the following content
    deb http://dev.monetdb.org/downloads/deb/ quantal monetdb
    deb-src http://dev.monetdb.org/downloads/deb/ quantal monetdb
    
  2. Get the GPG public key by
    $ wget --output-document=- http://dev.monetdb.org/downloads/MonetDB-GPG-KEY | sudo apt-key add -
    
  3. Install the MonetDB
    $ sudo apt-get install monetdb5-sql monetdb-client
    
    A pseudo user monetdb will be created.

    The following .deb files are downloaded and installed

    libmonetdb-stream3
    libmonetdb-client6
    libmonetdb9
    monetdb-client
    monetdb5-server
    monetdb5-sql
    
  4. Make sure the pseudo user monetdb can write to the directory /var/lib/monetdb by
    $ sudo chown monetdb:monetdb /var/lib/monetdb
    

MonetDB executable binaries

/usr/bin/mclient
/usr/bin/monetdb
/usr/bin/monetdbd
/usr/bin/mserver5
/usr/bin/msqldump

MonetDB concepts

MonetDB operates on "dbfarms". A dbfarm is actually just a directory where the actual database will be stored.

A dbfarm is managed by the MonetDB daemon process monetdbd, and each individual database in a dbfarm is managed by MonetDB server process mserver5.

Create/Configure/Start/Shutdown a MonetDB dbfarm

Let's say a dbfarm will be located at /data/monetdb
  • Create: monetdbd create /data/monetdb
  • Start: monetdbd start /data/monetdb

    The MonetDB daemon process monetdbd by default will listen at port 50000

  • Show parameters: monetdbd get all /data/monetdb
  • Set port number to 54321: monetdbd set port=54321 /data/monetdb

  • Stop: monetdbd stop /data/monetdb
The dbfarm parameters are stored in /data/monetdb/.merovingian_properties

The dbfarm log file is /data/monetdb/merovingian.log

Create a MonetDB database

Suppose the MonetDB daemon process is already running and listening at port 50000. If not, see above steps to create a dbfarm and start the daemon process.

Suppose a database mydb is to be created and used. The steps are

  1. Create: monetdb create mydb

    A database named mydb will be created and be put in maintenance (locked) mode. A database in such a mode will not be started automatically by MonetDB daemon process monetdbd when a client requests connection.

    The database is located under /data/monetdb/mydb/

  2. Start: monetdb start mydb

  3. Unlock: monetdb release mydb

Process tree

monetdbd start /data/monetdb
 \_ /usr/bin/mserver5 --dbpath=/data/monetdb/mydb --set merovingian_uri mapi:monetdb://myhost:50000/mydb --set mapi_open false --set mapi_port 0 --set mapi_usock /data/monetdb/mydb/.mapi.sock --set monet_vault_key /data/monetdb/mydb/.vaultkey --set gdk_nr_threads 2 --set max_clients 64 --set sql_optimizer default_pipe --set monet_daemon yes

Other useful MonetDB commands

  • List all available databases: monetdb discover

  • Show status of all available databases: monetdb status

    The usual statuses of a database are Stopped, Running, or Locked.

  • Show status of mydb: monetdb status mydb

  • Show parameters of mydb: monetdb get all mydb

  • Stop mydb: monetdb stop mydb.

    Alternatively, one can use monetdb kill mydb, which is equivalent to sending a SIGKILL signal to the MonetDB server process mserver5.

Create schema and user in a MonetDB database

  1. Launch the MonetDB client: Use the default user monetdb and default password monetdb to log in
    $ mclient -u monetdb -d mydb
    
  2. Create a user foobar
    sql> CREATE USER "foobar" WITH PASSWORD 'foobar' NAME 'Foo Bar' SCHEMA "sys";
    
  3. Create a schema foobardb for user foobar
    sql> CREATE SCHEMA "foobardb" AUTHORIZATION "foobar";
    sql> ALTER USER "foobar" SET SCHEMA "foobardb";
    

Useful commands for MonetDB client

  • Describe table (or view) tbl: \d tbl

  • Describe schema sch: \dn sch

  • List all tables: \d

  • List all schemas: \dn

  • List all system objects: \dS

  • List all authorizations: SELECT * FROM AUTHS;

  • List database parameters: SELECT * FROM env() env;

Use DBVisualizer as a MonetDB client

  1. Download JDBC driver here and put it under DBVisualizer's JDBC directory, say C:\Program Files (x86)\DbVisualizer\jdbc\monetdb
  2. Launch DBVisualizer. It will find this new driver but it does not recognize it.
  3. Select Tool -> Driver Manager menu, add a new driver called MonetDB.

    In URL format, enter jdbc:monetdb://hostname/database

    In Driver File Paths, point to the JAR file under C:\Program Files (x86)\DbVisualizer\jdbc\monetdb

    Now the Driver Class pull-down menu should have nl.cwi.monetdb.jdbc.MonetDriver as an option. Select it.

    Close Driver Manager dialog box.

  4. Restart DBVisualizer. Create a new connection to MonetDB mydb

    In JDBC Driver, select MonetDB.

    In Database URL, enter jdbc:monetdb://localhost/mydb.

Migrate data from MySQL to MonetDB

  1. Dump the table, say t1 of database mydb, into a "|" separate file:
    mysql> USE mydb;
    mysql> SELECT * FROM t1 INTO OUTFILE '/tmp/t1.tbl' FIELDS TERMINATED BY '|';
    
  2. Create the table in MonetDB. One can show table schema in MySQL by
    mysql> SHOW CREATE TABLE t1;
    
    Note that MonetDB does not have indices, so remove all KEY clauses in the table scheme.

    DATETIME in MySQL is TIMESTAMP in MonetDB, FLOAT in MySQL is REAL in MonetDB.

    MonetDB uses single quotes for strings and double quotes for table or schema names.

  3. Once the table is in MonetDB, use COPY INTO syntax:
    $ mclient -u foobar -d foobardb -s "COPY INTO t1 FROM STDIN" - < /tmp/t1.tbl
    
    Note the minus sign - in the command line.
Tips about MonetDB bulk load here. An even faster bulk load is binary mode.

Notes on manual installation of Intel 64-bit Linux C/C++ compiler version 2013.4.183

To install Intel 64-bit Linux C/C++ compiler version 2013.4.183 manually, one needs to unpack the following:
  • intel-compilerpro-common-183-13.1-4.noarch.rpm
  • intel-compilerpro-devel-183-13.1-4.x86_64.rpm
  • intel-compilerproc-183-13.1-4.x86_64.rpm
  • intel-compilerproc-common-183-13.1-4.noarch.rpm
  • intel-compilerproc-devel-183-13.1-4.x86_64.rpm
  • intel-openmp-183-13.1-4.x86_64.rpm
  • intel-openmp-devel-183-13.1-4.x86_64.rpm

Notes on Calpont InfiniDB

Calpont InfiniDB Enterprise

Calpont InfiniDB Community Edition

(Below I assume Calpont InfiniDB Enterprise version 3.6.1, whose MySQL frontend is version 5.1.39)

Configuration

Suppose it is to be run by non-root user foobar and it is already installed under /opt/Calpont
  1. Create an entry in /etc/sudoer by root:
    foobar ALL = NOPASSWD: ALL
    
  2. Set up environment variables
    export INFINIDB_INSTALL_DIR=/opt/Calpont
    export LD_LIBRARY_PATH=$INFINIDB_INSTALL_DIR/lib:$INFINIDB_INSTALL_DIR/mysql/lib/mysql:$LD_LIBRARY_PATH
    source /opt/Calpont/bin/calpontAlias
    
  3. Under /opt/Calpont/lib, make sure symbolic links such as libbatchloader.so (linked to libbatchloader.so.1.0.0) are created.

    To check, execute ldd /opt/Calpont/bin/PrimProc

  4. Edit /opt/Calpont/mysql/my.cnf file.

    Set up socket file location, port number, datadir, max_allowed_packet, etc.

  5. Execute /opt/Calpont/bin/postConfigure

Start/Shutdown Server

  1. Set up environment variables
    export INFINIDB_INSTALL_DIR=/opt/Calpont
    export LD_LIBRARY_PATH=$INFINIDB_INSTALL_DIR/lib:$INFINIDB_INSTALL_DIR/mysql/lib/mysql:$LD_LIBRARY_PATH
    
  2. Start server: /opt/Calpont/bin/calpontConsole startSystem
  3. Stop server: /opt/Calpont/bin/calpontConsole shutdownSystem y

Command-line tools, log files

  • (MySQL) client

    /opt/Calpont/mysql/bin/mysql --defaults-file=/opt/Calpont/mysql/my.cnf -u root

  • Console

    /opt/Calpont/bin/calpontConsole

  • Debug logs

    tail -f /var/log/Calpont/debug.log

  • Info logs

    tail -f /var/log/Calpont/info.log

Process tree

Once Calpont InfiniDB is running (single-node mode), one should see the following process tree:
/bin/bash /opt/Calpont/bin/run.sh /opt/Calpont/bin/ProcMon
 \_ /opt/Calpont/bin/ProcMon
     \_ [ProcMgr]
     \_ /opt/Calpont/bin/controllernode fg
     \_ /opt/Calpont/bin/ServerMonitor
     \_ /opt/Calpont/bin/workernode DBRM_Worker1 fg
     \_ [PrimProc]
     \_ [ExeMgr]
     \_ [WriteEngineServ]
     \_ [DMLProc]
     \_ [DDLProc]
/bin/sh /opt/Calpont/mysql//bin/mysqld_safe --defaults-file=/opt/Calpont/mysql/my.cnf --datadir=....
 \_ /opt/Calpont/mysql/libexec/mysqld --defaults-file=/opt/Calpont/mysql//my.cnf --basedir=/opt/Calpont ...

Confluence Wiki Markup Langauge cheat sheet

Source: Confluence Wiki Notation Guide

Escape character \
Bold text *Bold text*
Italic text _Italic text_
Citation ??Citation??
Monospaced text {{Monospaced text}}
Single-line quote
bq. Single-line quote
Multiple-line
quote
{quote}
Multiple-line
quote
{quote}
    
Colored text {color:green}Colored text{color}
Underlined (inserted) text +Underlined (inserted) text+
Deleted text -Deleted text-
Superscript ^Superscript^
Subscript ~Subscript~

Heading size 1

.h1 Heading size 1
Heading size 5
.h5 Heading size 5
  • Bulleted list item 1
  • Bulleted list item 2
* Bulleted list item 1
* Bulleted list item 2
    
Note the space between * and the text following

* can be replaced by -

  1. Numbered list item 1
  2. Numbered list item 2
# Numbered list item 1
# Numbered list item 2
    
Note the space between # and the text following
  • List item 1
    1. Nested list item 1
    2. Nested list item 2
  • List item 2
* List item 1
*# Nested list item 1
*# Nested list item 2
* List item 2
    
Horizontal
rule
Horizontal
----
rule
    
(Long dash symbol) — ---
(Short dash symbol) – --
External link [External link|http://www.microsoft.com]
http://www.microsoft.com [http://www.microsoft.com]
Jump to my anchor
[#myAnchor]

[Jump to my anchor|#myAnchor]
    
External image !https://www.microsoft.com/logo.png!
Attached image "violin.png" !violin.jpg!
Attached image "violin.png" from "gallery macro" page !gallery macro^violin.jpg!
Image attributes !https://www.microsoft.com/logo.png|align=right, width=500!
Emoticons See here for a complete list.
Code block with syntax highlighting For XML:
{code:xml}
<test>
<another tag="attribute"/>
</test>
{code}
    

For source code:
{code:language=java|title=Bar.java|borderStyle=solid}
public String getFoo()
{ return foo; }
{code}
    
xml can be replaced by Java (the default), JavaScript, ActionScript, HTML, SQL (see here for a complete list)
Note/Info/Tip/Warning block
{note}Be careful{note}

{tip}Ask a question!{tip}

{warning:title=Don't Panic}Be happy.{warning}
	
{info:title=Don't Panic|icon=false}Be happy.{info}
	
Noformat block
{noformat}I do not want this text formatted!{noformat}
	
Charting See here for details.

"features" in ARM Linux's /proc/cpuinfo

Ever wonder what these "features" (hardware capabilities) in ARM Linux's /proc/cpuinfo mean ? Here is the answer (from Linux kernel source file arch/arm/include/asm/hwcap.h):

26bit Program Counter is 26-bit long
crunch MaverickCrunch instructoin extension. Now obsolete.
edsp DSP Enhancement instructions
fastmult Fast multiplication
fpa Floating Point Accelerator instructions. Now obsolete.
half ?
idiv Integer divide instructions available in both ARM and Thumb
idiva Integer divide instructions available in ARM only
idivt Integer divide instructions available in Thumb only
iwmmxt Intel Wireless MMX Technology, an implementation of MMX on embedded processors.
java Jazelle technology
neon NEON SIMD instructions
swp SWP (SWaP) instruction, which is used to implement a binary semaphore (mutex)
thumb Thumb instructions
thumbee Thumb Execution Environment
tls Thread local storage
vfp Vector Floating Point instruction extension
vfpv3 Vector Floating Point instruction extension version 3, with 32 double-precision registers
vfpv3d16 Vector Floating Point instruction extension version 3, with 16 double-precision registers
vfpv4 Vector Floating Point instruction extension version 4
Also, check this link.

Notes on ARM Cortex-A processors

Useful references

ARM Cortex-A series

Brawny cores still beat wimpy cores, most of the time by Urs Holzle of Google

ARM versus x86 performance comparison

12-core ARM cluster benchmarked against Intel Atom, Ivy Bridge, AMD Fusion

ARM (Nvidia Tegra 2/3, Samsung Exynos 5) vs. & Intel Core i7 performance & power consumption comparison

ARM Cortex-A family architecture by Hiroshige Goto

ARM Cortex-A series

Architecture Instruction set Extra features Clock rate (GHz) Cores Implementations
Cortex-A5 ARM v7, Jazelle, Thumb-2 DSP, VFPv3 floating point (optional), NEON SIMD (optional)
In-order, 9-stage single-decode pipeline
0.53-1 1-4
Cortex-A7 ARM v7, Jazelle, Thumb-2 DSP, VFPv4 floating point, NEON SIMD, virtualization
In-order, 8-stage limited dual-decode pipeline
> 1 1-4
Cortex-A8 ARM v7, Thumb-2 VFPv3 floating point, NEON SIMD 0.53-1 1 Apple A4, Freescale I.MX5?, Samsung Hummingbird, TI OMAP3
In-order, 13-stage dual-decode pipeline
Cortex-A9 ARM v7, Jazelle, Thumb-2 DSP, VFPv3 floating point (optional), NEON SIMD (optional) 0.8-2 1-4 Apple A5/A6, nVidia Tegra 2, Samsung Exynos 4210, ST-Ericsson NovaThor U8500, TI OMAP4
Out-of-order, 9-12 stage dual-decode pipeline
Cortex-A15 ARM v7, Jazelle, Thumb-2 TrustZone, VFPv4 floating point, 40-bit Large Physical Address Extensions, Virtualization, L2 cache
Out-of-order, 15+ stages triple-decode pipeline
1-2.5 1-32 ST-Ericsson Nova A9600, TI OMAP5, Samsung Exynos5

ASCII codes

American Standard Code for Information Interchange (ASCII) codes

Dec Hex  Oct  Meaning

  0 0x00 \000  NUL Null character, Ctrl-@
  1 0x01 \001  SOH Start of Header, Ctrl-A
  2 0x02 \002  STX Start of Text, Ctrl-B
  3 0x03 \003  ETX End of Text, Ctrl-C
  4 0x04 \004  EOT End of Transmission, End-of-file, Ctrl-D
  5 0x05 \005  ENQ Enquiry, Ctrl-E
  6 0x06 \006  ACK Acknowledgment, Ctrl-F
  7 0x07 \007  BEL Bell, Ctrl-G, '\a'
  8 0x08 \010   BS Backspace, Ctrl-H, '\b'
  9 0x09 \011   HT Horizontal Tab, Ctrl-I, '\t'
 10 0x0A \012   LF Line Feed, Ctrl-J, '\n'
 11 0x0B \013   VT Vertical Tab, Ctrl-K, '\v'
 12 0x0C \014   FF Form Feed, Ctrl-L, '\f'
 13 0x0D \015   CR Carriage Return, Ctrl-M, '\r'
 14 0x0E \016   SO Shift Out, Ctrl-N
 15 0x0F \017   SI Shift In, Ctrl-O
 16 0x10 \020  DLE Data Link Escape, Ctrl-P
 17 0x11 \021  DC1 XON/Device Control 1, Ctrl-Q
 18 0x12 \022  DC2 Device Control 2
 19 0x13 \023  DC3 XOFF/Device Control 3, Ctrl-S
 20 0x14 \024  DC4 Device Control 4
 21 0x15 \025  NAK Negative Acknowledgement
 22 0x16 \026  SYN Synchronous Idle
 23 0x17 \027  ETB End of Transmission Block
 24 0x18 \030  CAN Cancel, Ctrl-X
 25 0x19 \031   EM End of Medium
 26 0x1A \032  SUB Substitute
 27 0x1B \033  ESC Escape, Ctrl-[, '\e'
 28 0x1C \034   FS File Separator, Ctrl+\
 29 0x1D \035   GS Group Separator
 30 0x1E \036   RS Record Separator
 31 0x1F \037   US Unit Separator
 32 0x20 \040   SP Space
 33 0x21 \041    ! Exclamation mark
 34 0x22 \042    " Double quote
 35 0x23 \043    # Number sign, pound, hash
 36 0x24 \044    $ Dollar sign
 37 0x25 \045    % Percent
 38 0x26 \046    & Ampersand
 39 0x27 \047    ' Single quote, apostrophe, prime
 40 0x28 \050    ( Left/opening parenthesis
 41 0x29 \051    ) Right/closing parenthesis
 42 0x2A \052    * Asterisk
 43 0x2B \053    + Plus
 44 0x2C \054    , Comma
 45 0x2D \055    - Minus, dash
 46 0x2E \056    . Dot, period, full stop
 47 0x2F \057    / Forward slash
 48 0x30 \060    0
 49 0x31 \061    1
 50 0x32 \062    2
 51 0x33 \063    3
 52 0x34 \064    4
 53 0x35 \065    5
 54 0x36 \066    6
 55 0x37 \067    7
 56 0x38 \070    8
 57 0x39 \071    9
 58 0x3A \072    : Colon
 59 0x3B \073    ; Semi-colon
 60 0x3C \074    < Less than
 61 0x3D \075    = Equal sign
 62 0x3E \076    > Greater than
 63 0x3F \077    ? Question mark
 64 0x40 \100    @ At sign
 65 0x41 \101    A
 66 0x42 \102    B
 67 0x43 \103    C
 68 0x44 \104    D
 69 0x45 \105    E
 70 0x46 \106    F
 71 0x47 \107    G
 72 0x48 \110    H
 73 0x49 \111    I
 74 0x4A \112    J
 75 0x4B \113    K
 76 0x4C \114    L
 77 0x4D \115    M
 78 0x4E \116    N
 79 0x4F \117    O
 80 0x50 \120    P
 81 0x51 \121    Q
 82 0x52 \122    R
 83 0x53 \123    S
 84 0x54 \124    T
 85 0x55 \125    U
 86 0x56 \126    V
 87 0x57 \127    W
 88 0x58 \130    X
 89 0x59 \131    Y
 90 0x5A \132    Z
 91 0x5B \133    [ Left/opening bracket, square brackets
 92 0x5C \134    \ Back slash
 93 0x5D \135    ] Right/closing bracket, square brackets
 94 0x5E \136    ^ Caret, circumflex
 95 0x5F \137    _ Underscore
 96 0x60 \140    ` Grave accent
 97 0x61 \141    a
 98 0x62 \142    b
 99 0x63 \143    c
100 0x64 \144    d
101 0x65 \145    e
102 0x66 \146    f
103 0x67 \147    g
104 0x68 \150    h
105 0x69 \151    i
106 0x6A \152    j
107 0x6B \153    k
108 0x6C \154    l
109 0x6D \155    m
110 0x6E \156    n
111 0x6F \157    o
112 0x70 \160    p
113 0x71 \161    q
114 0x72 \162    r
115 0x73 \163    s
116 0x74 \164    t
117 0x75 \165    u
118 0x76 \166    v
119 0x77 \167    w
120 0x78 \170    x
121 0x79 \171    y
122 0x7A \172    z
123 0x7B \173    { Left/opening brace, curly brackets
124 0x7C \174    | Vertical bar, pipe
125 0x7D \175    } Right/closing brace, curly brackets
126 0x7E \176    ~ Tilde
127 0x7F \177  DEL Delete

(The following are valid in HTML 4 ISO 8859-1 encoding)

160 0xA0 \240    Non-breaking space (&nbsp;)
161 0xA1 \241  ¡ Inverted exclamation mark
162 0xA2 \242  ¢ Cent sign
163 0xA3 \243  £ Pound sign
164 0xA4 \244  ¤ Currency sign
165 0xA5 \245  ¥ Yen sign
166 0xA6 \246  ¦ Broken vertical bar
167 0xA7 \247  § Section sign (&sect;)
168 0xA8 \250  ¨ Spacing diaeresis, umlaut
169 0xA9 \251  © Copyright sign (&copy;)
170 0xAA \252  ª Feminine ordinal indicator
171 0xAB \253  « Left double angle quotes (&laquo;)
172 0xAC \254  ¬ Logical NOT sign (&not;)
173 0xAD \255  ­  Soft hyphen
174 0xAE \256  ® Registered trade mark sign (&reg;)
175 0xAF \257  ¯ Spacing macron, overline
176 0xB0 \260  ° Degree sign (&deg;)
177 0xB1 \261  ± Plus-or-minus sign (&plusmn;)
178 0xB2 \262  ² Superscript two, squared (&sup2;)
179 0xB3 \263  ³ Superscript three, cubed (&sup3;)
180 0xB4 \264  ´ Acute accent, spacing acute
181 0xB5 \265  µ Micro sign (&micro;)
182 0xB6 \266  ¶ Pilcrow sign, paragraph sign
183 0xB7 \267  · Middle dot, Georgian comma
184 0xB8 \270  ¸ Spacing cedilla
185 0xB9 \271  ¹ Superscript one (&sup1;)
186 0xBA \272  º Masculine ordinal indicator
187 0xBB \273  » Right double angle quotes (&raquo;)
188 0xBC \274  ¼ Fraction one quarter (&frac14;)
189 0xBD \275  ½ Fraction one half (&frac12;)
190 0xBE \276  ¾ Fraction three quarters (&frac34;)
191 0xBF \277  ¿ Inverted question mark