jueves, 26 de septiembre de 2019

MongoDB sharding and replicas - my personal notes


Install latest mongo

    sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
    echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu $(lsb_release -cs)/mongodb-org/4.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb.list

    sudo apt-get update && sudo apt-get install -y mongodb-org-mongos mongodb-org-server mongodb-org-shell mongodb-org-tools
    pip install pymongo -U

Distributed Mongo setup:

Set one mongo as master and the rest just point to it, in this example is our master server. Depend of your hardware you may prepend next command before mongod

  • numactl –interleave=all

This commands should be executed only on master:

  • Start them all on MASTER, systemd files for master/slave will be at the end of the post

        mkdir -p /data/{config,}db
        /usr/bin/numactl --interleave=all /usr/bin/mongod --config /etc/mongod.conf --shardsvr --bind_ip --port 27017 --replSet rs0
        /usr/bin/numactl --interleave=all /usr/bin/mongod --configsvr --replSet cuckoo_config --bind_ip
        # if fails, you need to point to cuckoo_config/localhost:27019
        /usr/bin/mongos --configdb cuckoo_config/ --port 27020

  • initialize the replica set aka backup for config server

        mongo --host --port 27019 --eval "rs.initiate({_id: "cuckoo_config", configsvr: true, members: [{ _id: 0, host: "" }]})"

  • if you see: “No host described in new configuration 1 for replica set cuckoo_config maps to this node”,

    • Solution. rs.initialize()
  • To add extra config servers rs.add(“cuckoo_config/”)

  • This should be started on all nodes including master:

    mkdir -p /data/{config,}db
    /usr/bin/mongod --shardsvr --bind_ip --port 27017 --replSet rs0
    /usr/bin/mongos --configdb cuckoo_config/ --port 27020
    # enable data query from slaves
    mongo --eval 'rs.slaveOk()'

After execute that on each node, go to master and execute

mongo --port 27017 --eval "rs.initiate({_id: "rs0",version: 1,members: [
         { _id: 0, host : "", priority: 1 },
         { _id: 1, host : "", priority: 0.5},
         { _id: 2, host : "", priority: 0.5},
         { _id: 3, host : "", priority: 0.5},
         { _id: 4, host : "", priority: 0.5},

To add more clients, execute on master mongo server:

    mongos --configdb cuckoo_config/ --port 27020
    mongo --port 27020
    sh.addShard( "rs0/")
    sh.addShard( "rs0/")
    sh.addShard( "rs0/")
    sh.addShard( "rs0/")
    sh.addShard( "rs0/")

Where 192.168.1.(2,3,4,5) is our cuckoo slaves::

    mongo --port 27020
    use cuckoo
    db.analysis.ensureIndex ( {"_id": "hashed" } )
    db.calls.ensureIndex ( {"_id": "hashed" } )
    sh.shardCollection("cuckoo.analysis", { "_id": "hashed" })
    sh.shardCollection("cuckoo.calls", { "_id": "hashed" })

Convert standalone shards, to replica shards

To see stats on master:

    mongo --host --port 27020
  • Modify cuckoo reporting.conf [mongodb] to point all mongos in reporting.conf to
    host =
    port = 27020

To remove shard node:

  • To see all shards:`

    • db.adminCommand( { listShards: 1 } )
  • Then:

    • use admin
    • db.runCommand({removeShard: "SHARD_NAME_HERE"})
  • Errormsg - movePrimary may only be run against the admin database.

  • Solution:

    • db.runCommand( { movePrimary: “cuckoo”, to: “SHARD_NAME_HERE” })
    • db.runCommand({removeShard: “SHARD_NAME_HERE”})
  • Stop draining:

    • use config
    • db.shards.update({},{$unset:{draining:true}}, false, true)
  • https://docs.mongodb.com/manual/tutorial/remove-shards-from-cluster/

Replica set configuration/reconfig

cfg = rs.conf();
cfg.members[0].priority = 2;
rs.reconfig(cfg, {force:true});
* https://docs.mongodb.com/manual/reference/method/rs.reconfig/

To remove members of replica set

cfg = rs.conf()
cfg.members = [cfg.members[0]]
rs.reconfig(cfg, {force : true})

Replica states:

Number  Name    State Description
0   STARTUP Not yet an active member of any set. All members start up in this state. The mongod parses the replica set configuration document while in STARTUP.
1   PRIMARY The member in state primary is the only member that can accept write operations. Eligible to vote.
2   SECONDARY   A member in state secondary is replicating the data store. Eligible to vote.
3   RECOVERING  Members either perform startup self-checks, or transition from completing a rollback or resync. Eligible to vote.
5   STARTUP2    The member has joined the set and is running an initial sync. Eligible to vote.
6   UNKNOWN The member’s state, as seen from another member of the set, is not yet known.
7   ARBITER Arbiters do not replicate data and exist solely to participate in elections. Eligible to vote.
8   DOWN    The member, as seen from another member of the set, is unreachable.
9   ROLLBACK    
This member is actively performing a rollback. Eligible to vote. Data is not available for reads from this member.

Starting in version 4.2, MongoDB kills all in-progress user operations when a member enters the ROLLBACK state.

10  REMOVED This member was once in a replica set but was subsequently removed.

Debug the problems commands:

  • journalctl -xe
  • tail /var/log/mongodb/mongod.log

Errors and Solutions:

  • exception in initAndListen: 20 Attempted to create a lock file on a read-only directory: /var/lib/mongodb, terminating

    • SOLUTION: sudo chown mongodb:mongodb /var/lib/mongodb -R
  • exception in initAndListen: 98 Unable to lock file: /var/lib/mongodb/mongod.lock Resource temporarily unavailable. Is a mongod instance already running?, terminating

    • You have running instance, close it
  • permission problem:

    • sudo chown -R mongodb:mongodb /var/lib/mongodb/
    • sudo chmod -R 755 /var/lib/mongodb
  • Error - Unit mongod.service is masked

  • Solution: sudo systemctl unmask mongod

  • Error - WT_PANIC: WiredTiger library panic

  • Solution:

    • mongod –repair –dbpath /database/db –storageEngine wiredTiger
  • Error - HostUnreachable: Connection refused

  • Solution:

    • Check if the mongo is run on that server
  • Error - “E11000 duplicate key error collection: admin.system.version index: id dup key: { : "shardIdentity" }”,

  • Solution - remove /data on slave

  • Error:

    • “can’t add shard ‘X:27017’ because a local database ‘cuckoo’ exists in another shard0001”
  • Solution:

    • drop mongodb on shard
      • use DB_NAME
      • db.dropDatabase()
  • Error:

    • replSetReconfig should only be run on PRIMARY, but my state is REMOVED; use the "force" argument to override
  • Solution:

    • rs.reconfig(cfg, { force: true })
  • Error:

    • Either all host names in a replica set configuration must be localhost references, or none must be; found 1 out of 2
  • Solution:

Default Port Description

  • 27017
    • The default port for mongod and mongos instances. You can change this port with port or –port.
  • 27018
    • The default port for mongod when running with –shardsvr runtime operation or the shardsvr value for the clusterRole setting in a configuration file.
  • 27019
    • The default port for mongod when running with –configsvr runtime operation or the configsvr value for the clusterRole setting in a configuration file.

Systemd config files

see systemd folder - /etc/systemd/system/

systemctl daemon-reload &&
sudo systemctl restart mongo*
sudo systemctl status mongos
sudo systemctl status mongodb
sudo systemctl status mongodb_config # only master
sudo systemctl enable mongos && sudo systemctl enable mongosdb
sudo systemctl enable mongos_config # only master

mongod_config.service - only on master

# /etc/systemd/system/mongod_config.service
Description=High-performance, schema-free document-oriented database
After=network.target mongodb.service

ExecStartPre=/bin/mkdir -p /data/db
ExecStartPre=/bin/chown mongodb:mongodb /data/db -R
ExecStart=/usr/bin/numactl --interleave=all /usr/bin/mongod --quiet --configsvr --replSet cuckoo_config --bind_ip_all
ExecReload=/bin/kill -HUP
# enable on ramfs servers
# --wiredTigerCacheSizeGB=50



# /etc/systemd/system/mongodb.service
Description=High-performance, schema-free document-oriented database

ExecStartPre=/bin/mkdir -p /data/db
ExecStartPre=/bin/chown mongodb:mongodb /data/db -R
# https://www.tutorialspoint.com/mongodb/mongodb_replication.htm
ExecStart=/usr/bin/numactl --interleave=all /usr/bin/mongod --quiet --shardsvr --bind_ip_all --port 27017 --replSet rs0
ExecReload=/bin/kill -HUP $MAINPID
# enable on ramfs servers
# --wiredTigerCacheSizeGB=50



# /etc/systemd/system/mongos.service
Description=Mongo shard service
ExecStart=/usr/bin/mongos --configdb cuckoo_config/ --port 27020

Security and checklist

For more information see: