jueves, 26 de septiembre de 2019

MongoDB sharding and replicas - my personal notes

mongo_sharding_replicas

Install latest mongo

    sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
    echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu $(lsb_release -cs)/mongodb-org/4.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb.list

    sudo apt-get update && sudo apt-get install -y mongodb-org-mongos mongodb-org-server mongodb-org-shell mongodb-org-tools
    pip install pymongo -U

Distributed Mongo setup:

Set one mongo as master and the rest just point to it, in this example 192.168.1.1 is our master server. Depend of your hardware you may prepend next command before mongod

  • numactl –interleave=all

This commands should be executed only on master:

  • Start them all on MASTER, systemd files for master/slave will be at the end of the post 

        mkdir -p /data/{config,}db
    /usr/bin/numactl --interleave=all /usr/bin/mongod --config /etc/mongod.conf --shardsvr --bind_ip 0.0.0.0 --port 27017 --replSet rs0
    /usr/bin/numactl --interleave=all /usr/bin/mongod --configsvr --replSet cuckoo_config --bind_ip 0.0.0.0
    # if fails, you need to point to cuckoo_config/localhost:27019
    /usr/bin/mongos --configdb cuckoo_config/192.168.1.1:27019 --port 27020

  • initialize the replica set aka backup for config server 

        mongo --host 127.0.0.1 --port 27019 --eval "rs.initiate({_id: "cuckoo_config", configsvr: true, members: [{ _id: 0, host: "192.168.1.1:27019" }]})"

  • if you see: “No host described in new configuration 1 for replica set cuckoo_config maps to this node”,

    • Solution. rs.initialize()

  • To add extra config servers rs.add(“cuckoo_config/192.168.1.2:27017”)

  • This should be started on all nodes including master:

    mkdir -p /data/{config,}db
    /usr/bin/mongod --shardsvr --bind_ip 0.0.0.0 --port 27017 --replSet rs0
    /usr/bin/mongos --configdb cuckoo_config/192.168.1.1:27019 --port 27020
    # enable data query from slaves
    mongo --eval 'rs.slaveOk()'

After execute that on each node, go to master and execute 

mongo --port 27017 --eval "rs.initiate({_id: "rs0",version: 1,members: [
         { _id: 0, host : "192.168.1.1:27017", priority: 1 },
         { _id: 1, host : "192.168.1.2:27017", priority: 0.5},
         { _id: 2, host : "192.168.1.3:27017", priority: 0.5},
         { _id: 3, host : "192.168.1.4:27017", priority: 0.5},
         { _id: 4, host : "192.168.1.5:27017", priority: 0.5},
]})"

To add more clients, execute on master mongo server:

    mongos --configdb cuckoo_config/192.168.1.1:27019 --port 27020
    mongo --port 27020
    sh.addShard( "rs0/192.168.1.1:27017")
    sh.addShard( "rs0/192.168.1.2:27017")
    sh.addShard( "rs0/192.168.1.3:27017")
    sh.addShard( "rs0/192.168.1.4:27017")
    sh.addShard( "rs0/192.168.1.5:27017")

Where 192.168.1.(2,3,4,5) is our cuckoo slaves::

    mongo --port 27020
    use cuckoo
    db.analysis.ensureIndex ( {"_id": "hashed" } )
    db.calls.ensureIndex ( {"_id": "hashed" } )
    sh.enableSharding("cuckoo")
    sh.shardCollection("cuckoo.analysis", { "_id": "hashed" })
    sh.shardCollection("cuckoo.calls", { "_id": "hashed" })

Convert standalone shards, to replica shards

To see stats on master:

    mongo --host 127.0.0.1 --port 27020
    sh.status()
  • Modify cuckoo reporting.conf [mongodb] to point all mongos in reporting.conf to 
    host = 127.0.0.1
port = 27020

To remove shard node:

  • To see all shards:`

    • db.adminCommand( { listShards: 1 } )

  • Then:

    • use admin
    • db.runCommand({removeShard: "SHARD_NAME_HERE"})

  • Errormsg - movePrimary may only be run against the admin database.

  • Solution:

    • db.runCommand( { movePrimary: “cuckoo”, to: “SHARD_NAME_HERE” })
    • db.runCommand({removeShard: “SHARD_NAME_HERE”})

  • Stop draining:

    • use config
    • db.shards.update({},{$unset:{draining:true}}, false, true)

  • https://docs.mongodb.com/manual/tutorial/remove-shards-from-cluster/

Replica set configuration/reconfig

cfg = rs.conf();
cfg.members[0].priority = 2;
rs.reconfig(cfg, {force:true});
* https://docs.mongodb.com/manual/reference/method/rs.reconfig/

To remove members of replica set

cfg = rs.conf()
cfg.members = [cfg.members[0]]
rs.reconfig(cfg, {force : true})

Replica states:

Number  Name    State Description
0   STARTUP Not yet an active member of any set. All members start up in this state. The mongod parses the replica set configuration document while in STARTUP.
1   PRIMARY The member in state primary is the only member that can accept write operations. Eligible to vote.
2   SECONDARY   A member in state secondary is replicating the data store. Eligible to vote.
3   RECOVERING  Members either perform startup self-checks, or transition from completing a rollback or resync. Eligible to vote.
5   STARTUP2    The member has joined the set and is running an initial sync. Eligible to vote.
6   UNKNOWN The member’s state, as seen from another member of the set, is not yet known.
7   ARBITER Arbiters do not replicate data and exist solely to participate in elections. Eligible to vote.
8   DOWN    The member, as seen from another member of the set, is unreachable.
9   ROLLBACK    
This member is actively performing a rollback. Eligible to vote. Data is not available for reads from this member.

Starting in version 4.2, MongoDB kills all in-progress user operations when a member enters the ROLLBACK state.

10  REMOVED This member was once in a replica set but was subsequently removed.

Debug the problems commands:

  • journalctl -xe
  • tail /var/log/mongodb/mongod.log

Errors and Solutions:

  • exception in initAndListen: 20 Attempted to create a lock file on a read-only directory: /var/lib/mongodb, terminating

    • SOLUTION: sudo chown mongodb:mongodb /var/lib/mongodb -R

  • exception in initAndListen: 98 Unable to lock file: /var/lib/mongodb/mongod.lock Resource temporarily unavailable. Is a mongod instance already running?, terminating

    • You have running instance, close it

  • permission problem:

    • sudo chown -R mongodb:mongodb /var/lib/mongodb/
    • sudo chmod -R 755 /var/lib/mongodb

  • Error - Unit mongod.service is masked

  • Solution: sudo systemctl unmask mongod

  • Error - WT_PANIC: WiredTiger library panic

  • Solution:

    • mongod –repair –dbpath /database/db –storageEngine wiredTiger

  • Error - HostUnreachable: Connection refused

  • Solution:

    • Check if the mongo is run on that server

  • Error - “E11000 duplicate key error collection: admin.system.version index: id dup key: { : "shardIdentity" }”,

  • Solution - remove /data on slave

  • Error:

    • “can’t add shard ‘X:27017’ because a local database ‘cuckoo’ exists in another shard0001”

  • Solution:

    • drop mongodb on shard
      • use DB_NAME
      • db.dropDatabase()

  • Error:

    • replSetReconfig should only be run on PRIMARY, but my state is REMOVED; use the "force" argument to override

  • Solution:

    • rs.reconfig(cfg, { force: true })

  • Error:

    • Either all host names in a replica set configuration must be localhost references, or none must be; found 1 out of 2
  • Solution:

Default Port Description

  • 27017
    • The default port for mongod and mongos instances. You can change this port with port or –port.
  • 27018
    • The default port for mongod when running with –shardsvr runtime operation or the shardsvr value for the clusterRole setting in a configuration file.
  • 27019
    • The default port for mongod when running with –configsvr runtime operation or the configsvr value for the clusterRole setting in a configuration file.

Systemd config files

see systemd folder - /etc/systemd/system/ 

systemctl daemon-reload &&
sudo systemctl restart mongo*
sudo systemctl status mongos
sudo systemctl status mongodb
sudo systemctl status mongodb_config # only master
sudo systemctl enable mongos && sudo systemctl enable mongosdb
sudo systemctl enable mongos_config # only master

mongod_config.service - only on master

# /etc/systemd/system/mongod_config.service
[Unit]
Description=High-performance, schema-free document-oriented database
After=network.target mongodb.service

[Service]
User=root
ExecStartPre=/bin/mkdir -p /data/db
ExecStartPre=/bin/chown mongodb:mongodb /data/db -R
ExecStart=/usr/bin/numactl --interleave=all /usr/bin/mongod --quiet --configsvr --replSet cuckoo_config --bind_ip_all
ExecReload=/bin/kill -HUP
Restart=always
# enable on ramfs servers
# --wiredTigerCacheSizeGB=50
User=mongodb
Group=mongodb
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=mongodb_config

[Install]
WantedBy=multi-user.target

mongod.service

# /etc/systemd/system/mongodb.service
[Unit]
Description=High-performance, schema-free document-oriented database
Wants=network.target
After=network.target

[Service]
ExecStartPre=/bin/mkdir -p /data/db
ExecStartPre=/bin/chown mongodb:mongodb /data/db -R
# https://www.tutorialspoint.com/mongodb/mongodb_replication.htm
ExecStart=/usr/bin/numactl --interleave=all /usr/bin/mongod --quiet --shardsvr --bind_ip_all --port 27017 --replSet rs0
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
# enable on ramfs servers
# --wiredTigerCacheSizeGB=50
#User=mongodb
#Group=mongodb
#StandardOutput=syslog
#StandardError=syslog
#SyslogIdentifier=mongodb

[Install]
WantedBy=multi-user.target

mongos.service

# /etc/systemd/system/mongos.service
[Unit]
Description=Mongo shard service
After=network.target
After=bind9.service
[Service]
PIDFile=/var/run/mongos.pid
User=root
ExecStart=/usr/bin/mongos --configdb cuckoo_config/192.168.1.1:27019 --port 27020
[Install]
WantedBy=multi-user.target

Security and checklist

For more information see:

Publicado por en
Etiquetas: , , , , , , ,

No hay comentarios:

Publicar un comentario

Suscribirse a: Enviar comentarios (Atom)