- Install latest mongo
- Distributed Mongo setup:
- Convert standalone shards, to replica shards
- To see stats on master:
- To remove shard node:
- Replica set configuration/reconfig
- To remove members of replica set
- Debug the problems commands:
- Errors and Solutions:
- Systemd config files
- Security and checklist
- For more information see:
Install latest mongo
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu $(lsb_release -cs)/mongodb-org/4.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb.list
sudo apt-get update && sudo apt-get install -y mongodb-org-mongos mongodb-org-server mongodb-org-shell mongodb-org-tools
pip install pymongo -U
Distributed Mongo setup:
Set one mongo as master and the rest just point to it, in this example 192.168.1.1 is our master server. Depend of your hardware you may prepend next command before mongod
- numactl –interleave=all
This commands should be executed only on master:
-
Start them all on MASTER, systemd files for master/slave will be at the end of the post
mkdir -p /data/{config,}db /usr/bin/numactl --interleave=all /usr/bin/mongod --config /etc/mongod.conf --shardsvr --bind_ip 0.0.0.0 --port 27017 --replSet rs0 /usr/bin/numactl --interleave=all /usr/bin/mongod --configsvr --replSet cuckoo_config --bind_ip 0.0.0.0 # if fails, you need to point to cuckoo_config/localhost:27019 /usr/bin/mongos --configdb cuckoo_config/192.168.1.1:27019 --port 27020 -
initialize the replica set aka backup for config server
mongo --host 127.0.0.1 --port 27019 --eval "rs.initiate({_id: "cuckoo_config", configsvr: true, members: [{ _id: 0, host: "192.168.1.1:27019" }]})" -
if you see: “No host described in new configuration 1 for replica set cuckoo_config maps to this node”,
- Solution.
rs.initialize()
- Solution.
-
To add extra config servers rs.add(“cuckoo_config/192.168.1.2:27017”)
-
This should be started on all nodes including master:
mkdir -p /data/{config,}db
/usr/bin/mongod --shardsvr --bind_ip 0.0.0.0 --port 27017 --replSet rs0
/usr/bin/mongos --configdb cuckoo_config/192.168.1.1:27019 --port 27020
# enable data query from slaves
mongo --eval 'rs.slaveOk()'
After execute that on each node, go to master and execute
mongo --port 27017 --eval "rs.initiate({_id: "rs0",version: 1,members: [
{ _id: 0, host : "192.168.1.1:27017", priority: 1 },
{ _id: 1, host : "192.168.1.2:27017", priority: 0.5},
{ _id: 2, host : "192.168.1.3:27017", priority: 0.5},
{ _id: 3, host : "192.168.1.4:27017", priority: 0.5},
{ _id: 4, host : "192.168.1.5:27017", priority: 0.5},
]})"
- In case of even number of nodes, one should be arbiterOnly: Ex:
- { _id: 4, host : “192.168.1.6:27017”, arbiterOnly: true}
- https://docs.mongodb.com/manual/tutorial/add-replica-set-arbiter/
To add more clients, execute on master mongo server:
mongos --configdb cuckoo_config/192.168.1.1:27019 --port 27020
mongo --port 27020
sh.addShard( "rs0/192.168.1.1:27017")
sh.addShard( "rs0/192.168.1.2:27017")
sh.addShard( "rs0/192.168.1.3:27017")
sh.addShard( "rs0/192.168.1.4:27017")
sh.addShard( "rs0/192.168.1.5:27017")
Where 192.168.1.(2,3,4,5) is our cuckoo slaves::
mongo --port 27020
use cuckoo
db.analysis.ensureIndex ( {"_id": "hashed" } )
db.calls.ensureIndex ( {"_id": "hashed" } )
sh.enableSharding("cuckoo")
sh.shardCollection("cuckoo.analysis", { "_id": "hashed" })
sh.shardCollection("cuckoo.calls", { "_id": "hashed" })
Convert standalone shards, to replica shards
To see stats on master:
mongo --host 127.0.0.1 --port 27020
sh.status()
- Modify cuckoo reporting.conf [mongodb] to point all mongos in reporting.conf to
host = 127.0.0.1 port = 27020
To remove shard node:
-
To see all shards:`
db.adminCommand( { listShards: 1 } )
-
Then:
use admindb.runCommand({removeShard: "SHARD_NAME_HERE"})
-
Errormsg - movePrimary may only be run against the admin database.
-
Solution:
- db.runCommand( { movePrimary: “cuckoo”, to: “SHARD_NAME_HERE” })
- db.runCommand({removeShard: “SHARD_NAME_HERE”})
-
Stop draining:
- use config
- db.shards.update({},{$unset:{draining:true}}, false, true)
-
https://docs.mongodb.com/manual/tutorial/remove-shards-from-cluster/
Replica set configuration/reconfig
cfg = rs.conf();
cfg.members[0].priority = 2;
rs.reconfig(cfg, {force:true});
To remove members of replica set
cfg = rs.conf()
cfg.members = [cfg.members[0]]
rs.reconfig(cfg, {force : true})
-
Example how to change host dirrection
cfg = rs.conf() cfg.members[0].host = "192.168.1.2:27017" rs.reconfig(cfg, {force : true}) -
https://docs.mongodb.com/manual/tutorial/reconfigure-replica-set-with-unavailable-members/
Replica states:
Number Name State Description 0 STARTUP Not yet an active member of any set. All members start up in this state. The mongod parses the replica set configuration document while in STARTUP. 1 PRIMARY The member in state primary is the only member that can accept write operations. Eligible to vote. 2 SECONDARY A member in state secondary is replicating the data store. Eligible to vote. 3 RECOVERING Members either perform startup self-checks, or transition from completing a rollback or resync. Eligible to vote. 5 STARTUP2 The member has joined the set and is running an initial sync. Eligible to vote. 6 UNKNOWN The member’s state, as seen from another member of the set, is not yet known. 7 ARBITER Arbiters do not replicate data and exist solely to participate in elections. Eligible to vote. 8 DOWN The member, as seen from another member of the set, is unreachable. 9 ROLLBACK This member is actively performing a rollback. Eligible to vote. Data is not available for reads from this member. Starting in version 4.2, MongoDB kills all in-progress user operations when a member enters the ROLLBACK state. 10 REMOVED This member was once in a replica set but was subsequently removed.
Debug the problems commands:
- journalctl -xe
- tail /var/log/mongodb/mongod.log
Errors and Solutions:
-
exception in initAndListen: 20 Attempted to create a lock file on a read-only directory: /var/lib/mongodb, terminating
- SOLUTION:
sudo chown mongodb:mongodb /var/lib/mongodb -R
- SOLUTION:
-
exception in initAndListen: 98 Unable to lock file: /var/lib/mongodb/mongod.lock Resource temporarily unavailable. Is a mongod instance already running?, terminating
- You have running instance, close it
-
permission problem:
sudo chown -R mongodb:mongodb /var/lib/mongodb/sudo chmod -R 755 /var/lib/mongodb
-
Error - Unit mongod.service is masked
-
Solution:
sudo systemctl unmask mongod -
Error - WT_PANIC: WiredTiger library panic
-
Solution:
- mongod –repair –dbpath /database/db –storageEngine wiredTiger
-
Error - HostUnreachable: Connection refused
-
Solution:
- Check if the mongo is run on that server
-
Error - “E11000 duplicate key error collection: admin.system.version index: id dup key: { : "shardIdentity" }”,
-
Solution - remove /data on slave
-
Error:
- “can’t add shard ‘X:27017’ because a local database ‘cuckoo’ exists in another shard0001”
-
Solution:
- drop mongodb on shard
- use DB_NAME
- db.dropDatabase()
- drop mongodb on shard
-
Error:
- replSetReconfig should only be run on PRIMARY, but my state is REMOVED; use the "force" argument to override
-
Solution:
- rs.reconfig(cfg, { force: true })
-
Error:
- Either all host names in a replica set configuration must be localhost references, or none must be; found 1 out of 2
- Solution:
- Set hostnames in /etc/hosts for all mongo nodes
- https://docs.mongodb.com/manual/tutorial/change-hostnames-in-a-replica-set/
Default Port Description
- 27017
- The default port for mongod and mongos instances. You can change this port with port or –port.
- 27018
- The default port for mongod when running with –shardsvr runtime operation or the shardsvr value for the clusterRole setting in a configuration file.
- 27019
- The default port for mongod when running with –configsvr runtime operation or the configsvr value for the clusterRole setting in a configuration file.
Systemd config files
see systemd folder - /etc/systemd/system/
systemctl daemon-reload && sudo systemctl restart mongo* sudo systemctl status mongos sudo systemctl status mongodb sudo systemctl status mongodb_config # only master sudo systemctl enable mongos && sudo systemctl enable mongosdb sudo systemctl enable mongos_config # only master
mongod_config.service - only on master
# /etc/systemd/system/mongod_config.service [Unit] Description=High-performance, schema-free document-oriented database After=network.target mongodb.service [Service] User=root ExecStartPre=/bin/mkdir -p /data/db ExecStartPre=/bin/chown mongodb:mongodb /data/db -R ExecStart=/usr/bin/numactl --interleave=all /usr/bin/mongod --quiet --configsvr --replSet cuckoo_config --bind_ip_all ExecReload=/bin/kill -HUP Restart=always # enable on ramfs servers # --wiredTigerCacheSizeGB=50 User=mongodb Group=mongodb StandardOutput=syslog StandardError=syslog SyslogIdentifier=mongodb_config [Install] WantedBy=multi-user.target
mongod.service
# /etc/systemd/system/mongodb.service [Unit] Description=High-performance, schema-free document-oriented database Wants=network.target After=network.target [Service] ExecStartPre=/bin/mkdir -p /data/db ExecStartPre=/bin/chown mongodb:mongodb /data/db -R # https://www.tutorialspoint.com/mongodb/mongodb_replication.htm ExecStart=/usr/bin/numactl --interleave=all /usr/bin/mongod --quiet --shardsvr --bind_ip_all --port 27017 --replSet rs0 ExecReload=/bin/kill -HUP $MAINPID Restart=always # enable on ramfs servers # --wiredTigerCacheSizeGB=50 #User=mongodb #Group=mongodb #StandardOutput=syslog #StandardError=syslog #SyslogIdentifier=mongodb [Install] WantedBy=multi-user.target
mongos.service
# /etc/systemd/system/mongos.service [Unit] Description=Mongo shard service After=network.target After=bind9.service [Service] PIDFile=/var/run/mongos.pid User=root ExecStart=/usr/bin/mongos --configdb cuckoo_config/192.168.1.1:27019 --port 27020 [Install] WantedBy=multi-user.target
Security and checklist
- https://docs.mongodb.com/manual/tutorial/enable-authentication/
- https://docs.mongodb.com/manual/administration/security-checklist/
- https://docs.mongodb.com/manual/core/security-users/#sharding-security
For more information see:
- https://docs.mongodb.com/manual/tutorial/remove-shards-from-cluster/
- http://dws.la/deploying-a-sharded-cluster-in-mongodb/
- https://www.simplilearn.com/replication-and-sharding-mongodb-tutorial-video
- https://dinfratechsource.com/2018/11/10/mongodb-essentials/
- https://www.tutorialspoint.com/mongodb/mongodb_sharding.htm
- https://eladnava.com/deploy-a-highly-available-mongodb-replica-set-on-aws/
