mongodb getting crashed on server
Hi I am having problems with mongo db on my server. Mongo db is getting crashed in few days. I have added a shell script to reboot mongod service on server if crashed but I want to know what should I do to make sure it do not crash at all.
Also mongodb.log
file in "/var/log/mongodb"
is not showing the reason on crash. It show me connections closing and ending only.
I am adding few lines from the log file here:
2017-06-06T06:49:52.305+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35871 #279 (11 connections now open)
2017-06-06T06:50:39.846+0000 I NETWORK [conn278] end connection 127.0.0.1:35869 (10 connections now open)
2017-06-06T06:51:03.870+0000 I NETWORK [conn264] end connection 127.0.0.1:35819 (9 connections now open)
2017-06-06T06:51:05.048+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35875 #280 (10 connections now open)
2017-06-06T06:51:32.902+0000 I NETWORK [conn276] end connection 127.0.0.1:35865 (9 connections now open)
2017-06-06T06:52:13.954+0000 I NETWORK [conn266] end connection 127.0.0.1:35826 (8 connections now open)
2017-06-06T06:52:20.958+0000 I NETWORK [conn221] end connection 127.0.0.1:35457 (7 connections now open)
2017-06-06T06:52:22.432+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35882 #281 (8 connections now open)
2017-06-06T06:52:39.052+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35883 #282 (9 connections now open)
2017-06-06T06:52:44.981+0000 I NETWORK [conn268] end connection 127.0.0.1:35829 (8 connections now open)
2017-06-06T06:52:46.619+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35885 #283 (9 connections now open)
2017-06-06T06:53:03.509+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35886 #284 (10 connections now open)
2017-06-06T06:53:04.399+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35887 #285 (11 connections now open)
2017-06-06T06:53:06.160+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35888 #286 (12 connections now open)
2017-06-06T06:54:25.183+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35889 #287 (13 connections now open)
2017-06-06T06:54:55.284+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35890 #288 (14 connections now open)
2017-06-06T06:55:27.122+0000 I NETWORK [conn279] end connection 127.0.0.1:35871 (13 connections now open)
2017-06-06T06:55:28.117+0000 I NETWORK [conn269] end connection 127.0.0.1:35832 (12 connections now open)
2017-06-06T06:57:09.066+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35894 #289 (13 connections now open)
2017-06-06T06:57:09.092+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35895 #290 (14 connections now open)
2017-06-06T06:57:17.206+0000 I NETWORK [conn287] end connection 127.0.0.1:35889 (13 connections now open)
2017-06-06T06:57:24.214+0000 I NETWORK [conn281] end connection 127.0.0.1:35882 (12 connections now open)
2017-06-06T06:58:59.404+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35898 #291 (13 connections now open)
2017-06-06T06:59:17.301+0000 I NETWORK [conn277] end connection 127.0.0.1:35866 (12 connections now open)
2017-06-06T06:59:46.326+0000 I NETWORK [conn288] end connection 127.0.0.1:35890 (11 connections now open)
2017-06-06T06:59:49.328+0000 I NETWORK [conn274] end connection 127.0.0.1:35860 (10 connections now open)
2017-06-06T07:00:03.342+0000 I NETWORK [conn290] end connection 127.0.0.1:35895 (9 connections now open)
2017-06-06T07:05:46.650+0000 I NETWORK [conn275] end connection 127.0.0.1:35863 (8 connections now open)
2017-06-06T13:15:43.047+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35995 #292 (9 connections now open)
2017-06-06T13:25:08.760+0000 I NETWORK [conn292] end connection 127.0.0.1:35995 (8 connections now open)
2017-06-07T04:07:48.262+0000 I NETWORK [conn285] end connection 127.0.0.1:35887 (7 connections now open)
2017-06-07T06:00:43.026+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36231 #293 (8 connections now open)
2017-06-07T06:10:21.038+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36236 #294 (9 connections now open)
2017-06-07T06:12:08.906+0000 I NETWORK [conn293] end connection 127.0.0.1:36231 (8 connections now open)
2017-06-07T06:13:03.738+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36238 #295 (9 connections now open)
2017-06-07T06:13:13.005+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36239 #296 (10 connections now open)
2017-06-07T06:14:10.995+0000 I NETWORK [conn294] end connection 127.0.0.1:36236 (9 connections now open)
2017-06-07T06:15:02.038+0000 I NETWORK [conn291] end connection 127.0.0.1:35898 (8 connections now open)
2017-06-07T06:15:16.486+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36244 #297 (9 connections now open)
2017-06-07T06:15:26.680+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36245 #298 (10 connections now open)
2017-06-07T06:16:43.118+0000 I NETWORK [conn282] end connection 127.0.0.1:35883 (9 connections now open)
2017-06-07T06:17:07.667+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36248 #299 (10 connections now open)
2017-06-07T06:17:19.869+0000 I CONTROL [main] ***** SERVER RESTARTED *****
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] MongoDB starting : pid=26429 port=27017 dbpath=/var/lib/mongodb 64-bit host=ip-172-31-28-133
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] db version v3.2.6
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] allocator: tcmalloc
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] modules: none
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] build environment:
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] distmod: ubuntu1404
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] distarch: x86_64
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] target_arch: x86_64
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2017-06-07T06:17:19.963+0000 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-06-07T06:17:19.963+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.553+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2017-06-07T06:17:20.554+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongodb/diagnostic.data'
I have noticed that information is provided when the service is restarted. But I am unable to figure out the problem behind the crashes.
I suspect its a memory issue as google searches show memory errors. But since they are not recorded in the log file.
I have a t2.micro
instance on the AWS EC2 where mongo script is running.
How can I prevent mongo from crashing?
14.04 amazon-ec2 mongodb
|
show 1 more comment
Hi I am having problems with mongo db on my server. Mongo db is getting crashed in few days. I have added a shell script to reboot mongod service on server if crashed but I want to know what should I do to make sure it do not crash at all.
Also mongodb.log
file in "/var/log/mongodb"
is not showing the reason on crash. It show me connections closing and ending only.
I am adding few lines from the log file here:
2017-06-06T06:49:52.305+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35871 #279 (11 connections now open)
2017-06-06T06:50:39.846+0000 I NETWORK [conn278] end connection 127.0.0.1:35869 (10 connections now open)
2017-06-06T06:51:03.870+0000 I NETWORK [conn264] end connection 127.0.0.1:35819 (9 connections now open)
2017-06-06T06:51:05.048+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35875 #280 (10 connections now open)
2017-06-06T06:51:32.902+0000 I NETWORK [conn276] end connection 127.0.0.1:35865 (9 connections now open)
2017-06-06T06:52:13.954+0000 I NETWORK [conn266] end connection 127.0.0.1:35826 (8 connections now open)
2017-06-06T06:52:20.958+0000 I NETWORK [conn221] end connection 127.0.0.1:35457 (7 connections now open)
2017-06-06T06:52:22.432+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35882 #281 (8 connections now open)
2017-06-06T06:52:39.052+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35883 #282 (9 connections now open)
2017-06-06T06:52:44.981+0000 I NETWORK [conn268] end connection 127.0.0.1:35829 (8 connections now open)
2017-06-06T06:52:46.619+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35885 #283 (9 connections now open)
2017-06-06T06:53:03.509+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35886 #284 (10 connections now open)
2017-06-06T06:53:04.399+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35887 #285 (11 connections now open)
2017-06-06T06:53:06.160+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35888 #286 (12 connections now open)
2017-06-06T06:54:25.183+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35889 #287 (13 connections now open)
2017-06-06T06:54:55.284+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35890 #288 (14 connections now open)
2017-06-06T06:55:27.122+0000 I NETWORK [conn279] end connection 127.0.0.1:35871 (13 connections now open)
2017-06-06T06:55:28.117+0000 I NETWORK [conn269] end connection 127.0.0.1:35832 (12 connections now open)
2017-06-06T06:57:09.066+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35894 #289 (13 connections now open)
2017-06-06T06:57:09.092+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35895 #290 (14 connections now open)
2017-06-06T06:57:17.206+0000 I NETWORK [conn287] end connection 127.0.0.1:35889 (13 connections now open)
2017-06-06T06:57:24.214+0000 I NETWORK [conn281] end connection 127.0.0.1:35882 (12 connections now open)
2017-06-06T06:58:59.404+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35898 #291 (13 connections now open)
2017-06-06T06:59:17.301+0000 I NETWORK [conn277] end connection 127.0.0.1:35866 (12 connections now open)
2017-06-06T06:59:46.326+0000 I NETWORK [conn288] end connection 127.0.0.1:35890 (11 connections now open)
2017-06-06T06:59:49.328+0000 I NETWORK [conn274] end connection 127.0.0.1:35860 (10 connections now open)
2017-06-06T07:00:03.342+0000 I NETWORK [conn290] end connection 127.0.0.1:35895 (9 connections now open)
2017-06-06T07:05:46.650+0000 I NETWORK [conn275] end connection 127.0.0.1:35863 (8 connections now open)
2017-06-06T13:15:43.047+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35995 #292 (9 connections now open)
2017-06-06T13:25:08.760+0000 I NETWORK [conn292] end connection 127.0.0.1:35995 (8 connections now open)
2017-06-07T04:07:48.262+0000 I NETWORK [conn285] end connection 127.0.0.1:35887 (7 connections now open)
2017-06-07T06:00:43.026+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36231 #293 (8 connections now open)
2017-06-07T06:10:21.038+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36236 #294 (9 connections now open)
2017-06-07T06:12:08.906+0000 I NETWORK [conn293] end connection 127.0.0.1:36231 (8 connections now open)
2017-06-07T06:13:03.738+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36238 #295 (9 connections now open)
2017-06-07T06:13:13.005+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36239 #296 (10 connections now open)
2017-06-07T06:14:10.995+0000 I NETWORK [conn294] end connection 127.0.0.1:36236 (9 connections now open)
2017-06-07T06:15:02.038+0000 I NETWORK [conn291] end connection 127.0.0.1:35898 (8 connections now open)
2017-06-07T06:15:16.486+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36244 #297 (9 connections now open)
2017-06-07T06:15:26.680+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36245 #298 (10 connections now open)
2017-06-07T06:16:43.118+0000 I NETWORK [conn282] end connection 127.0.0.1:35883 (9 connections now open)
2017-06-07T06:17:07.667+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36248 #299 (10 connections now open)
2017-06-07T06:17:19.869+0000 I CONTROL [main] ***** SERVER RESTARTED *****
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] MongoDB starting : pid=26429 port=27017 dbpath=/var/lib/mongodb 64-bit host=ip-172-31-28-133
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] db version v3.2.6
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] allocator: tcmalloc
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] modules: none
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] build environment:
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] distmod: ubuntu1404
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] distarch: x86_64
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] target_arch: x86_64
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2017-06-07T06:17:19.963+0000 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-06-07T06:17:19.963+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.553+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2017-06-07T06:17:20.554+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongodb/diagnostic.data'
I have noticed that information is provided when the service is restarted. But I am unable to figure out the problem behind the crashes.
I suspect its a memory issue as google searches show memory errors. But since they are not recorded in the log file.
I have a t2.micro
instance on the AWS EC2 where mongo script is running.
How can I prevent mongo from crashing?
14.04 amazon-ec2 mongodb
How did you know it has crashed?
– Arun
Jun 8 '17 at 10:16
my website show connection error and when I reboot the mongod service then website start working again.
– Curious Developer
Jun 8 '17 at 10:51
Take a look at the/var/lib/mongodb/diagnostic.data
– Arun
Jun 8 '17 at 10:55
this folder have a lot of metric named files for example "metrics.2017-06-07T06-17-21Z-00000" containing encrypt data.
– Curious Developer
Jun 8 '17 at 11:17
no file named diagnostic.data?
– Arun
Jun 8 '17 at 11:21
|
show 1 more comment
Hi I am having problems with mongo db on my server. Mongo db is getting crashed in few days. I have added a shell script to reboot mongod service on server if crashed but I want to know what should I do to make sure it do not crash at all.
Also mongodb.log
file in "/var/log/mongodb"
is not showing the reason on crash. It show me connections closing and ending only.
I am adding few lines from the log file here:
2017-06-06T06:49:52.305+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35871 #279 (11 connections now open)
2017-06-06T06:50:39.846+0000 I NETWORK [conn278] end connection 127.0.0.1:35869 (10 connections now open)
2017-06-06T06:51:03.870+0000 I NETWORK [conn264] end connection 127.0.0.1:35819 (9 connections now open)
2017-06-06T06:51:05.048+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35875 #280 (10 connections now open)
2017-06-06T06:51:32.902+0000 I NETWORK [conn276] end connection 127.0.0.1:35865 (9 connections now open)
2017-06-06T06:52:13.954+0000 I NETWORK [conn266] end connection 127.0.0.1:35826 (8 connections now open)
2017-06-06T06:52:20.958+0000 I NETWORK [conn221] end connection 127.0.0.1:35457 (7 connections now open)
2017-06-06T06:52:22.432+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35882 #281 (8 connections now open)
2017-06-06T06:52:39.052+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35883 #282 (9 connections now open)
2017-06-06T06:52:44.981+0000 I NETWORK [conn268] end connection 127.0.0.1:35829 (8 connections now open)
2017-06-06T06:52:46.619+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35885 #283 (9 connections now open)
2017-06-06T06:53:03.509+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35886 #284 (10 connections now open)
2017-06-06T06:53:04.399+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35887 #285 (11 connections now open)
2017-06-06T06:53:06.160+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35888 #286 (12 connections now open)
2017-06-06T06:54:25.183+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35889 #287 (13 connections now open)
2017-06-06T06:54:55.284+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35890 #288 (14 connections now open)
2017-06-06T06:55:27.122+0000 I NETWORK [conn279] end connection 127.0.0.1:35871 (13 connections now open)
2017-06-06T06:55:28.117+0000 I NETWORK [conn269] end connection 127.0.0.1:35832 (12 connections now open)
2017-06-06T06:57:09.066+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35894 #289 (13 connections now open)
2017-06-06T06:57:09.092+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35895 #290 (14 connections now open)
2017-06-06T06:57:17.206+0000 I NETWORK [conn287] end connection 127.0.0.1:35889 (13 connections now open)
2017-06-06T06:57:24.214+0000 I NETWORK [conn281] end connection 127.0.0.1:35882 (12 connections now open)
2017-06-06T06:58:59.404+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35898 #291 (13 connections now open)
2017-06-06T06:59:17.301+0000 I NETWORK [conn277] end connection 127.0.0.1:35866 (12 connections now open)
2017-06-06T06:59:46.326+0000 I NETWORK [conn288] end connection 127.0.0.1:35890 (11 connections now open)
2017-06-06T06:59:49.328+0000 I NETWORK [conn274] end connection 127.0.0.1:35860 (10 connections now open)
2017-06-06T07:00:03.342+0000 I NETWORK [conn290] end connection 127.0.0.1:35895 (9 connections now open)
2017-06-06T07:05:46.650+0000 I NETWORK [conn275] end connection 127.0.0.1:35863 (8 connections now open)
2017-06-06T13:15:43.047+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35995 #292 (9 connections now open)
2017-06-06T13:25:08.760+0000 I NETWORK [conn292] end connection 127.0.0.1:35995 (8 connections now open)
2017-06-07T04:07:48.262+0000 I NETWORK [conn285] end connection 127.0.0.1:35887 (7 connections now open)
2017-06-07T06:00:43.026+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36231 #293 (8 connections now open)
2017-06-07T06:10:21.038+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36236 #294 (9 connections now open)
2017-06-07T06:12:08.906+0000 I NETWORK [conn293] end connection 127.0.0.1:36231 (8 connections now open)
2017-06-07T06:13:03.738+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36238 #295 (9 connections now open)
2017-06-07T06:13:13.005+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36239 #296 (10 connections now open)
2017-06-07T06:14:10.995+0000 I NETWORK [conn294] end connection 127.0.0.1:36236 (9 connections now open)
2017-06-07T06:15:02.038+0000 I NETWORK [conn291] end connection 127.0.0.1:35898 (8 connections now open)
2017-06-07T06:15:16.486+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36244 #297 (9 connections now open)
2017-06-07T06:15:26.680+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36245 #298 (10 connections now open)
2017-06-07T06:16:43.118+0000 I NETWORK [conn282] end connection 127.0.0.1:35883 (9 connections now open)
2017-06-07T06:17:07.667+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36248 #299 (10 connections now open)
2017-06-07T06:17:19.869+0000 I CONTROL [main] ***** SERVER RESTARTED *****
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] MongoDB starting : pid=26429 port=27017 dbpath=/var/lib/mongodb 64-bit host=ip-172-31-28-133
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] db version v3.2.6
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] allocator: tcmalloc
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] modules: none
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] build environment:
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] distmod: ubuntu1404
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] distarch: x86_64
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] target_arch: x86_64
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2017-06-07T06:17:19.963+0000 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-06-07T06:17:19.963+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.553+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2017-06-07T06:17:20.554+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongodb/diagnostic.data'
I have noticed that information is provided when the service is restarted. But I am unable to figure out the problem behind the crashes.
I suspect its a memory issue as google searches show memory errors. But since they are not recorded in the log file.
I have a t2.micro
instance on the AWS EC2 where mongo script is running.
How can I prevent mongo from crashing?
14.04 amazon-ec2 mongodb
Hi I am having problems with mongo db on my server. Mongo db is getting crashed in few days. I have added a shell script to reboot mongod service on server if crashed but I want to know what should I do to make sure it do not crash at all.
Also mongodb.log
file in "/var/log/mongodb"
is not showing the reason on crash. It show me connections closing and ending only.
I am adding few lines from the log file here:
2017-06-06T06:49:52.305+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35871 #279 (11 connections now open)
2017-06-06T06:50:39.846+0000 I NETWORK [conn278] end connection 127.0.0.1:35869 (10 connections now open)
2017-06-06T06:51:03.870+0000 I NETWORK [conn264] end connection 127.0.0.1:35819 (9 connections now open)
2017-06-06T06:51:05.048+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35875 #280 (10 connections now open)
2017-06-06T06:51:32.902+0000 I NETWORK [conn276] end connection 127.0.0.1:35865 (9 connections now open)
2017-06-06T06:52:13.954+0000 I NETWORK [conn266] end connection 127.0.0.1:35826 (8 connections now open)
2017-06-06T06:52:20.958+0000 I NETWORK [conn221] end connection 127.0.0.1:35457 (7 connections now open)
2017-06-06T06:52:22.432+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35882 #281 (8 connections now open)
2017-06-06T06:52:39.052+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35883 #282 (9 connections now open)
2017-06-06T06:52:44.981+0000 I NETWORK [conn268] end connection 127.0.0.1:35829 (8 connections now open)
2017-06-06T06:52:46.619+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35885 #283 (9 connections now open)
2017-06-06T06:53:03.509+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35886 #284 (10 connections now open)
2017-06-06T06:53:04.399+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35887 #285 (11 connections now open)
2017-06-06T06:53:06.160+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35888 #286 (12 connections now open)
2017-06-06T06:54:25.183+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35889 #287 (13 connections now open)
2017-06-06T06:54:55.284+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35890 #288 (14 connections now open)
2017-06-06T06:55:27.122+0000 I NETWORK [conn279] end connection 127.0.0.1:35871 (13 connections now open)
2017-06-06T06:55:28.117+0000 I NETWORK [conn269] end connection 127.0.0.1:35832 (12 connections now open)
2017-06-06T06:57:09.066+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35894 #289 (13 connections now open)
2017-06-06T06:57:09.092+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35895 #290 (14 connections now open)
2017-06-06T06:57:17.206+0000 I NETWORK [conn287] end connection 127.0.0.1:35889 (13 connections now open)
2017-06-06T06:57:24.214+0000 I NETWORK [conn281] end connection 127.0.0.1:35882 (12 connections now open)
2017-06-06T06:58:59.404+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35898 #291 (13 connections now open)
2017-06-06T06:59:17.301+0000 I NETWORK [conn277] end connection 127.0.0.1:35866 (12 connections now open)
2017-06-06T06:59:46.326+0000 I NETWORK [conn288] end connection 127.0.0.1:35890 (11 connections now open)
2017-06-06T06:59:49.328+0000 I NETWORK [conn274] end connection 127.0.0.1:35860 (10 connections now open)
2017-06-06T07:00:03.342+0000 I NETWORK [conn290] end connection 127.0.0.1:35895 (9 connections now open)
2017-06-06T07:05:46.650+0000 I NETWORK [conn275] end connection 127.0.0.1:35863 (8 connections now open)
2017-06-06T13:15:43.047+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35995 #292 (9 connections now open)
2017-06-06T13:25:08.760+0000 I NETWORK [conn292] end connection 127.0.0.1:35995 (8 connections now open)
2017-06-07T04:07:48.262+0000 I NETWORK [conn285] end connection 127.0.0.1:35887 (7 connections now open)
2017-06-07T06:00:43.026+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36231 #293 (8 connections now open)
2017-06-07T06:10:21.038+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36236 #294 (9 connections now open)
2017-06-07T06:12:08.906+0000 I NETWORK [conn293] end connection 127.0.0.1:36231 (8 connections now open)
2017-06-07T06:13:03.738+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36238 #295 (9 connections now open)
2017-06-07T06:13:13.005+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36239 #296 (10 connections now open)
2017-06-07T06:14:10.995+0000 I NETWORK [conn294] end connection 127.0.0.1:36236 (9 connections now open)
2017-06-07T06:15:02.038+0000 I NETWORK [conn291] end connection 127.0.0.1:35898 (8 connections now open)
2017-06-07T06:15:16.486+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36244 #297 (9 connections now open)
2017-06-07T06:15:26.680+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36245 #298 (10 connections now open)
2017-06-07T06:16:43.118+0000 I NETWORK [conn282] end connection 127.0.0.1:35883 (9 connections now open)
2017-06-07T06:17:07.667+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36248 #299 (10 connections now open)
2017-06-07T06:17:19.869+0000 I CONTROL [main] ***** SERVER RESTARTED *****
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] MongoDB starting : pid=26429 port=27017 dbpath=/var/lib/mongodb 64-bit host=ip-172-31-28-133
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] db version v3.2.6
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] allocator: tcmalloc
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] modules: none
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] build environment:
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] distmod: ubuntu1404
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] distarch: x86_64
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] target_arch: x86_64
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2017-06-07T06:17:19.963+0000 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-06-07T06:17:19.963+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.553+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2017-06-07T06:17:20.554+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongodb/diagnostic.data'
I have noticed that information is provided when the service is restarted. But I am unable to figure out the problem behind the crashes.
I suspect its a memory issue as google searches show memory errors. But since they are not recorded in the log file.
I have a t2.micro
instance on the AWS EC2 where mongo script is running.
How can I prevent mongo from crashing?
14.04 amazon-ec2 mongodb
14.04 amazon-ec2 mongodb
asked Jun 8 '17 at 10:07
Curious DeveloperCurious Developer
23118
23118
How did you know it has crashed?
– Arun
Jun 8 '17 at 10:16
my website show connection error and when I reboot the mongod service then website start working again.
– Curious Developer
Jun 8 '17 at 10:51
Take a look at the/var/lib/mongodb/diagnostic.data
– Arun
Jun 8 '17 at 10:55
this folder have a lot of metric named files for example "metrics.2017-06-07T06-17-21Z-00000" containing encrypt data.
– Curious Developer
Jun 8 '17 at 11:17
no file named diagnostic.data?
– Arun
Jun 8 '17 at 11:21
|
show 1 more comment
How did you know it has crashed?
– Arun
Jun 8 '17 at 10:16
my website show connection error and when I reboot the mongod service then website start working again.
– Curious Developer
Jun 8 '17 at 10:51
Take a look at the/var/lib/mongodb/diagnostic.data
– Arun
Jun 8 '17 at 10:55
this folder have a lot of metric named files for example "metrics.2017-06-07T06-17-21Z-00000" containing encrypt data.
– Curious Developer
Jun 8 '17 at 11:17
no file named diagnostic.data?
– Arun
Jun 8 '17 at 11:21
How did you know it has crashed?
– Arun
Jun 8 '17 at 10:16
How did you know it has crashed?
– Arun
Jun 8 '17 at 10:16
my website show connection error and when I reboot the mongod service then website start working again.
– Curious Developer
Jun 8 '17 at 10:51
my website show connection error and when I reboot the mongod service then website start working again.
– Curious Developer
Jun 8 '17 at 10:51
Take a look at the
/var/lib/mongodb/diagnostic.data
– Arun
Jun 8 '17 at 10:55
Take a look at the
/var/lib/mongodb/diagnostic.data
– Arun
Jun 8 '17 at 10:55
this folder have a lot of metric named files for example "metrics.2017-06-07T06-17-21Z-00000" containing encrypt data.
– Curious Developer
Jun 8 '17 at 11:17
this folder have a lot of metric named files for example "metrics.2017-06-07T06-17-21Z-00000" containing encrypt data.
– Curious Developer
Jun 8 '17 at 11:17
no file named diagnostic.data?
– Arun
Jun 8 '17 at 11:21
no file named diagnostic.data?
– Arun
Jun 8 '17 at 11:21
|
show 1 more comment
1 Answer
1
active
oldest
votes
If there was a problem, MongoDB would have printed an error message or at least indicated it was shutting down. When those messages do not exist it means that the process was hard-killed by something else on the server.I'd recommend checking your system logs to see if the Operating System decided to kill the process for some reason.
the very high levels of anon memory persist even through MongoDB restarts. This indicates that the memory issue is not being caused by MongoDB and instead by something else.
please refer this link
https://jira.mongodb.org/browse/SERVER-22000
It is recommended reserving servers for just MongoDB and allocating only one mongod process per server.
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "89"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f923330%2fmongodb-getting-crashed-on-server%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
If there was a problem, MongoDB would have printed an error message or at least indicated it was shutting down. When those messages do not exist it means that the process was hard-killed by something else on the server.I'd recommend checking your system logs to see if the Operating System decided to kill the process for some reason.
the very high levels of anon memory persist even through MongoDB restarts. This indicates that the memory issue is not being caused by MongoDB and instead by something else.
please refer this link
https://jira.mongodb.org/browse/SERVER-22000
It is recommended reserving servers for just MongoDB and allocating only one mongod process per server.
add a comment |
If there was a problem, MongoDB would have printed an error message or at least indicated it was shutting down. When those messages do not exist it means that the process was hard-killed by something else on the server.I'd recommend checking your system logs to see if the Operating System decided to kill the process for some reason.
the very high levels of anon memory persist even through MongoDB restarts. This indicates that the memory issue is not being caused by MongoDB and instead by something else.
please refer this link
https://jira.mongodb.org/browse/SERVER-22000
It is recommended reserving servers for just MongoDB and allocating only one mongod process per server.
add a comment |
If there was a problem, MongoDB would have printed an error message or at least indicated it was shutting down. When those messages do not exist it means that the process was hard-killed by something else on the server.I'd recommend checking your system logs to see if the Operating System decided to kill the process for some reason.
the very high levels of anon memory persist even through MongoDB restarts. This indicates that the memory issue is not being caused by MongoDB and instead by something else.
please refer this link
https://jira.mongodb.org/browse/SERVER-22000
It is recommended reserving servers for just MongoDB and allocating only one mongod process per server.
If there was a problem, MongoDB would have printed an error message or at least indicated it was shutting down. When those messages do not exist it means that the process was hard-killed by something else on the server.I'd recommend checking your system logs to see if the Operating System decided to kill the process for some reason.
the very high levels of anon memory persist even through MongoDB restarts. This indicates that the memory issue is not being caused by MongoDB and instead by something else.
please refer this link
https://jira.mongodb.org/browse/SERVER-22000
It is recommended reserving servers for just MongoDB and allocating only one mongod process per server.
answered Jan 5 at 5:21
MamtaMamta
1
1
add a comment |
add a comment |
Thanks for contributing an answer to Ask Ubuntu!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f923330%2fmongodb-getting-crashed-on-server%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
How did you know it has crashed?
– Arun
Jun 8 '17 at 10:16
my website show connection error and when I reboot the mongod service then website start working again.
– Curious Developer
Jun 8 '17 at 10:51
Take a look at the
/var/lib/mongodb/diagnostic.data
– Arun
Jun 8 '17 at 10:55
this folder have a lot of metric named files for example "metrics.2017-06-07T06-17-21Z-00000" containing encrypt data.
– Curious Developer
Jun 8 '17 at 11:17
no file named diagnostic.data?
– Arun
Jun 8 '17 at 11:21