mongodb getting crashed on server












0















Hi I am having problems with mongo db on my server. Mongo db is getting crashed in few days. I have added a shell script to reboot mongod service on server if crashed but I want to know what should I do to make sure it do not crash at all.



Also mongodb.log file in "/var/log/mongodb"is not showing the reason on crash. It show me connections closing and ending only.



I am adding few lines from the log file here:



2017-06-06T06:49:52.305+0000 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:35871 #279 (11 connections now open)
2017-06-06T06:50:39.846+0000 I NETWORK [conn278] end connection 127.0.0.1:35869 (10 connections now open)
2017-06-06T06:51:03.870+0000 I NETWORK [conn264] end connection 127.0.0.1:35819 (9 connections now open)
2017-06-06T06:51:05.048+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35875 #280 (10 connections now open)
2017-06-06T06:51:32.902+0000 I NETWORK [conn276] end connection 127.0.0.1:35865 (9 connections now open)
2017-06-06T06:52:13.954+0000 I NETWORK [conn266] end connection 127.0.0.1:35826 (8 connections now open)
2017-06-06T06:52:20.958+0000 I NETWORK [conn221] end connection 127.0.0.1:35457 (7 connections now open)
2017-06-06T06:52:22.432+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35882 #281 (8 connections now open)
2017-06-06T06:52:39.052+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35883 #282 (9 connections now open)
2017-06-06T06:52:44.981+0000 I NETWORK [conn268] end connection 127.0.0.1:35829 (8 connections now open)
2017-06-06T06:52:46.619+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35885 #283 (9 connections now open)
2017-06-06T06:53:03.509+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35886 #284 (10 connections now open)
2017-06-06T06:53:04.399+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35887 #285 (11 connections now open)
2017-06-06T06:53:06.160+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35888 #286 (12 connections now open)
2017-06-06T06:54:25.183+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35889 #287 (13 connections now open)
2017-06-06T06:54:55.284+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35890 #288 (14 connections now open)
2017-06-06T06:55:27.122+0000 I NETWORK [conn279] end connection 127.0.0.1:35871 (13 connections now open)
2017-06-06T06:55:28.117+0000 I NETWORK [conn269] end connection 127.0.0.1:35832 (12 connections now open)
2017-06-06T06:57:09.066+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35894 #289 (13 connections now open)
2017-06-06T06:57:09.092+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35895 #290 (14 connections now open)
2017-06-06T06:57:17.206+0000 I NETWORK [conn287] end connection 127.0.0.1:35889 (13 connections now open)
2017-06-06T06:57:24.214+0000 I NETWORK [conn281] end connection 127.0.0.1:35882 (12 connections now open)
2017-06-06T06:58:59.404+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35898 #291 (13 connections now open)
2017-06-06T06:59:17.301+0000 I NETWORK [conn277] end connection 127.0.0.1:35866 (12 connections now open)
2017-06-06T06:59:46.326+0000 I NETWORK [conn288] end connection 127.0.0.1:35890 (11 connections now open)
2017-06-06T06:59:49.328+0000 I NETWORK [conn274] end connection 127.0.0.1:35860 (10 connections now open)
2017-06-06T07:00:03.342+0000 I NETWORK [conn290] end connection 127.0.0.1:35895 (9 connections now open)
2017-06-06T07:05:46.650+0000 I NETWORK [conn275] end connection 127.0.0.1:35863 (8 connections now open)
2017-06-06T13:15:43.047+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35995 #292 (9 connections now open)
2017-06-06T13:25:08.760+0000 I NETWORK [conn292] end connection 127.0.0.1:35995 (8 connections now open)
2017-06-07T04:07:48.262+0000 I NETWORK [conn285] end connection 127.0.0.1:35887 (7 connections now open)
2017-06-07T06:00:43.026+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36231 #293 (8 connections now open)
2017-06-07T06:10:21.038+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36236 #294 (9 connections now open)
2017-06-07T06:12:08.906+0000 I NETWORK [conn293] end connection 127.0.0.1:36231 (8 connections now open)
2017-06-07T06:13:03.738+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36238 #295 (9 connections now open)
2017-06-07T06:13:13.005+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36239 #296 (10 connections now open)
2017-06-07T06:14:10.995+0000 I NETWORK [conn294] end connection 127.0.0.1:36236 (9 connections now open)
2017-06-07T06:15:02.038+0000 I NETWORK [conn291] end connection 127.0.0.1:35898 (8 connections now open)
2017-06-07T06:15:16.486+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36244 #297 (9 connections now open)
2017-06-07T06:15:26.680+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36245 #298 (10 connections now open)
2017-06-07T06:16:43.118+0000 I NETWORK [conn282] end connection 127.0.0.1:35883 (9 connections now open)
2017-06-07T06:17:07.667+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36248 #299 (10 connections now open)
2017-06-07T06:17:19.869+0000 I CONTROL [main] ***** SERVER RESTARTED *****
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] MongoDB starting : pid=26429 port=27017 dbpath=/var/lib/mongodb 64-bit host=ip-172-31-28-133
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] db version v3.2.6
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] allocator: tcmalloc
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] modules: none
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] build environment:
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] distmod: ubuntu1404
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] distarch: x86_64
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] target_arch: x86_64
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2017-06-07T06:17:19.963+0000 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-06-07T06:17:19.963+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.553+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2017-06-07T06:17:20.554+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongodb/diagnostic.data'


I have noticed that information is provided when the service is restarted. But I am unable to figure out the problem behind the crashes.



I suspect its a memory issue as google searches show memory errors. But since they are not recorded in the log file.



I have a t2.micro instance on the AWS EC2 where mongo script is running.
How can I prevent mongo from crashing?










share|improve this question























  • How did you know it has crashed?

    – Arun
    Jun 8 '17 at 10:16











  • my website show connection error and when I reboot the mongod service then website start working again.

    – Curious Developer
    Jun 8 '17 at 10:51











  • Take a look at the /var/lib/mongodb/diagnostic.data

    – Arun
    Jun 8 '17 at 10:55













  • this folder have a lot of metric named files for example "metrics.2017-06-07T06-17-21Z-00000" containing encrypt data.

    – Curious Developer
    Jun 8 '17 at 11:17













  • no file named diagnostic.data?

    – Arun
    Jun 8 '17 at 11:21
















0















Hi I am having problems with mongo db on my server. Mongo db is getting crashed in few days. I have added a shell script to reboot mongod service on server if crashed but I want to know what should I do to make sure it do not crash at all.



Also mongodb.log file in "/var/log/mongodb"is not showing the reason on crash. It show me connections closing and ending only.



I am adding few lines from the log file here:



2017-06-06T06:49:52.305+0000 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:35871 #279 (11 connections now open)
2017-06-06T06:50:39.846+0000 I NETWORK [conn278] end connection 127.0.0.1:35869 (10 connections now open)
2017-06-06T06:51:03.870+0000 I NETWORK [conn264] end connection 127.0.0.1:35819 (9 connections now open)
2017-06-06T06:51:05.048+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35875 #280 (10 connections now open)
2017-06-06T06:51:32.902+0000 I NETWORK [conn276] end connection 127.0.0.1:35865 (9 connections now open)
2017-06-06T06:52:13.954+0000 I NETWORK [conn266] end connection 127.0.0.1:35826 (8 connections now open)
2017-06-06T06:52:20.958+0000 I NETWORK [conn221] end connection 127.0.0.1:35457 (7 connections now open)
2017-06-06T06:52:22.432+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35882 #281 (8 connections now open)
2017-06-06T06:52:39.052+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35883 #282 (9 connections now open)
2017-06-06T06:52:44.981+0000 I NETWORK [conn268] end connection 127.0.0.1:35829 (8 connections now open)
2017-06-06T06:52:46.619+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35885 #283 (9 connections now open)
2017-06-06T06:53:03.509+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35886 #284 (10 connections now open)
2017-06-06T06:53:04.399+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35887 #285 (11 connections now open)
2017-06-06T06:53:06.160+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35888 #286 (12 connections now open)
2017-06-06T06:54:25.183+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35889 #287 (13 connections now open)
2017-06-06T06:54:55.284+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35890 #288 (14 connections now open)
2017-06-06T06:55:27.122+0000 I NETWORK [conn279] end connection 127.0.0.1:35871 (13 connections now open)
2017-06-06T06:55:28.117+0000 I NETWORK [conn269] end connection 127.0.0.1:35832 (12 connections now open)
2017-06-06T06:57:09.066+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35894 #289 (13 connections now open)
2017-06-06T06:57:09.092+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35895 #290 (14 connections now open)
2017-06-06T06:57:17.206+0000 I NETWORK [conn287] end connection 127.0.0.1:35889 (13 connections now open)
2017-06-06T06:57:24.214+0000 I NETWORK [conn281] end connection 127.0.0.1:35882 (12 connections now open)
2017-06-06T06:58:59.404+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35898 #291 (13 connections now open)
2017-06-06T06:59:17.301+0000 I NETWORK [conn277] end connection 127.0.0.1:35866 (12 connections now open)
2017-06-06T06:59:46.326+0000 I NETWORK [conn288] end connection 127.0.0.1:35890 (11 connections now open)
2017-06-06T06:59:49.328+0000 I NETWORK [conn274] end connection 127.0.0.1:35860 (10 connections now open)
2017-06-06T07:00:03.342+0000 I NETWORK [conn290] end connection 127.0.0.1:35895 (9 connections now open)
2017-06-06T07:05:46.650+0000 I NETWORK [conn275] end connection 127.0.0.1:35863 (8 connections now open)
2017-06-06T13:15:43.047+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35995 #292 (9 connections now open)
2017-06-06T13:25:08.760+0000 I NETWORK [conn292] end connection 127.0.0.1:35995 (8 connections now open)
2017-06-07T04:07:48.262+0000 I NETWORK [conn285] end connection 127.0.0.1:35887 (7 connections now open)
2017-06-07T06:00:43.026+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36231 #293 (8 connections now open)
2017-06-07T06:10:21.038+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36236 #294 (9 connections now open)
2017-06-07T06:12:08.906+0000 I NETWORK [conn293] end connection 127.0.0.1:36231 (8 connections now open)
2017-06-07T06:13:03.738+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36238 #295 (9 connections now open)
2017-06-07T06:13:13.005+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36239 #296 (10 connections now open)
2017-06-07T06:14:10.995+0000 I NETWORK [conn294] end connection 127.0.0.1:36236 (9 connections now open)
2017-06-07T06:15:02.038+0000 I NETWORK [conn291] end connection 127.0.0.1:35898 (8 connections now open)
2017-06-07T06:15:16.486+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36244 #297 (9 connections now open)
2017-06-07T06:15:26.680+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36245 #298 (10 connections now open)
2017-06-07T06:16:43.118+0000 I NETWORK [conn282] end connection 127.0.0.1:35883 (9 connections now open)
2017-06-07T06:17:07.667+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36248 #299 (10 connections now open)
2017-06-07T06:17:19.869+0000 I CONTROL [main] ***** SERVER RESTARTED *****
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] MongoDB starting : pid=26429 port=27017 dbpath=/var/lib/mongodb 64-bit host=ip-172-31-28-133
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] db version v3.2.6
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] allocator: tcmalloc
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] modules: none
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] build environment:
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] distmod: ubuntu1404
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] distarch: x86_64
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] target_arch: x86_64
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2017-06-07T06:17:19.963+0000 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-06-07T06:17:19.963+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.553+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2017-06-07T06:17:20.554+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongodb/diagnostic.data'


I have noticed that information is provided when the service is restarted. But I am unable to figure out the problem behind the crashes.



I suspect its a memory issue as google searches show memory errors. But since they are not recorded in the log file.



I have a t2.micro instance on the AWS EC2 where mongo script is running.
How can I prevent mongo from crashing?










share|improve this question























  • How did you know it has crashed?

    – Arun
    Jun 8 '17 at 10:16











  • my website show connection error and when I reboot the mongod service then website start working again.

    – Curious Developer
    Jun 8 '17 at 10:51











  • Take a look at the /var/lib/mongodb/diagnostic.data

    – Arun
    Jun 8 '17 at 10:55













  • this folder have a lot of metric named files for example "metrics.2017-06-07T06-17-21Z-00000" containing encrypt data.

    – Curious Developer
    Jun 8 '17 at 11:17













  • no file named diagnostic.data?

    – Arun
    Jun 8 '17 at 11:21














0












0








0








Hi I am having problems with mongo db on my server. Mongo db is getting crashed in few days. I have added a shell script to reboot mongod service on server if crashed but I want to know what should I do to make sure it do not crash at all.



Also mongodb.log file in "/var/log/mongodb"is not showing the reason on crash. It show me connections closing and ending only.



I am adding few lines from the log file here:



2017-06-06T06:49:52.305+0000 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:35871 #279 (11 connections now open)
2017-06-06T06:50:39.846+0000 I NETWORK [conn278] end connection 127.0.0.1:35869 (10 connections now open)
2017-06-06T06:51:03.870+0000 I NETWORK [conn264] end connection 127.0.0.1:35819 (9 connections now open)
2017-06-06T06:51:05.048+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35875 #280 (10 connections now open)
2017-06-06T06:51:32.902+0000 I NETWORK [conn276] end connection 127.0.0.1:35865 (9 connections now open)
2017-06-06T06:52:13.954+0000 I NETWORK [conn266] end connection 127.0.0.1:35826 (8 connections now open)
2017-06-06T06:52:20.958+0000 I NETWORK [conn221] end connection 127.0.0.1:35457 (7 connections now open)
2017-06-06T06:52:22.432+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35882 #281 (8 connections now open)
2017-06-06T06:52:39.052+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35883 #282 (9 connections now open)
2017-06-06T06:52:44.981+0000 I NETWORK [conn268] end connection 127.0.0.1:35829 (8 connections now open)
2017-06-06T06:52:46.619+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35885 #283 (9 connections now open)
2017-06-06T06:53:03.509+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35886 #284 (10 connections now open)
2017-06-06T06:53:04.399+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35887 #285 (11 connections now open)
2017-06-06T06:53:06.160+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35888 #286 (12 connections now open)
2017-06-06T06:54:25.183+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35889 #287 (13 connections now open)
2017-06-06T06:54:55.284+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35890 #288 (14 connections now open)
2017-06-06T06:55:27.122+0000 I NETWORK [conn279] end connection 127.0.0.1:35871 (13 connections now open)
2017-06-06T06:55:28.117+0000 I NETWORK [conn269] end connection 127.0.0.1:35832 (12 connections now open)
2017-06-06T06:57:09.066+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35894 #289 (13 connections now open)
2017-06-06T06:57:09.092+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35895 #290 (14 connections now open)
2017-06-06T06:57:17.206+0000 I NETWORK [conn287] end connection 127.0.0.1:35889 (13 connections now open)
2017-06-06T06:57:24.214+0000 I NETWORK [conn281] end connection 127.0.0.1:35882 (12 connections now open)
2017-06-06T06:58:59.404+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35898 #291 (13 connections now open)
2017-06-06T06:59:17.301+0000 I NETWORK [conn277] end connection 127.0.0.1:35866 (12 connections now open)
2017-06-06T06:59:46.326+0000 I NETWORK [conn288] end connection 127.0.0.1:35890 (11 connections now open)
2017-06-06T06:59:49.328+0000 I NETWORK [conn274] end connection 127.0.0.1:35860 (10 connections now open)
2017-06-06T07:00:03.342+0000 I NETWORK [conn290] end connection 127.0.0.1:35895 (9 connections now open)
2017-06-06T07:05:46.650+0000 I NETWORK [conn275] end connection 127.0.0.1:35863 (8 connections now open)
2017-06-06T13:15:43.047+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35995 #292 (9 connections now open)
2017-06-06T13:25:08.760+0000 I NETWORK [conn292] end connection 127.0.0.1:35995 (8 connections now open)
2017-06-07T04:07:48.262+0000 I NETWORK [conn285] end connection 127.0.0.1:35887 (7 connections now open)
2017-06-07T06:00:43.026+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36231 #293 (8 connections now open)
2017-06-07T06:10:21.038+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36236 #294 (9 connections now open)
2017-06-07T06:12:08.906+0000 I NETWORK [conn293] end connection 127.0.0.1:36231 (8 connections now open)
2017-06-07T06:13:03.738+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36238 #295 (9 connections now open)
2017-06-07T06:13:13.005+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36239 #296 (10 connections now open)
2017-06-07T06:14:10.995+0000 I NETWORK [conn294] end connection 127.0.0.1:36236 (9 connections now open)
2017-06-07T06:15:02.038+0000 I NETWORK [conn291] end connection 127.0.0.1:35898 (8 connections now open)
2017-06-07T06:15:16.486+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36244 #297 (9 connections now open)
2017-06-07T06:15:26.680+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36245 #298 (10 connections now open)
2017-06-07T06:16:43.118+0000 I NETWORK [conn282] end connection 127.0.0.1:35883 (9 connections now open)
2017-06-07T06:17:07.667+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36248 #299 (10 connections now open)
2017-06-07T06:17:19.869+0000 I CONTROL [main] ***** SERVER RESTARTED *****
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] MongoDB starting : pid=26429 port=27017 dbpath=/var/lib/mongodb 64-bit host=ip-172-31-28-133
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] db version v3.2.6
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] allocator: tcmalloc
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] modules: none
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] build environment:
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] distmod: ubuntu1404
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] distarch: x86_64
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] target_arch: x86_64
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2017-06-07T06:17:19.963+0000 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-06-07T06:17:19.963+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.553+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2017-06-07T06:17:20.554+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongodb/diagnostic.data'


I have noticed that information is provided when the service is restarted. But I am unable to figure out the problem behind the crashes.



I suspect its a memory issue as google searches show memory errors. But since they are not recorded in the log file.



I have a t2.micro instance on the AWS EC2 where mongo script is running.
How can I prevent mongo from crashing?










share|improve this question














Hi I am having problems with mongo db on my server. Mongo db is getting crashed in few days. I have added a shell script to reboot mongod service on server if crashed but I want to know what should I do to make sure it do not crash at all.



Also mongodb.log file in "/var/log/mongodb"is not showing the reason on crash. It show me connections closing and ending only.



I am adding few lines from the log file here:



2017-06-06T06:49:52.305+0000 I NETWORK  [initandlisten] connection accepted from 127.0.0.1:35871 #279 (11 connections now open)
2017-06-06T06:50:39.846+0000 I NETWORK [conn278] end connection 127.0.0.1:35869 (10 connections now open)
2017-06-06T06:51:03.870+0000 I NETWORK [conn264] end connection 127.0.0.1:35819 (9 connections now open)
2017-06-06T06:51:05.048+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35875 #280 (10 connections now open)
2017-06-06T06:51:32.902+0000 I NETWORK [conn276] end connection 127.0.0.1:35865 (9 connections now open)
2017-06-06T06:52:13.954+0000 I NETWORK [conn266] end connection 127.0.0.1:35826 (8 connections now open)
2017-06-06T06:52:20.958+0000 I NETWORK [conn221] end connection 127.0.0.1:35457 (7 connections now open)
2017-06-06T06:52:22.432+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35882 #281 (8 connections now open)
2017-06-06T06:52:39.052+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35883 #282 (9 connections now open)
2017-06-06T06:52:44.981+0000 I NETWORK [conn268] end connection 127.0.0.1:35829 (8 connections now open)
2017-06-06T06:52:46.619+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35885 #283 (9 connections now open)
2017-06-06T06:53:03.509+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35886 #284 (10 connections now open)
2017-06-06T06:53:04.399+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35887 #285 (11 connections now open)
2017-06-06T06:53:06.160+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35888 #286 (12 connections now open)
2017-06-06T06:54:25.183+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35889 #287 (13 connections now open)
2017-06-06T06:54:55.284+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35890 #288 (14 connections now open)
2017-06-06T06:55:27.122+0000 I NETWORK [conn279] end connection 127.0.0.1:35871 (13 connections now open)
2017-06-06T06:55:28.117+0000 I NETWORK [conn269] end connection 127.0.0.1:35832 (12 connections now open)
2017-06-06T06:57:09.066+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35894 #289 (13 connections now open)
2017-06-06T06:57:09.092+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35895 #290 (14 connections now open)
2017-06-06T06:57:17.206+0000 I NETWORK [conn287] end connection 127.0.0.1:35889 (13 connections now open)
2017-06-06T06:57:24.214+0000 I NETWORK [conn281] end connection 127.0.0.1:35882 (12 connections now open)
2017-06-06T06:58:59.404+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35898 #291 (13 connections now open)
2017-06-06T06:59:17.301+0000 I NETWORK [conn277] end connection 127.0.0.1:35866 (12 connections now open)
2017-06-06T06:59:46.326+0000 I NETWORK [conn288] end connection 127.0.0.1:35890 (11 connections now open)
2017-06-06T06:59:49.328+0000 I NETWORK [conn274] end connection 127.0.0.1:35860 (10 connections now open)
2017-06-06T07:00:03.342+0000 I NETWORK [conn290] end connection 127.0.0.1:35895 (9 connections now open)
2017-06-06T07:05:46.650+0000 I NETWORK [conn275] end connection 127.0.0.1:35863 (8 connections now open)
2017-06-06T13:15:43.047+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:35995 #292 (9 connections now open)
2017-06-06T13:25:08.760+0000 I NETWORK [conn292] end connection 127.0.0.1:35995 (8 connections now open)
2017-06-07T04:07:48.262+0000 I NETWORK [conn285] end connection 127.0.0.1:35887 (7 connections now open)
2017-06-07T06:00:43.026+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36231 #293 (8 connections now open)
2017-06-07T06:10:21.038+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36236 #294 (9 connections now open)
2017-06-07T06:12:08.906+0000 I NETWORK [conn293] end connection 127.0.0.1:36231 (8 connections now open)
2017-06-07T06:13:03.738+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36238 #295 (9 connections now open)
2017-06-07T06:13:13.005+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36239 #296 (10 connections now open)
2017-06-07T06:14:10.995+0000 I NETWORK [conn294] end connection 127.0.0.1:36236 (9 connections now open)
2017-06-07T06:15:02.038+0000 I NETWORK [conn291] end connection 127.0.0.1:35898 (8 connections now open)
2017-06-07T06:15:16.486+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36244 #297 (9 connections now open)
2017-06-07T06:15:26.680+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36245 #298 (10 connections now open)
2017-06-07T06:16:43.118+0000 I NETWORK [conn282] end connection 127.0.0.1:35883 (9 connections now open)
2017-06-07T06:17:07.667+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:36248 #299 (10 connections now open)
2017-06-07T06:17:19.869+0000 I CONTROL [main] ***** SERVER RESTARTED *****
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] MongoDB starting : pid=26429 port=27017 dbpath=/var/lib/mongodb 64-bit host=ip-172-31-28-133
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] db version v3.2.6
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] allocator: tcmalloc
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] modules: none
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] build environment:
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] distmod: ubuntu1404
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] distarch: x86_64
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] target_arch: x86_64
2017-06-07T06:17:19.919+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2017-06-07T06:17:19.963+0000 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-06-07T06:17:19.963+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2017-06-07T06:17:20.514+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2017-06-07T06:17:20.515+0000 I CONTROL [initandlisten]
2017-06-07T06:17:20.553+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2017-06-07T06:17:20.554+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongodb/diagnostic.data'


I have noticed that information is provided when the service is restarted. But I am unable to figure out the problem behind the crashes.



I suspect its a memory issue as google searches show memory errors. But since they are not recorded in the log file.



I have a t2.micro instance on the AWS EC2 where mongo script is running.
How can I prevent mongo from crashing?







14.04 amazon-ec2 mongodb






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Jun 8 '17 at 10:07









Curious DeveloperCurious Developer

23118




23118













  • How did you know it has crashed?

    – Arun
    Jun 8 '17 at 10:16











  • my website show connection error and when I reboot the mongod service then website start working again.

    – Curious Developer
    Jun 8 '17 at 10:51











  • Take a look at the /var/lib/mongodb/diagnostic.data

    – Arun
    Jun 8 '17 at 10:55













  • this folder have a lot of metric named files for example "metrics.2017-06-07T06-17-21Z-00000" containing encrypt data.

    – Curious Developer
    Jun 8 '17 at 11:17













  • no file named diagnostic.data?

    – Arun
    Jun 8 '17 at 11:21



















  • How did you know it has crashed?

    – Arun
    Jun 8 '17 at 10:16











  • my website show connection error and when I reboot the mongod service then website start working again.

    – Curious Developer
    Jun 8 '17 at 10:51











  • Take a look at the /var/lib/mongodb/diagnostic.data

    – Arun
    Jun 8 '17 at 10:55













  • this folder have a lot of metric named files for example "metrics.2017-06-07T06-17-21Z-00000" containing encrypt data.

    – Curious Developer
    Jun 8 '17 at 11:17













  • no file named diagnostic.data?

    – Arun
    Jun 8 '17 at 11:21

















How did you know it has crashed?

– Arun
Jun 8 '17 at 10:16





How did you know it has crashed?

– Arun
Jun 8 '17 at 10:16













my website show connection error and when I reboot the mongod service then website start working again.

– Curious Developer
Jun 8 '17 at 10:51





my website show connection error and when I reboot the mongod service then website start working again.

– Curious Developer
Jun 8 '17 at 10:51













Take a look at the /var/lib/mongodb/diagnostic.data

– Arun
Jun 8 '17 at 10:55







Take a look at the /var/lib/mongodb/diagnostic.data

– Arun
Jun 8 '17 at 10:55















this folder have a lot of metric named files for example "metrics.2017-06-07T06-17-21Z-00000" containing encrypt data.

– Curious Developer
Jun 8 '17 at 11:17







this folder have a lot of metric named files for example "metrics.2017-06-07T06-17-21Z-00000" containing encrypt data.

– Curious Developer
Jun 8 '17 at 11:17















no file named diagnostic.data?

– Arun
Jun 8 '17 at 11:21





no file named diagnostic.data?

– Arun
Jun 8 '17 at 11:21










1 Answer
1






active

oldest

votes


















0














If there was a problem, MongoDB would have printed an error message or at least indicated it was shutting down. When those messages do not exist it means that the process was hard-killed by something else on the server.I'd recommend checking your system logs to see if the Operating System decided to kill the process for some reason.



the very high levels of anon memory persist even through MongoDB restarts. This indicates that the memory issue is not being caused by MongoDB and instead by something else.
please refer this link
https://jira.mongodb.org/browse/SERVER-22000



It is recommended reserving servers for just MongoDB and allocating only one mongod process per server.






share|improve this answer























    Your Answer








    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "89"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f923330%2fmongodb-getting-crashed-on-server%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    0














    If there was a problem, MongoDB would have printed an error message or at least indicated it was shutting down. When those messages do not exist it means that the process was hard-killed by something else on the server.I'd recommend checking your system logs to see if the Operating System decided to kill the process for some reason.



    the very high levels of anon memory persist even through MongoDB restarts. This indicates that the memory issue is not being caused by MongoDB and instead by something else.
    please refer this link
    https://jira.mongodb.org/browse/SERVER-22000



    It is recommended reserving servers for just MongoDB and allocating only one mongod process per server.






    share|improve this answer




























      0














      If there was a problem, MongoDB would have printed an error message or at least indicated it was shutting down. When those messages do not exist it means that the process was hard-killed by something else on the server.I'd recommend checking your system logs to see if the Operating System decided to kill the process for some reason.



      the very high levels of anon memory persist even through MongoDB restarts. This indicates that the memory issue is not being caused by MongoDB and instead by something else.
      please refer this link
      https://jira.mongodb.org/browse/SERVER-22000



      It is recommended reserving servers for just MongoDB and allocating only one mongod process per server.






      share|improve this answer


























        0












        0








        0







        If there was a problem, MongoDB would have printed an error message or at least indicated it was shutting down. When those messages do not exist it means that the process was hard-killed by something else on the server.I'd recommend checking your system logs to see if the Operating System decided to kill the process for some reason.



        the very high levels of anon memory persist even through MongoDB restarts. This indicates that the memory issue is not being caused by MongoDB and instead by something else.
        please refer this link
        https://jira.mongodb.org/browse/SERVER-22000



        It is recommended reserving servers for just MongoDB and allocating only one mongod process per server.






        share|improve this answer













        If there was a problem, MongoDB would have printed an error message or at least indicated it was shutting down. When those messages do not exist it means that the process was hard-killed by something else on the server.I'd recommend checking your system logs to see if the Operating System decided to kill the process for some reason.



        the very high levels of anon memory persist even through MongoDB restarts. This indicates that the memory issue is not being caused by MongoDB and instead by something else.
        please refer this link
        https://jira.mongodb.org/browse/SERVER-22000



        It is recommended reserving servers for just MongoDB and allocating only one mongod process per server.







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Jan 5 at 5:21









        MamtaMamta

        1




        1






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Ask Ubuntu!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f923330%2fmongodb-getting-crashed-on-server%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            Human spaceflight

            Can not write log (Is /dev/pts mounted?) - openpty in Ubuntu-on-Windows?

            File:DeusFollowingSea.jpg