
{"id":9737,"date":"2022-06-21T13:53:06","date_gmt":"2022-06-21T13:53:06","guid":{"rendered":"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/documentation\/rdk_broadband_documentation\/components\/testanddiagnostic\/"},"modified":"2025-03-25T05:18:33","modified_gmt":"2025-03-25T05:18:33","slug":"testanddiagnostic","status":"publish","type":"page","link":"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/components\/testanddiagnostic\/","title":{"rendered":"TestAndDiagnostic"},"content":{"rendered":"<p><!DOCTYPE html PUBLIC \"-\/\/W3C\/\/DTD HTML 4.0 Transitional\/\/EN\" \"http:\/\/www.w3.org\/TR\/REC-html40\/loose.dtd\"><br \/>\n <html><body><\/p>\n<div class=\"confluence-information-macro confluence-information-macro-information conf-macro output-block\" data-hasbody=\"true\" data-macro-name=\"info\"><span class=\"aui-icon aui-icon-small aui-iconfont-info confluence-information-macro-icon\"> <\/span><\/p>\n<div class=\"confluence-information-macro-body\">\n<p><span>This page is under development<\/span><\/p>\n<\/div>\n<\/div>\n<div class=\"toc-macro client-side-toc-macro  conf-macro output-block\" data-headerelements=\"H1\" data-hasbody=\"false\" data-macro-name=\"toc\"> <\/div>\n\n\n<h2 class=\"wp-block-heading wp-main-header\">Introduction<\/h2>\n\n\n<p>TAD (Test And Diagnostic) monitors the amount of free memory available in the system at run time. It triggers the memory\/Resource Reclamation (RR) process in TDM, when the amount of free memory drops below a configurable threshold. RR process can also be triggered by memory allocation failures which results in a notification being sent to TDM to try to reclaim memory.<\/p>\n<p>Selfheal is another feature implemented in Test And Diagnostic Component.<\/p>\n<p>Self-heal monitors:<\/p>\n<ul>\n<li>CPU usage<\/li>\n<li>Memory Usage<\/li>\n<li>Critical RDK-B processes<\/li>\n<\/ul>\n<p>Self-heal stores Reset Count and Reboot Count.<br \/>Self-heal takes required action like: Rebooting the device, Restarting required process based on predefined conditions.<br \/>Self-heal does connectivity test.<\/p>\n<p><\/p>\n\n\n<h2 class=\"wp-block-heading wp-main-header\">Feature<\/h2>\n\n\n<h3 class=\"wp-sub-header\">Selfheal \u2013 Resource Monitoring<\/h3>\n<p>Monitors the resources periodically (eg: 15 mins).\u00a0If &#8220;Average Memory Used&#8221; reaches threshold value, necessary action will be executed.\u00a0<\/p>\n<p>&#8220;resource_monitor.sh&#8221; script is used for monitoring Memory and CPU usage.<br \/>Located in the device path: &#8220;\/usr\/ccsp\/tad\/resource_monitor.sh&#8221;.<\/p>\n<h3 class=\"wp-sub-header\">Selfheal \u2013 Process Monitoring<\/h3>\n<p>Monitors the processes periodically (eg: 15 mins) based on process id (pid).\u00a0Based on the process id availability, required action will be taken such as restarting the process, rebooting the device.<strong><br \/><\/strong><\/p>\n<p>&#8220;task_health_monitor.sh&#8221; script is used for monitoring RDK-B processes. This is located at path: &#8220;\/usr\/ccsp\/tad\/task_health_monitor.sh&#8221;. We can monitor any RDK-B processes by adding the process pid in this script.<\/p>\n<p>Self-heal stores Reset Count and Reboot Count<\/p>\n<h3 class=\"wp-sub-header\">Selfheal \u2013 Connectivity Test<\/h3>\n<p>Self-heal does connectivity test. Ping test will be done through server IP\/URI (this needs to be configured). If server IP\/URI is not configured, Ping test won&#8217;t be executed and no action will be taken. If server is configured and ping test fails, reboot action will be executed.<\/p>\n<p>&#8220;self_heal_connectivity_test.sh&#8221; script is used for ping test<\/p>\n<h3 class=\"wp-sub-header\">Selfheal \u2013 Action<\/h3>\n<p>Self-heal takes the required action through &#8220;corrective_action.sh&#8221; script.\u00a0This script has implementation of the actions.<\/p>\n<p>Some of the actions are:<\/p>\n<p>rebootNeeded &#8211; Reboots the device<br \/>resetNeeded &#8211; Restarts the required process<br \/>storeInformation &#8211; Stores Memory and CPU usage<\/p>\n<p>In Raspberry Pi the functionality of self-heal feature is provided by systemd.<\/p>\n<p>\u00a0<\/p>\n\n\n<h2 class=\"wp-block-heading wp-main-header\">Code Flow<\/h2>\n\n\n<\/p>\n<p><img decoding=\"async\" src=\"\/wp-content\/uploads\/sites\/16\/2023\/09\/CODE%20FLOW%201.drawio.png\"><\/p>\n<h3 class=\"wp-sub-header\">Resource Monitoring &#8211;&nbsp;resource_monitor.sh<\/h3>\n<\/p>\n<p><img decoding=\"async\" src=\"\/wp-content\/uploads\/sites\/16\/2023\/09\/RESOURCE%20MONITORING%20DIAG.drawio.png\"><\/p>\n<ul>\n<li>resource_monitor.sh monitors the Memory and CPU usage<\/li>\n<li>Average memory and CPU thresholds will be obtained from syscfg.db (default avg_cpu_threshold:100, avg_memory_threshold:100)<\/li>\n<\/ul>\n<p><\/p>\n<ul>\n<li>Memory Usage Monitor\n<ul>\n<li>Gets the total, free and used memory details using free command<\/li>\n<li>AvgMemUsed = usedMem*100 \/ totalMem<\/li>\n<li>if AvgMemUsed &gt; memory_threshold, device will be rebooted<\/li>\n<\/ul>\n<\/li>\n<li>CPU Usage Monitor\n<ul>\n<li>Active CPU is considered as sum of user, system, iowait, irq, softirq, steal cpu<\/li>\n<li>CPU usage difference in every 30 seconds for a period of 5 mins, is considered as an Curr_CPULoad_Avg.<\/li>\n<li>If Curr_CPULoad_Avg &gt; cpu_threshold, corrective action will be taken<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><\/p>\n<p><\/p>\n<h3 class=\"wp-sub-header\">Process Monitoring &#8211;&nbsp;task_health_monitor.sh<\/h3>\n<\/p>\n<p><img decoding=\"async\" src=\"\/wp-content\/uploads\/sites\/16\/2023\/09\/process%20monitoring.drawio.png\"><\/p>\n<ul>\n<li>task_health_monitor.sh monitors the status of various taks periodically and takes the corrective action<\/li>\n<li>Default monitoring interval is 15mins and can be modified using resource_monitor_interval in syscfg.db<\/li>\n<li>Monitors\n<ul>\n<li>Health of peer processor, in case of dual core processors<\/li>\n<li>Other tasks added as part of the script<\/li>\n<\/ul>\n<\/li>\n<li>New tasks can be added by editing the script<\/li>\n<\/ul>\n<p><\/p>\n<h3 class=\"wp-sub-header\">Connectivity Test &#8211;&nbsp;self_heal_connectivity_test.sh<\/h3>\n<\/p>\n<p><img decoding=\"async\" src=\"\/wp-content\/uploads\/sites\/16\/2023\/09\/connectivity%20test.drawio.png\"><\/p>\n<ul>\n<li>Self_heal_connectivity_test.sh will run Ping and DNS tests.<\/li>\n<li>ConnTest_PingInterval in syscfg.db specifies the frequency of the connectivity test.<\/li>\n<li>If nothing specified, it is 60seconds by default.<\/li>\n<\/ul>\n<p><\/p>\n<ul>\n<li><strong>runPingTest<\/strong>\n<ul>\n<li>Gets the IP (default_router IP) from syscfg.db<\/li>\n<li>If no IP specified, it will try pinging to default gw<\/li>\n<li>If ping fails, takes the corrective action, which is none by default.<\/li>\n<\/ul>\n<\/li>\n<li><strong>runDNSPingTest<\/strong>\n<ul>\n<li>This is disabled by default. Can be enabled by selfheal_dns_pingtest_enable in syscfg.db<\/li>\n<li>Gets the urlToVerify from syscfg.db<\/li>\n<li>If nslookup fails, takes the corrective action, which is none by default<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n<h2 class=\"wp-block-heading wp-main-header\">Objects<\/h2>\n\n\n<p>Self heal objects in its DML layer:\u00a0<\/p>\n<div class=\"panel conf-macro output-block\" data-hasbody=\"true\" data-macro-name=\"panel\">\n<div class=\"panelContent\">Device.SelfHeal.X_RDKCENTRAL-COM<\/div>\n<\/div>\n<p>Self heal can be Enabled\/disabled by the below data model. By default, it is enabled<\/p>\n<div class=\"code panel pdl conf-macro output-block\" data-hasbody=\"true\" data-macro-name=\"code\">\n<div class=\"codeContent panelContent pdl\">\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">$ dmcli eRT getv Device.SelfHeal.X_RDKCENTRAL-COM_Enable\nCR component name is: eRT.com.cisco.spvtg.ccsp.CR\nsubsystem_prefix eRT.\ngetv from\/to component(eRT.com.cisco.spvtg.ccsp.tdm): Device.SelfHeal.X_RDKCENTRAL-COM_Enable\nExecution succeed.\nParameter    1 name: Device.SelfHeal.X_RDKCENTRAL-COM_Enable\n               type:       bool,    value: true<\/pre>\n<\/div>\n<\/div>\n<p>\u00a0<\/p>\n<p>Verify the selfheal feature running status<\/p>\n<div class=\"code panel pdl conf-macro output-block\" data-hasbody=\"true\" data-macro-name=\"code\">\n<div class=\"codeContent panelContent pdl\">\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">$ ps -Af | grep -i self\n 4449 root       0:00 {self_heal_conne} \/bin\/sh \/usr\/ccsp\/tad\/self_heal_connectivity_test.sh\n18921 root       0:00 grep -i self<\/pre>\n<\/div>\n<\/div>\n<h3 class=\"wp-sub-header\">Resource monitoring<\/h3>\n<p>The Below DM is used to verify the Average CPU threshold. By default the value is set to 100<\/p>\n<div class=\"code panel pdl conf-macro output-block\" data-hasbody=\"true\" data-macro-name=\"code\">\n<div class=\"codeContent panelContent pdl\">\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">$ dmcli eRT getv Device.SelfHeal.ResourceMonitor.X_RDKCENTRAL-COM_AvgCPUThreshold\nCR component name is: eRT.com.cisco.spvtg.ccsp.CR\nsubsystem_prefix eRT.\ngetv from\/to component(eRT.com.cisco.spvtg.ccsp.tdm): Device.SelfHeal.ResourceMonitor.X_RDKCENTRAL-COM_AvgCPUThreshold\nExecution succeed.\nParameter    1 name: Device.SelfHeal.ResourceMonitor.X_RDKCENTRAL-COM_AvgCPUThreshold\n               type:       uint,    value: 100<\/pre>\n<\/div>\n<\/div>\n<p>\u00a0<\/p>\n<p>The Below DM is used to verify the Average Memory threshold. By default the value is set to 100<\/p>\n<div class=\"code panel pdl conf-macro output-block\" data-hasbody=\"true\" data-macro-name=\"code\">\n<div class=\"codeContent panelContent pdl\">\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\">$ dmcli eRT getv Device.SelfHeal.ResourceMonitor.X_RDKCENTRAL-COM_AvgMemoryThreshold\nCR component name is: eRT.com.cisco.spvtg.ccsp.CR\nsubsystem_prefix eRT.\ngetv from\/to component(eRT.com.cisco.spvtg.ccsp.tdm): Device.SelfHeal.ResourceMonitor.X_RDKCENTRAL-COM_AvgMemoryThreshold\nExecution succeed.\nParameter    1 name: Device.SelfHeal.ResourceMonitor.X_RDKCENTRAL-COM_AvgMemoryThreshold\n               type:       uint,    value: 100<\/pre>\n<\/div>\n<\/div>\n<p>\u00a0<\/p>\n<p>\u00a0<\/p>\n<p>\u00a0<\/p>\n<p>\u00a0<\/p>","protected":false},"excerpt":{"rendered":"<p>This page is under development Introduction TAD (Test And Diagnostic) monitors the amount of free [&hellip;]<\/p>\n","protected":false},"author":28,"featured_media":0,"parent":9575,"menu_order":29,"comment_status":"open","ping_status":"closed","template":"","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"class_list":["post-9737","page","type-page","status-publish","hentry"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v24.9 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>TestAndDiagnostic - RDK Documentation Portal | Documentation<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/components\/testanddiagnostic\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"TestAndDiagnostic - RDK Documentation Portal | Documentation\" \/>\n<meta property=\"og:description\" content=\"This page is under development Introduction TAD (Test And Diagnostic) monitors the amount of free [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/components\/testanddiagnostic\/\" \/>\n<meta property=\"og:site_name\" content=\"RDK Documentation Portal | Documentation\" \/>\n<meta property=\"article:modified_time\" content=\"2025-03-25T05:18:33+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/components\/testanddiagnostic\/\",\"url\":\"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/components\/testanddiagnostic\/\",\"name\":\"TestAndDiagnostic - RDK Documentation Portal | Documentation\",\"isPartOf\":{\"@id\":\"https:\/\/developer.rdkcentral.com\/documentation\/#website\"},\"datePublished\":\"2022-06-21T13:53:06+00:00\",\"dateModified\":\"2025-03-25T05:18:33+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/components\/testanddiagnostic\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/components\/testanddiagnostic\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/components\/testanddiagnostic\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/developer.rdkcentral.com\/documentation\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Documentation\",\"item\":\"https:\/\/developer.rdkcentral.com\/documentation\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"RDK Broadband\",\"item\":\"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/\"},{\"@type\":\"ListItem\",\"position\":4,\"name\":\"Components\",\"item\":\"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/components\/\"},{\"@type\":\"ListItem\",\"position\":5,\"name\":\"TestAndDiagnostic\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/developer.rdkcentral.com\/documentation\/#website\",\"url\":\"https:\/\/developer.rdkcentral.com\/documentation\/\",\"name\":\"RDK Documentation Portal | Documentation\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/developer.rdkcentral.com\/documentation\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"TestAndDiagnostic - RDK Documentation Portal | Documentation","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/components\/testanddiagnostic\/","og_locale":"en_US","og_type":"article","og_title":"TestAndDiagnostic - RDK Documentation Portal | Documentation","og_description":"This page is under development Introduction TAD (Test And Diagnostic) monitors the amount of free [&hellip;]","og_url":"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/components\/testanddiagnostic\/","og_site_name":"RDK Documentation Portal | Documentation","article_modified_time":"2025-03-25T05:18:33+00:00","twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/components\/testanddiagnostic\/","url":"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/components\/testanddiagnostic\/","name":"TestAndDiagnostic - RDK Documentation Portal | Documentation","isPartOf":{"@id":"https:\/\/developer.rdkcentral.com\/documentation\/#website"},"datePublished":"2022-06-21T13:53:06+00:00","dateModified":"2025-03-25T05:18:33+00:00","breadcrumb":{"@id":"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/components\/testanddiagnostic\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/components\/testanddiagnostic\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/components\/testanddiagnostic\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/developer.rdkcentral.com\/documentation\/"},{"@type":"ListItem","position":2,"name":"Documentation","item":"https:\/\/developer.rdkcentral.com\/documentation\/"},{"@type":"ListItem","position":3,"name":"RDK Broadband","item":"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/"},{"@type":"ListItem","position":4,"name":"Components","item":"https:\/\/developer.rdkcentral.com\/documentation\/documentation\/rdk_broadband_documentation\/components\/"},{"@type":"ListItem","position":5,"name":"TestAndDiagnostic"}]},{"@type":"WebSite","@id":"https:\/\/developer.rdkcentral.com\/documentation\/#website","url":"https:\/\/developer.rdkcentral.com\/documentation\/","name":"RDK Documentation Portal | Documentation","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/developer.rdkcentral.com\/documentation\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/developer.rdkcentral.com\/documentation\/wp-json\/wp\/v2\/pages\/9737","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/developer.rdkcentral.com\/documentation\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/developer.rdkcentral.com\/documentation\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/developer.rdkcentral.com\/documentation\/wp-json\/wp\/v2\/users\/28"}],"replies":[{"embeddable":true,"href":"https:\/\/developer.rdkcentral.com\/documentation\/wp-json\/wp\/v2\/comments?post=9737"}],"version-history":[{"count":3,"href":"https:\/\/developer.rdkcentral.com\/documentation\/wp-json\/wp\/v2\/pages\/9737\/revisions"}],"predecessor-version":[{"id":11868,"href":"https:\/\/developer.rdkcentral.com\/documentation\/wp-json\/wp\/v2\/pages\/9737\/revisions\/11868"}],"up":[{"embeddable":true,"href":"https:\/\/developer.rdkcentral.com\/documentation\/wp-json\/wp\/v2\/pages\/9575"}],"wp:attachment":[{"href":"https:\/\/developer.rdkcentral.com\/documentation\/wp-json\/wp\/v2\/media?parent=9737"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}