USB Host MSC test -hung 'zombie' process

Validating software is the process of trying to get reasonable coverage of the functionality in reasonable situations.
A 2nd test at trying to push the USB Host MSC has produced an interesting result -

A new version of ps also supplies a deeper view of the tasks and how much the stack has been used.
The end result was a series of hung processes - to be poetic a zombie undead process!  
My take is that somewhere deep in the system (far deeper than I can go) occasionally the process isn't exiting correctly.

The good news for me is that the USB Host MSC has not failed in a bunch of system failures, zombie processes,  Hard Faults, and that is what I need from the system.
Trying to be a good netizen (is that a nutter!) I've raised an issue against it in case anybody else is playing in this area.bitbucket.org/nuttx/nuttx/issues/11/hung-process 

'ps' when called uses this as per this update.  
https://bitbucket.org/nuttx/apps/commits/dd7c4c94935889fee77d79a188b62f11aa3fb0c6

To enable the stack (and heap) to be monitored with a watermark (the functionality changed about this time - but the Stack Coloration needs to be set now for its usage in ps, whenever its done)  
From make menuconfigs  
-->Build Setup-->Debug Options
    [*] Stack coloration
    [*] Heap coloration
Specifically Clicking on "Stack coloration" and then pressing 'h'
"   Enable stack coloration to initialize the stack memory to the value
   of STACK_COLOR and enable the stack checking APIs that can be used  
   to monitor the level of stack usage."

So pullin the latest changes, making a new build
$mknx.sh
$stwr

nsh> mount -t vfat /dev/sda /usb0
nsh>

Taking the one test script of the previous blog and using vi (cheat sheet for vi) I did the following
shuntils----  
 date >> /usb0/t/l_free  
 date >> /usb0/t/l_date
 hello >> /usb0/t/l_free
 free >> /usb0/t/l_free
 ps   >> /usb0/t/l_free

------

shspawn-----

set FSTOP /usb0/t/fstop

until [ -f $FSTOP ]

do

 sh /usb0/t/shuntils &

 sleep 1

done

--------

nsh> sh /usb0/t/shspawn &

# However after about 15 minutes it had failures. (see later) and seems to have left some zombie processes.

#The pid in this code is a signed integer, and wraps at 16xxxx

nsh>

#however also failed on 

nsh> umount usb0

nsh: umount: umount failed: 16 "Device or resource busy"

It turns out this "Device or resource busy" also means files open on /usb0 so I'm guessing is likely to be derived from the hung processes - and while an indicator not the root cause.

nsh> uname -a
NuttX 17.6 nxd47wk_wkd47tst1_20160713_1444 arm stm32f4discovery

# off its starts testing with some manual 'ps'.

sh [121:100]   #new spawn with PID

sh [124:100]

sh [127:100]

ps

  PID PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK   STACK   USED  FILLED COMMAND

    0   0 FIFO     Kthread --- Ready              00000000      0      0   0.0%  Idle Task

    1 224 FIFO     Kthread --- Waiting  Signal    00000000   2028    292  14.3%  hpwork

    2 100 FIFO     Task    --- Waiting  Semaphore 00000000   1004    508  50.5%  usbhost

    3 100 FIFO     Task    --- Running            00000000   4076    852  20.9%  init

    4 100 RR       pthread --- Waiting  Signal    00000000   2044    788  38.5%  <pthread> 0x10002f80

nsh> sh [135:100]

sh [138:100]

sh [141:100]

sh [149:100]

sh [152:100]

ps

  PID PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK   STACK   USED  FILLED COMMAND

    0   0 FIFO     Kthread --- Ready              00000000      0      0   0.0%  Idle Task

    1 224 FIFO     Kthread --- Waiting  Signal    00000000   2028    292  14.3%  hpwork

    2 100 FIFO     Task    --- Waiting  Semaphore 00000000   1004    508  50.5%  usbhost

    3 100 FIFO     Task    --- Running            00000000   4076    852  20.9%  init

    4 100 RR       pthread --- Waiting  Signal    00000000   2044    788  38.5%  <pthread> 0x10002f80

nsh> sh [155:100]

sh [158:100]

sh [166:100]

ps sh [169:100]

  PID PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK   STACK   USED  FILLED COMMAND

    0   0 FIFO     Kthread --- Ready              00000000      0      0   0.0%  Idle Task

    1 224 FIFO     Kthread --- Waiting  Signal    00000000   2028    292  14.3%  hpwork

    2 100 FIFO     Task    --- Waiting  Semaphore 00000000   1004    508  50.5%  usbhost

    3 100 FIFO     Task    --- Running            00000000   4076    852  20.9%  init

    4 100 RR       pthread --- Waiting  Signal    00000000   2044    788  38.5%  <pthread> 0x10002f80

  169 100 RR       pthread --- Ready              00000000   2044    812  39.7%  <pthread> 0x10004420

#NOTE: A spawned file snapshoted while running

nsh> sh [172:100]

sh [175:100]

sh [183:100]

sh [186:100]

# Hey Hey off for lunch while the system works 

#start is- Jul 13 14:54:49 2016

#sadly on returning  find that it failed again (with more of a log) 20minutes later at  Jul 13 15:14:33 2016

# and the log shows  ps: open failed: 24 "Too many open files" from  nuttx/include/errno.h:

Jul 13 15:14:32 2016

Hello, World!!

             total       used       free    largest

Mem:        188096      31392     156704     120368

  PID PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK   STACK   USED  FILLED COMMAND

    0   0 FIFO     Kthread --- Ready              00000000      0      0   0.0%  Idle Task

    1 224 FIFO     Kthread --- Waiting  Signal    00000000   2028    292  14.3%  hpwork

    2 100 FIFO     Task    --- Waiting  Semaphore 00000000   1004    508  50.5%  usbhost

    3 100 FIFO     Task    --- Waiting  Semaphore 00000000   4076    852  20.9%  init

    4 100 RR       pthread --- Waiting  Signal    00000000   2044    788  38.5%  <pthread> 0x10002f80

 5065 100 RR       pthread --- Running            00000000   2044   1060  51.8%  <pthread> 0x10005310

  859 100 RR       pthread --- Waiting  Semaphore 00000000   2044    900  44.0%  <pthread> 0x10004420

Jul 13 15:14:33 2016

Hello, World!!

<<<<missing ps>>>>>

Jul 13 15:14:34 2016

Hello, World!!

<<<<<strange ps>>>>

             total       used       free    largest

Mem:        188096      36336     151760     120368

  PID PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK   STACK   USED  FILLED COMMAND

nsh: ps: open failed: 24 "Too many open files"

    0                                                      nsh: ps: open failed: 24

     0      0   0.0%  nsh: ps: open failed: 24

Jul 13 15:14:35 2016

Hello, World!!

<<<<<strange ps>>>>

             total       used       free    largest

Mem:        188096      36336     151760     120368

  PID PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK   STACK   USED  FILLED COMMAND

nsh: ps: open failed: 24

    0                                                      nsh: ps: open failed: 24

     0      0   0.0%  nsh: ps: open failed: 24

# so now stop the script from running by 

nsh> echo stop > /usb0/t/fstop   #pwd is usb/tst1

#Trying 'ps' there are a few strang processes still present, that don't look like they died.

nsh> ps

  PID PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK   STACK   USED  FILLED COMMAND

    0   0 FIFO     Kthread --- Ready              00000000      0      0   0.0%  Idle Task

    1 224 FIFO     Kthread --- Waiting  Signal    00000000   2028    292  14.3%  hpwork

    2 100 FIFO     Task    --- Waiting  Semaphore 00000000   1004    508  50.5%  usbhost

    3 100 FIFO     Task    --- Running            00000000   4076    972  23.8%  init

  859 100 RR       pthread --- Waiting  Semaphore 00000000   2044    900  44.0%  <pthread> 0x10004420

 5069 100 RR       pthread --- Waiting  Semaphore 00000000   2044    812  39.7%  <pthread> 0x10005310

 7614 100 RR       pthread --- Waiting  Semaphore 00000000   2044    812  39.7%  <pthread> 0x10005f80

So the ps that failed while the script was running is that it has exceeded the CONFIG_NFILE_DESCRIPTORS  allocation of files at that point in time which is 8.
Of course the release investigation is why it isn't cleaning up, which is probably related to not dispatching the process correctly - leaving the zombie process.

include/errno.h: 24 -> EMFILE

fs/vfs/fs_open.c:
191 /* Associate the inode with a file structure */
192
193 fd = files_allocate(inode, oflags, 0, 0);
194 if (fd < 0)
195 {
196 ret = EMFILE;
197 goto errout_with_inode;
198 }

The suggestion from Greg Nuttx was "you need to set CONFIG_NFILE_DESCRIPTORS larger."
However trying builds  for 10, 14, 32 resulted in it running for a few seconds then a "Hard Fault Exception" - won't go there. This seems like another Nuttx brittleness.

The good news is that the file system is good and removable for inspection on Win10.

nsh> ls -ls usb0/tst1

/usb0/tst1:

 -rw-rw-rw-     172 shuntil

 -rw-rw-rw- 1123855 l_free

 -rw-rw-rw-     136 shuntils

 -rw-rw-rw-      79 shspawn

 -rw-rw-rw-   32823 l_date

 -rw-rw-rw-       6 fstop

So I'm learning that having two destination files l_date & l_free seems to cause a total increment in pid of 3. Possibly one for the new process "shuntils"  and one for each file - but that's a guess.
Rerunning this test with an extra date to verify time taken
/usb0/t/shuntils----
 date  >> /usb0/t/l_date
 hello >> /usb0/t/l_free
 date  >> /usb0/t/l_free
 free  >> /usb0/t/l_free
 ps    >> /usb0/t/l_free
 date  >> /usb0/t/l_free

------

/usb0/t/shspawn-----

set FSTOP /usb0/tst1/fstop

until [ -f $FSTOP ]

do

 sh /usb0/t//shuntils &

 sleep 1

done

--------

nsh> mount -t vfat /dev/sda /usb0

nsh> sh /usb0/t/shspawn &

For a  2sec interval - this is a lot slower - I ran it overnight and the pid wrapped at least twice

sh [32764:100] 
sh [32767:100] 
sh [7:100] 
sh [10:100] 

For some reason three was one zombie process pid=283, and it happened in the first pid 0-32767

sh [28060:100]
sh [28063:100]
ps
sh [28071:100]

  PID PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK   STACK   USED  FILLED COMMAND
    0   0 FIFO     Kthread --- Ready              00000000      0      0   0.0%  Idle Task
    1 224 FIFO     Kthread --- Waiting  Signal    00000000   2028    292  14.3%  hpwork
    2 100 FIFO     Task    --- Waiting  Semaphore 00000000   1004    508  50.5%  usbhost
    3 100 FIFO     Task    --- Running            00000000   4076    868  21.2%  init
    5 100 RR       pthread --- Waiting  Signal    00000000   2044    788  38.5%  <pthread> 0x10002f80
28071 100 RR       pthread --- Ready              00000000   2044   1100  53.8%  <pthread> 0x10005330
  283 100 RR       pthread --- Waiting  Semaphore 00000000   2044    812  39.7%  <pthread> 0x10004420
nsh> sh [28074:100]
sh [28078:100]
sh [28086:100]

So stopping

echo stop >/usb0/t/fstop

nsh> cd

nsh> umount usb0

nsh: umount: umount failed: 16

# Failed again - probably due to pid=283 - but no way of listing it.

# Checking ps, the stack useage is very pleasing, and the stack sizes could be reduced if needed - but better to have them to large at this stage.

nsh> ps

  PID PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK   STACK   USED  FILLED COMMAND

    0   0 FIFO     Kthread --- Ready              00000000      0      0   0.0%  Idle Task

    1 224 FIFO     Kthread --- Waiting  Signal    00000000   2028    292  14.3%  hpwork

    2 100 FIFO     Task    --- Waiting  Semaphore 00000000   1004    508  50.5%  usbhost

    3 100 FIFO     Task    --- Running            00000000   4076    972  23.8%  init

  283 100 RR       pthread --- Waiting  Semaphore 00000000   2044    812  39.7%  <pthread> 0x10004420

USB Host MSC succesfull test

Tutorial: Creating a test to validate a logging function on the USB Flash drive.

Validate in this context means - create a simple test that seems like it is reasonable exercise of some functionality and then let it run.

Test : Write entries to a log file.

The test script "shuntil"
------------------------------------
set FSTOP /usb0/tst1/fstop
until [ -f $FSTOP ]
do
 date >> /usb0/tst1/l_date
 free >> /usb0/tst1/l_free
 ps   >> /usb0/tst1/l_free
 sleep 1
done

------------------------------

is pretty simple. Write three entries to two files every second.

Initially I enabled vi 

Application Configuration à System Libraries and NSH Add-Ons

[*] VI Work-Alike Text Editor  ----  CONFIG_SYSTEM_VI

  (64)  Display width (columns) (NEW)

      (16)  Display height (rows) (NEW)

      (0)   Debug level (NEW)

      (2048) Builtin task stack size (NEW)

      (100) Builtin task priority (NEW)

I'm not that proficent in 'vi', I've always had another editor to use. So I spell it out for anybody else taking the plunge. The help is theapps/system/vi/vi.c

mknx.sh

stwr

To start the testing 

nsh> uname -a

NuttX 17.6 nxd47wk_wkd47tst1_20160713_1444 arm stm32f4discovery

nsh> mount -t vfat /dev/sda /usb0

nsh> cd /usb0/tst1

# It turns out vi doesn't use pwd - so start it up and then specify the write path at the end

nsh>vi

i #cmd to input - copy the test script in. There is a limit on Rx buffer size so copy/paste half the file and then next half. Mental note need to try xmondem,

#To finish - 

<ESC> : w /usb0/tsts1/shunitl<ret>

:q<ret>

>nsh ls -ls  # shows file

nsh> sh shutil &

# Thats it test is running. Best leave overnight.

# Come back next morning - 10hrs later.

echo stop >fstop   # stop test.

Inspect the files l_date for number of entries and then l_free.

Wonderful - an speeded up soak test that ran for 10hrs last night with flying colours. -

So close to 100K file level access to the USB flash, including append to a single file 72,000 times resulting in a file 17Mbytes. 

So that's cool !.

Update July14-0724pst:
Rerunning this test overnight with yesterdays latest update, stoping the test and attempting to remove the USB Flash drive for inspection I get
"Umount failed: 22" or "Invalid argument"


nsh> uname -a
NuttX 17.6 nxd47wk_wkd47tst1_20160713_1444 arm stm32f4discovery
nsh> pwd
/
nsh> ls
/:
 dev/
 proc/
 usb0/

nsh> umount usb0
nsh: umount: umount failed: 22

USB Host MSC test using the reference smt32d4discotst1

This is a follow on from previous reference board:
Description: Bring up a USB Host with MSC capability, and store periodic data on it.

Equipment Needed in addition to previous test.
*) a converter cable from USB micro to USB Type A Host
*) A USB Flash drive

The stm32f4discovery board has "USB OTG FS with micro-AB connector". 

This supports a USB Full Speed device to connect, and Nuttx drivers support a USB Host MSC capability - ie accessing a flash drive. 

$ make menuconfig

System Type--> STM32 Peripheral Support-

          [*] OTG FS

Device Drivers-->USB Host Driver Support

     [*]   Mass Storage Class Support       

<exit><exit>

Board Selection--> #notice the following are set

   (1024) USB host waiter stack size (NEW)

   (100) USB host waiter task priority (NEW)

<exit>

System Type-->USB FS Host Configuration

     (128) Rx Packet Size (NEW)

      (96) Non-periodic Tx FIFO Size (NEW)

      (128) Periodic Tx FIFO size (NEW)

      (128) Descriptor Size (NEW)

      [ ] Enable SOF interrupts (NEW)

#USB needs a work thread

RTOS Features-->Work queue support   CONFIG_SCHED_WORKQUEU

  [*] High priority (kernel) worker thread    CONFIG_SCHED_HPWORK

      (224) High priority worker thread priority (NEW)

      (50000) High priority worker thread period (NEW)

      (2048) High priority worker thread stack size (NEW)

 <exit>.....<save>
RTOS Features -->Stack and Heap Information
(these values are arbitrarily increased to insure enough ram - FUT needs to be characterized with a stack inspection program some time)
 (2048) Idle thread stack size 
  (4096) Main thread stack size 
  (2048) Minimum pthread stack size
  (2048) Default pthread stack size

$mknx.sh

 size nuttx

   text    data     bss     dec     hex filename

  86204     308    6148   92660   169f4 nuttx

$stwr

on the stm32f4discovery

uttShell (NSH) NuttX-17.6

nsh>

nsh> ls dev

/dev:

 console

 null

 rtc0

 ttyS0

# Plug in a Flash drive through the microUSB to USB Type A Receptacle, and look for  'sda'

nsh> ls dev

/dev:

 console

 null

 rtc0

 sda

 ttyS0

nsh>

#### Hurrah - its recognized the USB Flash drive.

mount -t vfat /dev/sda /usb0

nsh> ls

/:

 dev/

 proc/

 usb0/

nsh> ls /usb0

/usb0:

 System Volume Information/

 LOST.DIR/

.....

#### WOW!  Reading the directory   #### 

nsh> mkdir /usb0/tst1

#check if it writes to the directory
nsh> help > /usb0/tst1/h.txt

nsh> cat /usb0/tst1/h.txt

help usage:  help [-v] [<cmd>]

  [           dirname     free        mkfifo      rm          uname

  ?           date        help        mkrd        rmdir       umount

  basename    dd          hexdump     mh          set         unset

  break       df          kill        mount       sh          usleep

  cat         echo        ls          mv          sleep       xd

  cd          exec        mb          mw          test

  cp          exit        mkdir       ps          time

  cmp         false       mkfatfs     pwd         true

Builtin Apps:

# What meta info is there about the file? - but no date information

nsh> ls -lRs /usb0/tst1

/usb0/tst1:

 -rw-rw-rw-     619 h.txt

nsh>

#Now reset the board, with the USB Host installed, verify the USB Host is recognized and writable.

NuttShell (NSH) NuttX-17.6

nsh> ls /dev

/dev:

 console

 null

 rtc0

 sda

 ttyS0

nsh> mount -t vfat /dev/sda /usb0

nsh> ls /usb0/tst1

/usb0/tst1:

 h.txt

nsh> rm /usb0/tst1/h.txt

nsh> ls /usb0/tst1

/usb0/tst1:

nsh>

#File is gone - so writeable.

# Ok - so lets write to these directories from an app. This requires using an examples application that can be redirected.

$make menuconfig

Application Configuration --->Examples  

 [*] "Hello, World!" example

  (100) Hello task priority (NEW)

   (2048) Hello stack size (NEW)

<save><ok><exit>...

$mknx.sh

$stwr

nsh> help

help usage:  help [-v] [<cmd>]

  [           dirname     free        mkfifo      rm          uname

  ?           date        help        mkrd        rmdir       umount

  basename    dd          hexdump     mh          set         unset

  break       df          kill        mount       sh          usleep

  cat         echo        ls          mv          sleep       xd

  cd          exec        mb          mw          test

  cp          exit        mkdir       ps          time

  cmp         false       mkfatfs     pwd         true

Builtin Apps:

  hello

nsh> hello

Hello, World!!

nsh> mount -t vfat /dev/sda /usb0

nsh> ls

/:

 dev/

 proc/

 usb0/

nsh> free > /usb0/tst1/f.txt

nsh> cat /usb0/tst1/f.txt

             total       used       free    largest

Mem:        189120      13552     175568     121392

nsh> hello > /usb0/tst1/h.txt

nsh> cat /usb0/tst1/h.txt
Hello, World!!

nsh>
nsh> hello >> /usb0/tst1/h.txt
nsh>
nsh> cat /usb0/tst1/h.txt
Hello, World!!
Hello, World!!

nsh>

#Wowa - all working nicely (afterNuttx 17.7 or 17.6 afterJuly/7 - bugfix )

<tbd: modify a program to write periodic data to USB Flash>

Reference Test Machine - Stm32F4discovery

When building complex software its essential to do reference builds and designate a reference piece of hardware similar to the final target for testing and keep it in good condition. Too often with out this, testing is done on dodgy hardware that just causes strange problems until reference hardware is established.

My target application is for an IoT data logger, polling sensors "southbound", storing locally on an SDcard, or USB drive, and sending to the internet via serial to a WiFi or Cellular interface

I am using an open source hardware Olimex STM32F4 H407 for prototyping - as its industrial strength and likely to have a good product life. I've created a basic configs/olimex-stm32-h407 but it isn't widely used.

As an alternative hardware, for stability integrating similar functions, I'm  choosing the widely available STM32F4discovery that has had a lot of base builds performed on it in nuttx\configs\stm32f4discovery.
It is a demonstration board from ST for trying out their processor..
With the added baseboard this provides a lot of functionality, though I won't be attempting to exercise all the hardware.
For testing you will need
*) USB cable - Type A from the Ubuntu Host to USB mini on the stm32f4discovery card ST
*) USB to TTL/3V logic connector for the the Console - eg adafru.it/954 - also has power if needed.
*) STM baseboard  "STM32F4DIS-BB" or Digikey "497-13545-ND"

For the reference build - I'm going to try and build as much of a realistic project as I can for the target environment I'm looking at.

The reference environment starts from configs\stm32f4discovery\nsh\defconfig - and is built up from there.
On my ubuntu machine
$cd git
$mkdir nxhd47tst
$cd nxhd47tst

$git clone git@bitbucket.org:neilh20/anuttx.git nuttx

$git clone git@bitbucket.org:neilh20/nuttapps.git apps

$cd nuttx

$git checkout -b wkd47tst1   #work discoverystm32F407 test

#then I copy  the configs\stm32f4discovery to configs\stm32f4discotst

$cd configs

$cp -R stm32f4discovery stm32f4discotst

#Then per my previous blog nuttx-traceable-builds I do the following 

~/git/nxd47tst/nuttx (wkd47tst)]$

$nano .gitignore -->   remove /.config  /.Make.defs

$nano include/nuttx/.gitignore --> remove /config.h & /version.h

$cd tools

$  ./configure.sh stm32f4discotst/nsh
$ cd ..

$make menuconfig

Build Setup--> Build Host Platform (Linux)

System TypeàSTM32 Periperal Support [ ] OTG FS - disable CONFIG_STM32_OTGFS

$make

verify builds successfully.
$ size nuttx
   text    data     bss     dec     hex filename
  55961     100    2032   58093    e2ed nuttx

I'm using PA2/PA3 Uart1 for a debug console, so verify
$make menuconfig
System Type-->STM32 Peripheral-->
 [*] PWR
[*] SYSCFG
  [*] USART2
USART2 Driver Configuration (Standard serial driver)  ---> (X) Standard serial driver
Up one level to "System Type"
  [*] Exclude CCM SRAM from the heap
 [*] Dump stack on assertions
Board Selection
    Select target board (STMicro STM32F4-Discovery board)  --->
    [*] Board LED support
    [*] Board button support
(fut) [ ] STM32F4DIS-BB base board
Device Drivers
   [ ] Disable driver poll interfaces
   [*] Enable /dev/null
   [*] RAM Disk Support
   -*- SPI Driver Support  --->      [*]   SPI exchange
   [*] Serial Driver Support  --->
         [*]   Enable standard "upper-half" serial driver
         [*]   Support TIOCSERGSTRUCT
        Serial console (USART2)  --->
        USART2 Configuration  --->
            (128) Receive buffer size
#enable an enhanced debug I'm enabling a large TX buffer.
            (2048) Transmit buffer size
           (115200) BAUD rate
           (8) Character size
           (0) Parity setting
           (0) Uses 2 stop bits
           [ ] USART2 RTS flow control
           [ ] USART2 CTS flow control
           [ ] USART2 DMA support
<Save> <Exit> to "Configuration"
 File Systems  --->
 -*- Writable file system
    [*] FAT file system
    [*]   FAT upper/lower names
    [ ]   FAT long file names
    [*]   FAT timestamps
<save><exit>
Application Configuration-->NSH Library --> Configure Command Options
 [*] df: Enable [-h] man-readable format
<save> <ok> <exit> <exit>
#enable a date facility
Device Drivers-->Timer Driver Support -->RTC Driver Support
   [*]   Date/Time RTC Support
   [  ]   RTC Alarm Support (NEW)
   [*]   RTC Driver Support
   [*]     RTC IOCTLs
Application Configuration-->NSH Library -->Disablei ndividual Commands
 [ ] Disable date    # Ensure date turned off
System Type-->RTC clock source (LSI Clock)--> # select  Low Speed Internal
<save><ok> <exit>   till back to prompt
$mknx.sh      # it rebuilds all
$size nuttx
   text    data     bss     dec     hex filename
  73916     244    3972   78132   13134 nuttx

#Now with the USB cable plugged into an stm32F4disco (STM32F407)  download it
$stwr
#Connect up a console - I use "Tera Term", open the COM port associated with the F4disco
#On the F4disco press reset should get.
NuttShell (NSH) NuttX-17.6
nsh>
# Type in the following proving the build
nsh> uname -a
NuttX 17.6 nxd47wk_wkd47tst1_20160709_1742 arm stm32f4discovery


#Try help to list cmds with no buildin
nsh> help
help usage:  help [-v] [<cmd>]

  [           dirname     help        mkrd        rmdir       umount
  ?           dd          hexdump     mh          set         unset
  basename    df          kill        mount       sh          usleep
  break       echo        ls          mv          sleep       xd
  cat         exec        mb          mw          test
  cd          exit        mkdir       ps          time
  cp          false       mkfatfs     pwd         true
  cmp         free        mkfifo      rm          uname

Builtin Apps:
nsh>
# Make a ram disk to test writing a file to
mkrd 50
# Try ls dev - and checkout "ram0"
#then type
mkfatfs /dev/ram0
mount -t vfat /dev/ram0 /tmp

nsh>ls dev

nsh>ls tmp

nsh>help > /tmp/h.txt

nsh>cat /tmp/h.txt

#Hurrah created and wrote a text file to a temp directory. 

#List the df in human readable form

df -h

Filesystem    Size      Used  Available Mounted on

vfat         7680B        1K      6656B /tmp

nsh> free

             total       used       free    largest

Mem:        191376      33264     158112     123648

# Snap shot this

$git tag -a nxtst17.6_nxd47wk_wkd47tst1_20160709_1820 -m "tag attempt"
$git commit -a -m "version change"
$git push --tags

It can be grabbed with
eg from git
$mkdir wkd47t
$cd wkd47t

$git clone git@bitbucket.org:neilh20/anuttx.git nuttx

$git clone git@bitbucket.org:neilh20/nuttapps.git apps

$cd nuttx

$git checkout wkd47tst1 
$git checkout nxtst17.6_nxd47wk_wkd47tst1_20160709_1820
$make
<download>
<make any modifications and switch it to your own branch with>

git checkout -b <mybranch>

Nuttx Traceable Builds

With a Nuttx.org / https://github.com/nuttx   derived software build I find myself on a steep learning curve, a new OS that is evolving, and finding software issue(s) - mostly mine :).

The very nature of software development.

Nuttx covers a lot of different processor families and processors within those families  ~ all with a uniquely created .config to define a specific build.

A lot of options, and not [many|any]  test setups. Very easy to delete that .config while building features.

While there is a lot of functionality and leading edge stuff-  when hitting  an issue it very much  feels like the bleeding edge.

Of course if its my application software that has the issue - I can solve it and update the build:). If it appears a  nuttx software its a challenge learning it, and all those layers and configuration parms. How is it supposed to work ?

So how to provide a definitive reference build, that I can come back to, or anybody else on the net can easily access when an issue comes up.?  Or if a minor issue, I can capture the description in a bug description, and move on. Standard stability testing philosophy.


git on bitbucket is powerful, and if the following files are changed and a pre-processor  mknx.sh run then a definitive snapshsot is captured and displayable with "nsh> uname -a"  .

This is a work in progress. So from "nsh> uname -a" it expands as follows:

NuttX 17.6 nxd47wk_wkd47tst1_20160712_2203 arm stm32f4discovery

Hopefully mostly self explanatory

<work_directory>_<branch>_<date>   <arch> <board>

<work_directory> nxd47wk - which in my terminology is nx STM32F4(07)discovery boad
<branch> wk_nxd47_wk
<date>  YYYYMMDD_HHMM

So building for stm32f4discovery on my networked Ubuntu machine (I ssh from windows)
$cd ~/git
$mkdir nxd47wk
$cd  nxd47wk

$git clone git@bitbucket.org:neilh20/anuttx.git nuttx

$git clone git@bitbucket.org:neilh20/nuttapps.git apps

$cd nuttx

$cd tools
$ ./configure.sh stm32f4discovery/nsh
$ cd ..
git checkout -b wkd47tst1
~/git/nxd47wk/nuttx (wkd47tst1)]$

edit .gitignore -->   remove /.config  /.Make.defs

edit include/nuttx/.gitignore --> remove config.h & version.h

$make menuconfigs 

Build Setup--> Build Host Platform (Linux)

System TypeàSTM32 Periperal Support [ ] OTG FS - disable CONFIG_STM32_OTGFS

<Save>

and then build - I manually update the Nuttx build# in mknx.sh

$mknx.sh
download to the target
$stwr
then in the TTY window ( I use Tera Term )
nsh>
NuttShell (NSH) NuttX-17.6
nsh> uname -a
NuttX 17.6 nxd47wk_wkd47tst1_20160712_2203 arm stm32f4discovery

nsh>
So I have a build, lets provide a mechanism that can enable it to be duplicated somewhere else in the cyber world - like another directory.
Back on Ubuntu (an example)
~/git/nxd47wk/nuttx (wkd47tst1)]$ git commit -a -m "first nxtst17.6_nxd47wk_wkd47tst1_20160712_2203"

$git tag nxd47wk_wkd47tst1_20160712_2221 
$git push origin  wkd47tst1

now in another terminal window to my ubuntu machine I can duplicate that build (technically anywhere else in a nuttx linux environment with tools setup)
<New SSH term>
$mkdir nxd47tsta
$cd nxd47tsta

$git clone git@bitbucket.org:neilh20/anuttx.git nuttx

$git clone git@bitbucket.org:neilh20/nuttapps.git apps

$cd nuttx

$git checkout wkd47tst1  

$make

$stwr

and switching to the TTY connected with STM32F4(07)discovery 

- I get exactly the same build info

NuttShell (NSH) NuttX-17.6

nsh> uname -a

NuttX 17.6 nxd47wk_wkd47tst1_20160712_2221  arm stm32f4discovery

nsh>

            ~!~    It worked    ~!~:)

------------------------------------
where mknx.sh is in your ~/bin

#!/bin/bash

#  make with uptodate tag - uname -a

#chmod 755

function get_branch() {

      git branch --no-color | grep -E '^\*' | awk '{print $2}' \

        || echo "default_value"

      # or

      # git symbolic-ref --short -q HEAD || echo "default_value";

}

build_num=$(date +%Y%m%d_%H%M)

#echo $(__git_ps1)

branch_name=`get_branch`;

#repo_tree=${PWD##*/}

repo_tree=$(pwd | cut -d'/' -f5)

#echo "$repo_tree"

blabel="$repo_tree""_$branch_name""_$build_num"

echo "Building $blabel"

tools/version.sh -v 17.6 -b "$blabel" .version

make

------------------

similarly I have an alias stwr to ~/bin/st-flash-wr.sh

#!/bin/bash

#chmod 755

bin_file="nuttx.bin"

if [ "$1" != "" ]; then

    #echo "Using file $1"

    bin_file=$1

fi

echo "st-flash write $bin_file 0x8000000"

st-flash write $bin_file 0x8000000

------
160713: This blog updated for a changed format nxd47wk_wkd47tst1_20160712_2203 

Building the App example/alarms

Configuring a nuttx/app is done with "make menuconfig"
I find myself churning through the options trying to find dependencies to subsystems to turn them on.
The top level readme does help - however the general directions are to do it through the menuconfigs and the following README.txt describes it in terms of #defines.
https://bitbucket.org/nuttx/apps/src/master/examples/README.txt
So this is how I enable the subsystem for my test board olimex-STM32-H307 with TTY on Uart2
On my build machine - Ubuntu 15.10
cd tools
./configure.sh olimex-stm32-h407/nsh   
cd ..

make menuconfig

Requires RTOS FeaturesàRTOS Hooksà[*] Custom board/driver initialization 

Requires, Drivers -->Timer Driver Support àRTC Driver Support

.

Debug options

* Enable Debug Features

* Enable Debug Verbose Output

* RTC Debug Output

* Stack Colouration

* Generate Debug Symbols

Application Configurationà Examples

* RTC Alarm driver example (with defaults)

App Config à NSH LibàDisable Individual cmds-->

[] Disable date   # turn off disable

If on a board that is missing the 32KHz clock, stm32F4discovery board,
 then the RTC can function with another clock

 System Type àRTC clock source  LSI or HSI. They don't survive reset for some reason.

make clean

make

and then download it over the ST-LINK.

The output with RTC debugging enabled - using the initial power on time, looks like this

Ideally needs to set up date first but will work without it eg
nsh> date -s "JULY 06 13:45:00 2016"

nsh> alarm 15

alarm_daemon started

alarm_daemon: Running

Oprtc_dumpregs: *** Reading Time:

rtc_dumpregs:       TR: 00003532

rtc_dumpregs:       DR: 00002101

rtc_dumpregs:       CR: 00001100

rtc_dumpregs:      ISR: 00000127

rtc_dumpregs:     PRER: 007f00ff

rtc_dumpregs:     WUTR: 0000ffff

rtc_dumpregs:   CALIBR: 00000000

rtc_dumpregs:   ALRMAR: 80003333

rtc_dumpregs:   ALRMBR: 00000000

rtc_dumpregs:   SHIFTR: 00000000

rtc_dumpregs:     TSTR: 00000000

rtc_dumpregs:     TSDR: 00000000

rtc_dumpregs:    TSSSR: 00000000

rtc_dumpregs:     CALR: 00000000

rtc_dumpregs:    TAFCR: 00000000

rtc_dumpregs: ALRMASSR: 00000000

rtc_dumpregs: ALRMBSSR: 00000000

rtc_dumpregs: MAGICREG: facefeee

rtc_dumpregs: EXTI (RTSR FTSR ISR EVT): 1011

rtc_dumptime: Returning:

rtc_dumptime:   tm_sec: 00000020

rtc_dumptime:   tm_min: 00000023

rtc_dumptime:  tm_hour: 00000000

rtc_dumptime:  tm_mday: 00000001

rtc_dumptime:   tm_mon: 00000000

rtc_dumptime:  tm_year: 00000064

rtc_dumptime: New alarm time:

rtc_dumptime:   tm_sec: 00000030

rtc_dumptime:   tm_min: 00000023

rtc_dumptime:  tm_hour: 00000000

rtc_dumptime:  tm_mday: 00000001

rtc_dumptime:   tm_mon: 00000000

rtc_dumptime:  tm_year: 00000064
irtc_dumpregs: *** Set AlarmA:

rtc_dumpregs:       TR: 00003532

rtc_dumpregs:       DR: 00002101

rtc_dumpregs:       CR: 00001100

rtc_dumpregs:      ISR: 00000027

rtc_dumpregs:     PRER: 007f00ff

rtc_dumpregs:     WUTR: 0000ffff

rtc_dumpregs:   CALIBR: 00000000

rtc_dumpregs:   ALRMAR: 80003548

rtc_dumpregs:   ALRMBR: 00000000

rtc_dumpregs:   SHIFTR: 00000000

rtc_dumpregs:     TSTR: 00000000

rtc_dumpregs:     TSDR: 00000000

rtc_dumpregs:    TSSSR: 00000000

rtc_dumpregs:     CALR: 00000000

rtc_dumpregs:    TAFCR: 00000000

rtc_dumpregs: ALRMASSR: 00000000

rtc_dumpregs: ALRMBSSR: 00000000

rtc_dumpregs: MAGICREG: facefeee

rtc_dumpregs: EXTI (RTSR FTSR ISR EVT): 1011

ng /dev/rtc0

rtchw_set_alrmar:   TR: 00003532 ALRMAR: 80003548

Alarm set in 15 seconds

nsh>

(then when the alarm goes off)

rtc_dumpregs: *** Reading Time:

rtc_dumpregs:       TR: 00003548

rtc_dumpregs:       DR: 00002101

rtc_dumpregs:       CR: 00001100

rtc_dumpregs:      ISR: 00000127

rtc_dumpregs:     PRER: 007f00ff

rtc_dumpregs:     WUTR: 0000ffff

rtc_dumpregs:   CALIBR: 00000000

rtc_dumpregs:   ALRMAR: 80003548

rtc_dumpregs:   ALRMBR: 00000000

rtc_dumpregs:   SHIFTR: 00000000

rtc_dumpregs:     TSTR: 00000000

rtc_dumpregs:     TSDR: 00000000

rtc_dumpregs:    TSSSR: 00000000

rtc_dumpregs:     CALR: 00000000

rtc_dumpregs:    TAFCR: 00000000

rtc_dumpregs: ALRMASSR: 00000000

rtc_dumpregs: ALRMBSSR: 00000000

rtc_dumpregs: MAGICREG: facefeee

rtc_dumpregs: EXTI (RTSR FTSR ISR EVT): 1011

rtc_dumptime: Returning:

rtc_dumptime:   tm_sec: 00000030

rtc_dumptime:   tm_min: 00000023

rtc_dumptime:  tm_hour: 00000000

rtc_dumptime:  tm_mday: 00000001

rtc_dumptime:   tm_mon: 00000000

rtc_dumptime:  tm_year: 00000064

alarm_demon: alarm 0 received