This hopefully addresses memory leak that occurs when not using a Home Hub and connections constantly drop and re-establish.
To do: must add back logic for notifications using new hap linked-list structure, and then delete static hap array code.
Also adds: homeSpan.controllerListBegin(), homeSpan.controllerListEnd()
Also adds: Controller::isAdmin(), Controller::getID(), Controller::getLTPK()
To accomplish this, needed to move Controller definition from HAP.h to HomeSpan.h
Also, converted Controller from struct to class to ensure Controller data is protected from being modified now that it is exposed through condstant iterators.
Add getLen() and getTag() method to get length and tag of tlv8_t.
Also overrode subscript operator [] so you can access any element of internal uint8_t array.
Given previous additions (such as getVal()) there is now no reason to need to access the underlying std::unique_ptr directly.
Makes it easier to chain .add() functions as well as dynamically create sub TLVs.
Changing TLV8::add() in this fashion also required updating various use cases of TLV::add() in HAP.cpp where add() was used to reserve space for a blank TLV element. All changes have been tested.
HAPClient::putCharacteristics() now parses for "r":true in JSON, which is the HAP request for a write-response. When found, HomeSpan will return with a full write-response for all Characteristics updated using "207 Multi-Status", where "value" will be included only if "r":true was specified for a Characteristic.
HomeSpan processes "r":true for any Characteristic regardless if permissions includes WR.
To change the value of the Characteristic from within an update() method, use setVal() WITH OPTIONAL SECOND ARGUMENT AS FALSE to supress an EV broadcast (in the unlikely event that there was a notification subscriber).
To do: Consider generally disallowing the changing of any Characteristic value from within update() using setVal() unless the Characteristic has WR permission. And if so, automatically surpress EV notification.
Eliminated the need to instantiate a static HKDF structure within HAPClient since there is no storage within HKDF itself, just a single function call (create).
homeSpan.setPairingCode() operates silently if called from within a sketch, but will report error to Serial Monitor and HALT sketch if code is invalid.
Otherwise, if called from 'S', or from AP, or as default during start-up, diagnostics are printed as usual and sketch will NOT halt if invalid code is provided
* Converted various homeSpan.set() methods from void to Span& to enable chaining
* Update Reference.md
* Update Reference.md
* Update src.ino
* Update Pixel.ino
changed nPixels from uint8_t to int to allow for longer pixel strands (library is coded for int, so there is no need to limit to uint8_t in the example).
* Converted static HTTP Buffer to dynamic TempBuffer
Saved about 8K in RAM!
* Upgraded TempBuffer logic to use .get() to return pointer
Next up: update Network.cpp to use client.available() and reduce fixed memory usage.
* Update Network.cpp
* Replaced appropriate snprintf() with asprintf() and used *m in sscanf
Optimized use of heap memory instead of stack memory for temporary variables, and makes code easier to read, provided all heap usage is free() at end of each function.
Also removed char *hostName as a member variable of homeSpan. It's never actually used since MDNS constructs its own copy of hostName as needed.
* Removed duplicate if(POST /pairings)
* modified HAP::receivedEncrypted() to use TempBuffer instead of fixed stack array
* Added HAPClient::listControllers()
Will be used for pairings list request
* Completed adding LIST to HAPClient::postPairingsURL()
HomeSpan now correctly responds to pairing list requests
* Refactored TLV
* Added support for zero-length TLV
* Added SEPARATOR as a formal kTLVType (and updated listControllers() to use)
* Added `uint8_t *buf(tagType tag, uint8_t *src, int len);` to load buffer needing external memcpy (and updated listControllers() to use)
* Simplified `uint8_t *TLV<tagType, maxTags>::buf(tagType tag, int len)`
Also updated HAP.cpp to use new `uint8_t *TLV<tagType, maxTags>::buf(tagType tag, uint8_t *src, int len)`
* Refactored Controller Structure and Add/Remove/List/Save Functions
Changed fixed-size array `struct Controller[MAX_CONTROLLERS]` to a dynamic linked-list of Controllers. Re-coded all related functions. MAX_CONTROLLERS no sets the size of any buffers but is only used to limit the ultimate size of the linked-list.
Saved about 1K of RAM since most of the time there are only 2 Controllers defined (as opposed to the 16 allowed).
* Support Non-English WiFi name
* Preventing possibility of buffer overwrite.
* Update README.md
* Update README.md
* Bumped release to 1.8.1
* Added homeSpan.setWebLogCallback(String (*f)())
Allows extension of initial table produced in WebLog, where f is a function that returns a String. The returned String is copied into the WebLog HTML just before the </table> tag is printed in the initial table.
* Change webLogCallback() to pass String as reference
* Verbose WiFi Reconnect
Add the ability to turn off WiFi reconnect messages.
* Update Reference.md
* Update Logging.md
* Update README.md
* Added homeSpan.setWifiCallbackAll(int n)
Adds a second type of WiFi Callback that is called every time WiFi is established OR re-established after a disconnect. Passes the number of times WiFi has been connected as an argument.
* Update README.md
* Update Reference.md
* Update README.md
* Made conforming changes to setVerboseWiFiReconnect
* Update Reference.md
* Update README.md
* Update Reference.md
* Update Logging.md
* Changed Accessory Limit from 41 to 150 and added Memory Warning
Low Memory Watermark is compared to DEFAULT_LOW_MEM_THRESHOLD (currently set at 80,000 bytes) after HAP initialization but before WIFI connection. If Low Memory Watermark is below DEFAULT_LOW_MEM_THRESHOLD, a WARNING is issued.
Note: memory is based on heap with MALLOC_CAP_DEFAULT capabilities
* Update Reference.md
* Made vTaskDelay(5) in pollTask conditional of using autoPoll()
vTaskDelay(5) is only needed if polling is in its own task. If calling poll() directly, there is no need for any delay.
* Update Logging.md
* Add hint for UTF8 usage in names
* Moved vTaskDelay(5) from pollTask() to homeSpan.autoPoll()
Adding the vTaskDelay(5) to the code that spawns the separate task is more consistent with how Arduino-ESP32 spawns the main loop() task.
* Update HomeSpan.h
Forgot braces in for-loop of autoPoll()
* Update 07-AccessoryNames.ino
fix typo
* Update Reference.md
* Add MAC Address output to RemoteDevice.ino
Prompts user with reminder of which MAC Address to use in Main Device.
* Added new homeSpan.rebootCallback() function
An optional callback used to track the number of times the device reboots, where a reboot is defined as any restart where the uptime was less than a specified number of milliseconds (default=5000)
* Update src.ino
* Update Reference.md
* Update src.ino
* Update README.md
* Update Reference.md
* Update README.md
* Update 07-AccessoryNames.ino
* Update HAPClient::sendEncrypted() to send individual frames to save memory
Rather than use a large temporary buffer to store all encrypted frames to transmit all at once, each encrypted frame is transmitted as soon as created, which means the temporary buffer only needs to be large enough to store a single encrypted frame.
For a device with a large number of Accessories, this can save 15-20 KB of temporary storage, allowing for more Accessories.
Note this change does not reduce the size of the unencrypted message that is initially produced and can also be quite large. Potential for future project.
* Changed storage of required and optional Characteristics from unordered_set to vector
Significantly reduces memory usage - unordered_sets take up much more memory and using std::find from <algorithm> is as efficient as using find() method of an unordered_set.
To do: Remove all uses of unordered_set across HomeSpan wherever possible
* Changed unordered_set to vector when checking for duplicate AID
This eliminated the use of unordered_set everywhere in HomeSpan
* Added error checking for NVS storage
Also added NVS storage diagnostic to 'm' command.
* Fixing HTML syntax error in weblogs.
* PSRAM: Adds custom allocator and HS_MALLOC/HS_CALLOC macros
Forces all heap requests to be drawn from PSRAM, unless the device does not contain PSRAM.
To do: Provide easy-to-use NEW() macro; add custom allocator to unordered sets; auto-shrink vectors after updateDatabase()
* Added ps_new() macro
A more generic version of `new(*ptr)` that allocates from PSRAM if present, otherwise is that same a normal `new`
* Added HS_REALLOC macro for generalized realloc() when using PSRAM
When using PSRAM, changing the number of Accessories no longer creates any change to heap usage on internal RAM. The entire Accessory will be stored in PSRAM.
* Address WebLog size issue that limited number of entries
Added logic to sequentially transmit chunks of the HTML response String whenever it grows beyond 1024 bytes. This ensures the number of WebLog entries does not create a very large HTML String that eats up all the free heap space. WebLog entries are now only limited by storage of the entries themselves. Tested successfully with 500 entries.
* Change NVS storage for Characteristics to always use nvs_set_u64()
Reduces NVS consumption by from 3 to only 1 records when storing numerical values.
Also: Fixed memory problem in getCharacteristics by replacing dynamically-sized stack buffers with TempBuffer (must do this everywhere), which was causing stack probllem when combination number of Accessories and number of Characteristics got very large.
* Override new operator for SpanAccessory, SpanService, and SpanCharacteristic
New Operator now uses PSRAM if available, else uses Internal RAM. Obviates the need for the ps_new() macro, though will keep definition since it may be useful for any class that does not already override malloc() to use PSRAM.
* Streamlined NVS recall for String values
Avoids need for intermediate variable
* Create MaxAccessories.ino
Demonstrates use of custom partition and shows how PSRAM is used in background when available
* Create partitions.csv
* Changed various static and dynamic stack-based buffers to TempBuffer
Also added casting overload to TempBuffer so that get() method is no longer needed
* Eliminated .get() from TempBuffer objects where possible
Stylistic only. Cannot remove .get() when a TempBuffer is being re-cast. Also can't remove .get() when cast can't be determined, such as when used in Serial.printf() [potential to do: explore if this can be addressed, though this is stylistic only and has no impact on code]
* Added Characteristic Constants
Added constants to Characteristics that describe states.
Example: Characteristic::SecuritySystemTargetState::ARM_STAY
* Correction to time server URL doc
* Addressed issue with new Characteristic enum constants
* Update MaxAccessories.ino
* Added ability to turn off SpanPoint encryption
New class-level method SpanPoint::setEncryption(boolean encrypt). Default is to use encryption unless method is called with encrypt=false.
With encryption, 7 connections max; without encryption, 20 connections max.
Serial Monitor provides warnings if connections are exceeded.
* Update NOW.md
* Update README.md
* Update README.md
* Update README.md
* Added new TLV8 class
To do: Refactor HAP.cpp to use new TLV8 class
* Starting to integrate new TLV8 into postPairVerify
* Completed re-factoring of postPairVerify using new TLV8
Next: re-factor postPairSetup
* Preparing for re-factoring postPairSetup
* Starting re-factoring of postPairSetup
* added TLV length function
Checks that iterator exists (i.e. does not equal end()) and returns length, else returns -1. Can be used for checking for both the presence, and length, of a TLV record
* Completed initial re-factoring of postPairSetup - must test!
* Completed postPairSetup and fully tested - also changes TLV8 from vector to forward_list
Changed TLV8 from vector to forward_list to ensure iterators are always retained after adding new elements.
Nest: refactor postPairings
* Completed and fully tested refactoring of TLV8
Next up: remove old TLV code and clean up
* Removed all old TLV code
* Cleaning up postverify
* Added new initializer to TempBuffer
Allows TempBuffer to be initialized by concatenation of multiple existing buffers.
Constructor contains pairs of (T *buf), nElements, where last argument MUST BE NULL to signify end of variable argument list.
Will work with any type T, and nElements is in units of elements of type T.
* Start re-working SRP6A's use in Pair-Setup to make it local instead of global
* Initial integration of local SRP6A into Pair Setup
* Moved definitions of N3072 and other constants into SRP6A structure
* In process of updated Pair-Setup M3...
Cleaning up code by replacing calls to mbedTLS within Pair-Setup with arguments passed to/from SRP6A. Much better and better encapsulates SRP6A
* Completed Pair-Setup M3 update
* Completed refactoring of SRP6A and integration into all steps of Pair-Setup
* Remove include HAP.h from SRP
SRP now only requires Utils (for TempBuffer).
Also, moved HS_ALLOC definitions into Utils so it can be re-used by others files
* Initial test of StreamBuffer added to WebLog
* Moved Streamer from Utils.h into separate Streamer.cpp
But declared Streamer inside HomeSpan.h instead of separate Streamer.h since it should only be used when HomeSpan.h is used.
Also, changed class structure so that it is "final" by removing virtual tags and making all protected variables private.
* Begin integration of hapStream into HAP.ccp
* Moved StreamBuf parameter calls into HapOut
* Moved StreamBuf into HapOut as private nested class
* Added getSize() to HapOut and changed default behavior to NEVER output anything
Note: getSize() must be called before flush(), which resets byte counter
* Converted sprintfAttributes to printfAttributes and added encryption to HapOut
Success!
* Added SHA-384 hash computation to HapOut
Hashing of entire database is now done piecemeal within HapOut, preventing the need to load the entire database into a large buffer just to obtain a unique hash.
Modified updateDatabase() to use HapOut.getHash()
* Deleted sprintfAttributes for Span, SpanAccessory, and SpanService
These now use the new printfAttributes with HapOut
* Converted sprintfNotify and eventNotify to use HapOut
Successfully tested with MultiTest script
* Added back weblogCallback() functionality that uses hapOut
* Converted all remaining sprintfAttributes to printfAttributes
This removes last of large-sized JSON buffers - everything has now been converted to using hapOut.
Next up: Must add prettyPrint to hapOut to support 'd' command
* Moved prettyPrint into HapOut
This completes re-work of all JSON buffer memory management to use HAPOut.
Next up: Stress test with Max Accessories to see how many Accessories we can now handle.
* Changed HapOut delay from 1ms to 50ms for WiFi transmits. Seems to be much more stable.
* Update MaxAccessories.ino
* fix for PSRAM with new hapOut
Force hapOut to use internal memory only.
Tested on ESP32-S2 with/without PSRAM. However, crashes on ESP32-S3 when using PSRAM (?!)
* Changed hapOut delay back to 1ms - works fine
Identified issue with ESP32-S3 crashing when using PSRAM. Problem was not the code, but that the sketch was using pins that are needed to run the Octal PSRAM that comes with the Espressif S3 DevKit.
* Fixed CSS in weblog
Missing conversion from String to c_str() when using hapOut.
* Added homeSpan.getWebLog()
* Update src.ino
* Update src.ino
* started work on TLV streaming with << operator
Goal - eliminate sendEncrypted() completely by using hapOut in tlvRepond().
Next up: use function call instead of << operator so logic can be moved into TLV.h
* moved tlv streaming into tlv class
* refactored tlvRespond() to use hapOut instead of sendEncrypted()
To do:
1) update putPrepareURL() to use hapOut instead of sendEncrypted()
2) remove sendEncrypted() since it is no longer needed
3) try moving MAllocator template into separate file (memory.h?) so it can be called from any other module
4) remove #include "HomeSpan.h" from TLV8
* Moved all PSRAM definitions into separate PSRAM.h file
Now allows #include "HomeSpan.h" to be removed from TLV dependencies
* updated putPrepareURL and deleted receiveEncrypted()
Must test with Door example, which uses putPrepare
* Modified range checks so that "NAN" gets picked up as error for FLOAT Characteristics
* Truncate all STRING-based Characteristics to 64 characters when adding to JSON
HAP default max length of STRING Characteristics is 64 characters
* Added homeSpan.getAutoPollTask() to return task handler of autoPoll
Returns NULL if autoPoll has not been used.
Also added low stack size for both main Arduino loop task and HomeSpan autoPoll task to 'm' command
* Added optional triggerType to homeSpan.setControlPin()
Default is SpanButton::TRIGGER_ON_LOW. Follows same options as SpanButton, including the ability to provide your own triggerType function of form:
boolean f(int pin)
* Cleaned up some public/private permissioning inside PushButton and SpanButton
Ensure users can access constants like TRIGGER_ON_HIGH, TRIGGER_ON_LOW, etc. using any of these equivalent fully-qualified names:
* PushButton::TRIGGER_ON_HIGH
* SpanButton::TRIGGER_ON_HIGH
* SpanToggle::TRIGGER_ON_HIGH
* Update Reference.md
* Update Reference.md
* Added CustomNVSPartition Example
To do: Add to list of Other Examples in documentation.
* Update CustomNVSPartition.ino
* Update Tutorials.md
* Create ExternalReference.ino
* Update Tutorials.md
* Update Logging.md
* Update Reference.md
* New Services and Characteristics List (#751)
* auto-generation of service list
* Update makeServiceList
* Update makeServiceList
* Update test.md
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* Update Span.h
* Update test.md
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* update
* Create ServiceListHead.md
* Update ServiceListHead.md
* Update ServiceListHead.md
* Update test.md
* update
* update
* Update ServiceList.md
* update
* Update ServiceList.md
* update
* removed bold font for Characteristic Constants in ServiceList.md
* fix heavy_check_mark
* add <span> to ServiceList to prevent linebreaks before checkmark
* fixed typo in ServiceLabelNameSpace description
* Create ServistList_Expanded.md
* Update ServistList_Expanded.md
* Update ServiceList.md
* added ServiceList_Expanded
* added code to create pre-expanded Services and Characteristics List
* Update Tutorials.md
* Update Tutorials.md
* Added ConfiguredName to Switch and Lightbulb
* Moved error/warning output to end of 'i' output
* Add configuredName() to all applicable Services and deprecated Name()
* reformat ServiceList to remove expandable details
Expandable details are not as practical to navigate for a page this size. Easier to see everything laid out all at once.
* Re-organized Services List into Service Groups
* Update ServiceList.md
* converted Service Group names to upper case
* Added linebreak after each Service
* updated some descriptions
* Delete ServiceList_Expanded.md
* Revised Example 11 (Service Names) to use ConfiguredName Characteristic
Complete re-work of Example 11 - fully tested for iOS 17.2.1
* Documentation update (#765)
* Update Stepper.md
* Update README.md
* Update FAQ.md
* Update TVServices.md
* Get Arduino-ESP32 Version from <esp_arduino_version.h> instead of <core_version.h>
This works fine for all versions after 2.0.3, but returns 2.0.0 for versions 2.0.2 and 2.0.1, as well as 2.0.0. Not an issue since current version is 2.0.14.
* Update Solutions.md
* Added Stepper_UNIPOLAR and Stepper_UNL2003
ALSO: Re-arranged include files so that StepperControl.h is now part of HomeSpan.h. No need to include anything except for HomeSpan.h to use any of the Stepper Motor classes. Must update documentation accordingly.
NOTE: Always including StepperControl.h DOES NOT increase size of compile if no stepper motors are instantiated.
To do: Consider doing the same for Pixel, PWM, and RMT as well.
* Removed #include "extras/PwmPin.h" from all Tutorial Examples
Since PwmPin.h is already included in HomeSpan.h, there is no reason to separately include it in user sketches.
* Removed all #include "extras/*.h" from "Other Examples"
Including any extras/*.h files in a HomeSpan sketch is no longer necessary since these are already included in HomeSpan.h.
* Update PWM.md
* Update RMT.md
* Update Pixels.md
* Update RMT.md
* Update Stepper.md
* Update 17-LinkedServices.ino
* Update Stepper.md
* Update Stepper_TB6612.md
* Update Stepper_A3967.md
* Update Stepper.md
* Update Stepper.md
* Update Stepper.md
* Update StepperControl.h
* Update Stepper_A3967.md
* Create Stepper_ULN2003A.md
* Update Stepper_ULN2003A.md
* Update Stepper_ULN2003A.md
* Update Stepper_TB6612.md
* Update Stepper_A3967.md
* Create Stepper_UNIPOLAR
* Create Stepper_UNIPOLAR.md
* Update Stepper_UNIPOLAR.md
* Delete Stepper_UNIPOLAR
* Update Stepper_UNIPOLAR.md
* Update Stepper_UNIPOLAR.md
* Update Stepper_UNIPOLAR.md
* Update Stepper_UNIPOLAR.md
* Added option to specify NULL for 4th parameter of homeSpan.enableWebLog()
If 4th parameter (the statusURL) is NULL, web logging will NOT be enabled BUT time server will still acquire time if server address and time zone were specified as 2nd and 3rd parameters.
Note that not having web logging enabled only impacts ability to acquire weblog from HTTP request. It has no effect on the actual logging of WEBLOG() messages and thus homeSpan.getWebLog() works as expected.
* Allow user-defined data when receiving weblogs from callback (#770)
* Adding userData to getweblogs callback.
Allows user-specific data to be available in the callback that returns weblog HTML data.
* Adding userData to getweblogs callback.
Allows user-specific data to be available in the callback that returns weblog HTML data.
Return HTTP response code and content type only when no weblog callback defined
* Small clean-up and formatting of comments
* Bump version to 1.9.0
* Updated copyright dates to 2024
* Update README.md
* Update README.md
* Update README.md
* Update README.md
* Update README.md
* Update README.md
* Update Reference.md
* Update Logging.md
* Update README.md
* Update Logging.md
* Update README.md
* Update HomeSpan.h
* Update README.md
* Update FeatherPins.h
Changed F21 from pin 35 to pin 38 for ESP32-S3 to avoid conflicting with Octal PSRAM.
* Update README.md
* Update README.md
* Update Logging.md
* Update library.properties
---------
Co-authored-by: Yitao Jiang <jytscientist@hotmail.com>
Co-authored-by: Francois <franklamoureux@hotmail.com>
Co-authored-by: Michael Geramb <mail@michael-geramb.at>
Co-authored-by: frankonski <frankonski@users.noreply.github.com>
Allows user to define a custom style sheet to change the webLog output. Style sheet is based on pre-defined class:
bod1 = main body of web page, including header (in format of <h2>)
tabl1 = top table showing general status
tab2 = bottom table showing all weblog entries
* Changed ~SpanService to virtual function to ensure any user-defined destructor is also called
* Found/corrected bug in error message related to defining a Characteristic without any Service
* Added warning/error ID numbers to warning/error messages
* Other misc changes to clean up compiler warnings
Only initialize MDNS, WebLog, and OTA when first connecting to WiFi. Do not re-initialize upon re-connections after a disconnect.
Also, add number of disconnects, and reset_reason() to WebLog
Method homeSpan.updateDatabase(boolean updateMDNS) recomputes database hash and compares with previous hash. If changed, config number is updated and saved in NVS along with new hash. If updateMDNS is true, a new "c#" MDNS record is also broadcast with the new config number. This triggers HomeKit to call for a database refresh. Method returns true if config number has changed, or false if it has not.
The "C" command computes a hash of the current database config and updates the config number if needed. If config number has changed, the MDNS "c#" record is updated and the new config number is rebroadcast. This triggers HomeKit to immediately request an update of the HAP database so that the changes can be shortly reflected in the Home App without the need to close any Client connections or reboot the device.
The config number logic has also been updated/fixed. Previously it would create a hash from the full HAP database, which includes the value of each characteristic. Thus, initial value changes of Characteristics, as a result of stored values in NVS, would cause an update to the config number upon reboot.
This is not problematic, but also not intended, as a change in value is not really a change in the database config. The new logic computes a hash of the database that EXCLUDES all Characteristic values.
Gregg