Merge branch 'feature/add-garbage-collection' into typo3-13

ChangeLog

@@ -8,8 +8,10 @@ Upcoming version for TYPO3 13
 [TASK]    Don't use QueryBuilder->add() anymore because it has been removed, https://docs.typo3.org/c/typo3/cms-core/main/en-us/Changelog/13.0/Breaking-102875-QueryBuilderChanges.html
 
 Upcoming version
+[FEATURE] Add garbage collection for statistics table
 [BUGFIX]  Use correct table names for query builder or connection. Thanks to Andreas Kießling. https://github.com/tpwd/ke_search/issues/228
 [TASK]    Add check if MySQL indexes are enabled on the index table and show an error in the backend module if not
+[BUGFIX]  Respect configured Doktypes while Auto-Tagging #235. Thanks to magicHatOfTYPO3. https://github.com/tpwd/ke_search/pull/235
 
 Version 5.5.0, 17 May 2024
 [FEATURE] Add pagination in the backend module "Indexed content" function and avoid out of memory error. https://github.com/tpwd/ke_search/issues/100

Classes/Indexer/Types/Page.php

@@ -249,6 +249,13 @@ public function startIndexing()
         // excluded (see: http://forge.typo3.org/issues/49435)
         $where = ' (doktype = 1 OR doktype = 2 OR doktype = 4 OR doktype = 5 OR doktype = 254) ';
 
+        // respect configured doktypes but ensure it's only a commaseparated list of integer values to avoid a SQL injection.
+        if (!empty($this->indexerConfig['index_page_doctypes'])) {
+            $indexerConfigDoktypes = GeneralUtility::intExplode(',', $this->indexerConfig['index_page_doctypes'], true);
+            $dokTypes = implode(',', array_unique(array_merge([1, 2, 4, 5, 254], $indexerConfigDoktypes)));
+            $where = 'doktype in (' . $dokTypes . ')';
+        }
+
         // add the tags of each page to the global page array
         $this->addTagsToRecords($indexPids, $where);
 

Documentation/Statistics/Index.rst

@@ -6,21 +6,24 @@
 Statistics
 ==========
 
-All the search words that are submitted by frontend users are stored in statistic tables. This function is activated
-by default but can be deactivated in the plugin configuration.
+All the search words that are submitted by frontend users are stored in
+statistic tables. This function is activated by default but can be deactivated
+in the plugin configuration.
 
 .. figure:: /Images/Statistics/statistics-1.png
    :alt: Statistics checkbox in plugin
    :class: with-border
 
-You can see the statistics in the backend module by selecting :guilabel:`Searchword statistics` or by using the
+You can see the statistics in the backend module by selecting
+:guilabel:`Searchword statistics` or by using the
 :ref:`dashboard widget <widget-trending-search-phrases>`.
 
-The dashboard widget shows the search phrases used in the last seven days for the whole system (ignoring the folders in
-which the data is stored).
+The dashboard widget shows the search phrases used in the last seven days for
+the whole system (ignoring the folders in which the data is stored).
 
-The backend module function shows a simple statistic of the submitted searchwords of the last 30 days. The statistic shows the
-cumulative values for single searchwords. Maybe this will be extended in a later version.
+The backend module function shows a simple statistic of the submitted
+searchwords of the last 30 days. The statistic shows the cumulative values for
+single searchwords. Maybe this will be extended in a later version.
 
 .. figure:: /Images/Statistics/statistics-2.png
    :alt: Searchword statistics in backend module
@@ -29,20 +32,45 @@ cumulative values for single searchwords. Maybe this will be extended in a later
 Differences between folders and other pages
 ===========================================
 
-If you call the statistics function for a folder you will get the cumulative values
-for all statistic data that is stored there. If you call it for a page of another type you
-will get the cumulative values for searchwords that were entered on this explicit page.
+If you call the statistics function for a folder you will get the cumulative
+values for all statistic data that is stored there. If you call it for a page of
+another type you will get the cumulative values for searchwords that were
+entered on this explicit page.
 
 Technical background
 ====================
 
-Search phrases go to the table `tx_kesearch_stat_search`. Single search words are stored
-in the table `tx_kesearch_stat_words`.
+Search phrases go to the table `tx_kesearch_stat_search`. Single search words
+are stored in the table `tx_kesearch_stat_words`.
 
-The statistic data is stored in the folder that is set as the first storage point in your plugin configuration.
-Make sure that you set the FlexForm configuration option for activating the statistics function
-in the correct plugin if it does not work as expected. E.g. if you have several searchbox plugins
-that point to one central search result page, the value must be set on this result page.
+The statistic data is stored in the folder that is set as the first storage
+point in your plugin configuration. Make sure that you set the FlexForm
+configuration option for activating the statistics function in the correct
+plugin if it does not work as expected. E.g. if you have several searchbox
+plugins that point to one central search result page, the value must be set on
+this result page.
+
+Garbage collection
+==================
+
+Since version 5.5.1 the statistic tables are registered for garbage
+collection. This means that the garbage collection scheduler task will delete
+old entries from the tables. The default value for the garbage collection is
+180 days. You can change this value in the scheduler task configuration.
+
+This is useful if you use the "Autocomplete" feature of ke_search_premium
+because the autocomplete function uses the statistic tables to propose search
+words. If you have a lot of old data in the tables, the autocomplete function
+will propose old search words that are not relevant anymore or the performance
+of the autocomplete function will decrease.
+
+To activate garbage collection, please add two scheduler tasks of type
+`Table garbage collection` and select the tables `tx_kesearch_stat_search` and
+`tx_kesearch_stat_words`.
+
+.. figure:: /Images/Statistics/statistics-table-garbage-collection.png
+   :alt: Table garbage collection scheduler task
+   :class: with-border
 
 More options
 ============

Resources/Private/Language/locallang_mod.xlf

@@ -67,7 +67,7 @@
 				<source>Tag has been updated</source>
 			</trans-unit>
 			<trans-unit id="tag_has_been_updated_message" resname="tag_has_been_updated_message" >
-				<source>The given tag &quot;%s&quot; has been updated to &quot;%s&quot; because it did not match the criteria for a tag (length, allowed charachters, not in list of stop words).</source>
+				<source>The given tag &quot;%s&quot; has been updated to &quot;%s&quot; because it did not match the criteria for a tag (length, allowed characters, not in list of stop words).</source>
 			</trans-unit>
 		</body>
 	</file>

ext_localconf.php

@@ -93,4 +93,17 @@
         $GLOBALS['TYPO3_CONF_VARS']['FE']['cacheHash']['excludedParameters'],
         ['^tx_kesearch_pi1']
     );
+
+    // register statistics tables for garbage collection
+    // see https://docs.typo3.org/c/typo3/cms-scheduler/main/en-us/Installation/BaseTasks/Index.html#table-garbage-collection-task-example
+    if (\TYPO3\CMS\Core\Utility\ExtensionManagementUtility::isLoaded('scheduler')) {
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['scheduler']['tasks'][\TYPO3\CMS\Scheduler\Task\TableGarbageCollectionTask::class]['options']['tables']['tx_kesearch_stat_search'] = [
+            'dateField' => 'tstamp',
+            'expirePeriod' => '180', // days
+        ];
+        $GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['scheduler']['tasks'][\TYPO3\CMS\Scheduler\Task\TableGarbageCollectionTask::class]['options']['tables']['tx_kesearch_stat_word'] = [
+            'dateField' => 'tstamp',
+            'expirePeriod' => '180', // days
+        ];
+    }
 })();