Propose increase size of stockmaster StockID and Description

[dalers]

I'm still working through importing stock items but an issue has arisen with the stock ID (part number), description (title) and long description (detail) being imported are larger than the webERP column sizes.

The maximum column widths in the data being imported are:

• StockID: 36 characters vs 20 in webERP (50 char absolute max in Parts&Vendors)
• Description: 100 characters vs 50 in webERP (255 char absolute max in Parts&Vendors)

LongDescription is "text" which iiuc is 65K bytes, which is PNDetail in Parts&Vendors and has an max of 255 char so no issue. Fwiw, the absolute max for PNPartNumber (StockID) is 50 char and PNTitle (Description) is 255 char.

Q1. Are there any technical reasons why these column widths can't simply be increased to accomodate the data being imported? Would the only issues be table and report formating when scripts assume data will be shorter?

Q2. If there are no insurmountable technical reasons, would you be in favor of (or tolerant of) allowing the larger sizes in webERP?

Cheers,
Dale

Fwiw, posts that are somewhat related and in a general PLM theme include:

• Z_ImportPartCodes.php fails Z_ImportPartCodes.php fails #741

• Schema is missing an Item "Notes" field Schema is missing an Item "Notes" field #592

• How to Import BOMs How to Import BOMs #591

• What is purpose of Manufacturers and Brand Management? What is purpose of Manufacturers and Brand Management? #668

• BOM item quantity needs more decimal places for accuracy BOM item quantity needs more decimal places for accuracy #702

• Sales Orden Items with CSV with obsolete items error
  (specifically Sales Orden Items with CSV with obsolete items error #808 (comment))

[dalers]

Fyi, I just discovered a stock ID can't include a "space" character, That calls for the classic "space to underscore" global replacement!

[pakricard]

Q1. Are there any technical reasons why these column widths can't simply be increased to accomodate the data being imported? Would the only issues be table and report formating when scripts assume data will be shorter?

I can't foresee any technical issue, besides that StockID appears in many tables and a zillion queries and reports. All of them should be checked. It is not a complex change, but a long and tedious one.

Q2. If there are no insurmountable technical reasons, would you be in favor of (or tolerant of) allowing the larger sizes in webERP?

OK for me. As a reference, OpenCart uses model (our StockID) as varchar(64) and product name (our description) as varchar(255).
Our long description should be kept as text, to allow long, detailed, technical descriptions.

R

[dalers]

@pakricard

Our long description should be kept as text, to allow long, detailed, technical descriptions.

Interesting. Iiuc LongDescription is 65K char (8-bit, less if multi-byte) and you are using it more for ad hoc notes than a "long description" per se (e.g. the "Long Description" that might be required on a shipping document). My initial expectation was that LongDescription might be at most 5x the Description, or max 100 char. However, it seems at least for you, LongDescription can be MUCH longer.

Does anyone know off the top of their head if there are scripts that assume LongDescription is much shorter, or assume a specific purpose?

If everyone is treating LongDescription as 65K of ad hoc text, perhaps the legend in the GUI should be "Notes" instead of "Long Description" to better imply the intended use. What do you think?

[pakricard]

Hi:
We are using this field to store a longer description that the description, and some technical details. Mainly what appears on any website as product description.

[dalers]

Back to stockid, using ChangeColumnSize() to change stockid results in a foreign key error at least on MariaDb.

I see other functions in UpgradeDB_mariadb.php disable foreign key checking, should CangeColumnSize() do this also or is there a better way?

[dalers]

I wrapped the query to change the column size in ChangeColumnSize() with DB_[Ignore|Reinstate]ForeignKeys() but still get error:

Warning: mysqli_query(): (HY000/1833): Cannot change column 'stockid': used in a foreign key constraint 'weberp_kayden/ custitem _ibfk_1' of table 'weberp_kayden/custitem' in C:\xampp\htdocs\weberp\includes\ConnectDB_mariadb.php on line 63

Any suggestions? Fwiw I'm using MariaDB 10.4.

function ChangeColumnSize($Column, $Table, $Type, $Null, $Default, $Size) {
	$SQL = "SELECT CHARACTER_MAXIMUM_LENGTH
		FROM information_schema.columns
		WHERE TABLE_SCHEMA='" . $_SESSION['DatabaseName'] . "'
			AND TABLE_NAME='" . $Table . "'
			AND COLUMN_NAME='" . $Column . "'";
	$Result = DB_query($SQL);
	$MyRow = DB_fetch_row($Result);
	if ($MyRow[0] <> $Size) {
		DB_IgnoreForeignKeys();
		$Response = executeSQL("ALTER TABLE " . $Table . " CHANGE COLUMN " . $Column . " " . $Column . " " . $Type . " " . $Null . " DEFAULT '" . $Default . "'", false);
		DB_ReinstateForeignKeys();
		if ($Response == 0) {
			OutputResult(__('The column') . ' ' . $Column . ' ' . __('has been changed'), 'success');
		} else {
			OutputResult(__('The column') . ' ' . $Column . ' ' . __('could not be changed'), 'error');
		}
	} else {
		OutputResult(__('The column') . ' ' . $Column . ' ' . __('is already changed'), 'info');
	}
}

My update file looks like this (34.php). Note ChangeColumnSize() for description works.

<?php

// increase StockID to 64 char
ChangeColumnSize('stockid', 'stockmaster', 'VARCHAR(64)', ' NOT NULL ', '', '64');
// increase stock Description to 255 char
ChangeColumnSize('description', 'stockmaster', 'VARCHAR(255)', ' NOT NULL ', '', '255');

if ($_SESSION['Updates']['Errors'] == 0) {
	UpdateDBNo(basename(__FILE__, '.php'), __('Increase size of stockID and Description'));
}

After the update runs:

[dalers]

Can anyone say why the foreign key containts in the custitem table have embedded spaces in their names?

Iiuc these are the ONLY foreign key constraints with spaces in their names and are, in the least, confusing special cases:

• "[space]custitem[space]_ibfk_1"
• "[space]custitem[space]_ibfk_2"

Would it be possible to remove the spaces? What would be involved?

weberp/install/sql/tables/custitem.sql:

CREATE TABLE `custitem` (
  `debtorno` char(10) NOT NULL DEFAULT '',
  `stockid` varchar(20) NOT NULL DEFAULT '',
  `cust_part` varchar(20) NOT NULL DEFAULT '',
  `cust_description` varchar(30) NOT NULL DEFAULT '',
  `customersuom` char(50) NOT NULL DEFAULT '',
  `conversionfactor` double NOT NULL DEFAULT '1',
  PRIMARY KEY (`debtorno`,`stockid`),
  KEY `StockID` (`stockid`),
  KEY `Debtorno` (`debtorno`),
  CONSTRAINT ` custitem _ibfk_1` FOREIGN KEY (`stockid`) REFERENCES `stockmaster` (`stockid`),
  CONSTRAINT ` custitem _ibfk_2` FOREIGN KEY (`debtorno`) REFERENCES `debtorsmaster` (`debtorno`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

[pakricard]

I think it is a typo error that should be corrected.
It comes from commit 81252b5 last August.

R

[dalers]

Thanks @pakricard for investigating. I followd the commit back and iiuc the typo(?) was in the initial commit of custitem.sql and the new installer.

@timschofield, since this was your code, do you see any reason not to "correct" the fk names and remove the spaces?

[dalers]

iiuc, I suspect this also means databases migrated forwards have custitem table fk's without spaces in their names and only databases created with the installer have the extra spaces.

I suspect though this difference is benign and no one notices. I stumbled across the difference wanting to change the size of stockid and description columns, but it's only an annoyance - but a significant annoyance, especially because I'm also discovering different tools made hide or deal with leading spaces differently,

[dalers]

Edit. Made original post a reply to @pakricard's instead to get back to the original topic - increasing stockid and description sizes.

It seems I was naive thinking the only table with a foreign key to the stockmaster stockid column was the custitem table.

After copy/pasting the name of the fk directly from mariadb cli output I was able to drop the constraint from the custitem table but then attempting to change the size of stockid resulted in an fk error in the bom table. After investigating, it seems there are many tables containing an explit fk relationship to stockmaster stockid.

Btw, I don't know if the character at issue was actually a space or if it only appeared as a space and was actually e.g. a tab character.

I don't think it would be difficult to use an upgrade xxx.php file to delete all the fk constraints in all the tables, adjust the sizes of parent and child tables and then re-create the fks, but before doing that, can anyone think of any issues? @pakricard @timschofield? Can it be presumed that all scripts that worked with a shorter stockid and description will continue to work as as expected when the stockid and description are shorter (i.e. comply with the current column size)?

If there is no technical reason why stockid and description size can't be increased (albeit will be more work than originally expected), I think the choice comes down to either:

• increase the size of master data that already exists to support data being imported from a PLM data system and data being exported to an e-store data
• add new columns for "long" stockid and description (e.g. stockid_long and description_long) and update scripts as needed to use the new long columns.

Larger values makes webERP more versatile which presumably adds value, while adding new "long" columns (which I a suspect is a well-known pattern and not something I just dreamed up) creates an additional layer of indirection, requires more scripts to be updated and increases the chaos factor.

What do you think? WDYT?

Drop fk's

MariaDB [weberp_kayden]> alter table custitem drop foreign key ` custitem _ibfk_2`;
Query OK, 0 rows affected (0.007 sec)
Records: 0  Duplicates: 0  Warnings: 0

MariaDB [weberp_kayden]> alter table custitem drop foreign key ` custitem _ibfk_1`;
Query OK, 0 rows affected (0.014 sec)
Records: 0  Duplicates: 0  Warnings: 0

Try to change stockid size again

    -> CHANGE COLUMN stockid stockid VARCHAR(64) NOT NULL DEFAULT '';
ERROR 1833 (HY000): Cannot change column 'stockid': used in a foreign key constraint 'weberp_kayden/bom_ibfk_1' of table 'weberp_kayden/bom'

What? How many tables do have a foreign key constraint to stockid?
(note the custitem table with its two constraints is not included below because I have already deleted them).

MariaDB [weberp_kayden]> SELECT
    ->     TABLE_NAME AS 'Referencing Table',
    ->     COLUMN_NAME AS 'Foreign Key Column',
    ->     CONSTRAINT_NAME AS 'Constraint Name'
    -> FROM
    ->     information_schema.KEY_COLUMN_USAGE
    -> WHERE
    ->     REFERENCED_TABLE_NAME = 'stockmaster'
    ->     AND REFERENCED_COLUMN_NAME = 'stockid'
    ->     AND TABLE_SCHEMA = DATABASE(); -- Limits results to your current webERP database
+-----------------------------+--------------------+------------------------------------+
| Referencing Table           | Foreign Key Column | Constraint Name                    |
+-----------------------------+--------------------+------------------------------------+
| bom                         | parent             | bom_ibfk_1                         |
| bom                         | component          | bom_ibfk_2                         |
| contractbom                 | stockid            | contractbom_ibfk_3                 |
| locstock                    | stockid            | locstock_ibfk_2                    |
| loctransfers                | stockid            | loctransfers_ibfk_3                |
| mrpdemands                  | stockid            | mrpdemands_ibfk_2                  |
| offers                      | stockid            | offers_ibfk_2                      |
| orderdeliverydifferenceslog | stockid            | orderdeliverydifferenceslog_ibfk_1 |
| pickreqdetails              | stockid            | pickreqdetails_ibfk_1              |
| prices                      | stockid            | prices_ibfk_1                      |
| purchdata                   | stockid            | purchdata_ibfk_1                   |
| recurrsalesorderdetails     | stkcode            | recurrsalesorderdetails_ibfk_2     |
| salescatprod                | stockid            | salescatprod_ibfk_1                |
| salesorderdetails           | stkcode            | salesorderdetails_ibfk_2           |
| stockcheckfreeze            | stockid            | stockcheckfreeze_ibfk_1            |
| stockcounts                 | stockid            | stockcounts_ibfk_1                 |
| stockitemproperties         | stockid            | stockitemproperties_ibfk_1         |
| stockitemproperties         | stockid            | stockitemproperties_ibfk_3         |
| stockitemproperties         | stockid            | stockitemproperties_ibfk_5         |
| stockmoves                  | stockid            | stockmoves_ibfk_1                  |
| stockrequestitems           | stockid            | stockrequestitems_ibfk_2           |
| stockrequestitems           | stockid            | stockrequestitems_ibfk_4           |
| stockserialitems            | stockid            | stockserialitems_ibfk_1            |
| woitems                     | stockid            | woitems_ibfk_1                     |
| worequirements              | stockid            | worequirements_ibfk_2              |
+-----------------------------+--------------------+------------------------------------+
25 rows in set (0.004 sec)

MariaDB [weberp_kayden]>

[dalers]

P.S. There are 35 tables that have a "stockid" column, which I presume includes the 26 tables with an explicit foreign key constraint and means there are an additional 9 tables with an implicit foreign key relationship (treated as an fk by webERP code but without an explicit constraint).

Does my analysis seem correct?

MariaDB [weberp_kayden]> SELECT
    ->     TABLE_NAME,
    ->     COLUMN_NAME,
    ->     DATA_TYPE
    -> FROM
    ->     information_schema.COLUMNS
    -> WHERE
    ->     COLUMN_NAME = 'stockid'
    ->     AND TABLE_SCHEMA = DATABASE()
    ->     AND TABLE_NAME != 'stockmaster';
+------------------------------+-------------+-----------+
| TABLE_NAME                   | COLUMN_NAME | DATA_TYPE |
+------------------------------+-------------+-----------+
| assetmanager                 | stockid     | varchar   |
| contractbom                  | stockid     | varchar   |
| custitem                     | stockid     | varchar   |
| ediitemmapping               | stockid     | varchar   |
| employees                    | stockid     | varchar   |
| lastcostrollup               | stockid     | char      |
| locstock                     | stockid     | varchar   |
| loctransfercancellations     | stockid     | varchar   |
| loctransfers                 | stockid     | varchar   |
| mrpdemands                   | stockid     | varchar   |
| offers                       | stockid     | varchar   |
| orderdeliverydifferenceslog  | stockid     | varchar   |
| pickreqdetails               | stockid     | varchar   |
| pickserialdetails            | stockid     | varchar   |
| pricematrix                  | stockid     | varchar   |
| prices                       | stockid     | varchar   |
| purchdata                    | stockid     | char      |
| relateditems                 | stockid     | varchar   |
| salesanalysis                | stockid     | varchar   |
| salescatprod                 | stockid     | varchar   |
| sellthroughsupport           | stockid     | varchar   |
| shipmentcharges              | stockid     | varchar   |
| stockcheckfreeze             | stockid     | varchar   |
| stockcounts                  | stockid     | varchar   |
| stockdescriptiontranslations | stockid     | varchar   |
| stockitemproperties          | stockid     | varchar   |
| stockmoves                   | stockid     | varchar   |
| stockrequestitems            | stockid     | varchar   |
| stockserialitems             | stockid     | varchar   |
| stockserialmoves             | stockid     | varchar   |
| supplierdiscounts            | stockid     | varchar   |
| tenderitems                  | stockid     | varchar   |
| woitems                      | stockid     | char      |
| worequirements               | stockid     | varchar   |
| woserialnos                  | stockid     | varchar   |
+------------------------------+-------------+-----------+
35 rows in set (0.016 sec)

[pakricard]

Sounds right to me. Some tables use stkcode to refer to stockid (I guess for historical reasons, some developers preferred one or another).

Probably is time to unify all synonyms to stockid. Probably also unify all of them to varcahr, not char, to avoid type mismatches as PHP gets stricter on this issue at every new version.

R

[dalers]

Probably is time to unify all synonyms to stockid. Probably also unify all of them to varcahr, not char, to avoid type mismatches as PHP gets stricter on this issue at every new version.

It would not be difficult to make the types consistent (and I will do so in the PR, assuming we continue on this path), but imho there isn't enough benefit to unify the synonyms - there are only two occurances of stkcode and the fk's in the bom table ("parent" and "component") need different names to identify use in the context of a BOM.

[dalers]

Also, there could be more implicit fk relationships where the child column is something other than "stockid". I expect the likelihood of this is low enough to not demand an exhaustive review, but still there is no guarantee.

[dalers]

Fyi, found what appears to be the stockid foreign keys in two tables themselves being used as foreign keys in other tables. #835

Warning: mysqli_query(): (HY000/1833): Cannot change column 'stockid': used in a foreign key constraint 'weberp_kayden/stockserialmoves_ibfk_2' of table 'weberp_kayden/stockserialmoves' in C:\xampp\htdocs\weberp\includes\ConnectDB_mariadb.php on line 63

Warning: mysqli_query(): (HY000/1833): Cannot change column 'stockid': used in a foreign key constraint 'worequirements_ibfk_3' of table 'weberp_kayden.worequirements' in C:\xampp\htdocs\weberp\includes\ConnectDB_mariadb.php on line 63