Quantcast
Channel: Teradata Forums - Database
Viewing all articles
Browse latest Browse all 14773

Precision loss during expression evaluation - data type issue? - response (2) by tstrick4

$
0
0

Thanks for your help!
The data profile is true to the data type.  MVT_Gravity is the specific gravity of a substance so I need to take that all the way to 6 decimal places.  
MAX net_quant = 642,000.00
MIN net_qant = -333,200.00
MAX MVT_gravity = 1.797967
MIN MVT_gravity = 0.506400
As this is a conversion formula, those constants (8.338426855 and 0.0101578) need to stay at that level of precision for the calculation, then round my final answer to 4 decimals.  These are petroleum inventories so there are pretty strict standards regarding decimal precision.


Viewing all articles
Browse latest Browse all 14773

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>