25 questions
- Bountied 0
- Unanswered
- Frequent
- Score
- Trending
- Week
- Month
- Unanswered (my tags)
0
votes
1
answer
58
views
TPU for Pix-to-Pix possible?
I have model Pix-to-Pix. Not segmentation. Transfer from texture to texture. Loss = 'mse'.
And it works properly with GPU. But if i try to use TPU i get loss = 'nan'
I simplified the model by cutting ...
-1
votes
1
answer
145
views
Does SoX internally operate in Int32 precision, not in Float32? [closed]
I noticed that even when converting 24-bit Signed Integer PCM audio to Float32, clipping still occurs even after returning to the original gain level.
sox in.wav -b 32 -e floating-point out.wav
It ...
1
vote
2
answers
275
views
Float32 to float16 conversion in python and how it impacts np.isclose's atol
I am trying to compare each layer outputs of two ML models trained with different libs (tensorflow and a custom lighter one) that have the same valid performance in some sort of a "unitary test&...
1
vote
1
answer
296
views
Do bfloat types have any benefits over fp16 provided by VK_KHR_16bit_storage extension?
In vulkan api there are two extensions to use 16-bit types, namely VK_KHR_16bit_storage and VK_KHR_shader_float16_int8. So, if the hardware supports these extensions, fp16 type variables can be ...
2
votes
0
answers
260
views
Troubleshooting FP8 Conversion Discrepancy from Float32
I'm trying to convert a float32 to an fp8 float. There's some detail in the code that is causing a discrepancy in the results. I can't figure out what it is, could someone help me?
def ...
3
votes
0
answers
820
views
Clang tidy problem with _Float32, _Float64, _Float128
Why does clang-tidy give me warnings about _Float32, _Float64, _Float128? The program runs fine, but the squiggles appear even in the most basic program under the first character of any file (source ...
1
vote
2
answers
2k
views
Convert tflite model of float32 model to int8
Is it possible to convert a tflite float32 to tflite model of uint8? I read this page https://www.tensorflow.org/lite/performance/post_training_quantization, but they are only loading Keras of ...
2
votes
1
answer
99
views
The normal distribution for the computational error of the Gauss and sweep methods for solving systems of linear equations does not work
The methods for solving systems of linear equations themselves work correctly. Tested with solve. But when trying to solve a large number of matrices, some solution vectors do not converge with the ...
2
votes
1
answer
1k
views
Converting Pandas dataframe to float32 changes value of low precision number (by adding incorrect high precision decimals)
I have a very large dataset with values that don't require a lot of decimal point precision. In one test scenario, my dataframe is 102 MB, and all columns have a float64 datatype.
I was hoping to ...
0
votes
1
answer
136
views
javascript float32Array push changes value
I recently find out some weird behaviour in JS, and i might need some insight on it.
i have a js script that i need to translate into C#.
now, in the JS i have:
values = new Float32Array(n).fill(0) (...
0
votes
1
answer
168
views
Fast read/unpacking of float32 from int16 in Python
Say I have a Python script which reads some binary data, packed as int16. I want to convert this data to float32 as fast as possible.
Currently I am doing this, per file
data = np.fromfile(fid, 'int16'...
1
vote
1
answer
286
views
How to create a PyGLM array of float32 type from a list of python floats
I have this function to modify it:
def buffers_init_pnt() -> None:
global mPoints
global POINTS_VAOID
global POINTS_VBOID
global glmverts_pnt_init
floatlist = []
for item ...
1
vote
0
answers
983
views
OpenCV allowed data types
I am working with OpenCV in Python. I found that working with uint8 and float32 works in OpenCV 4. This question OpenCV Mat element types and their sizes shows some more data types but I got an error ...
0
votes
2
answers
8k
views
"Input contains NaN, infinity or a value too large for dtype('float32')There are no Nan or blanks in my data set but still wont let me proceed
Keep getting the error 'Input contains NaN, infinity or a value too large for dtype('float32')' when trying to run a random forest regressor model. I've searched my data set and cant see any infinite ...
0
votes
1
answer
114
views
Possible bug with inf or too large values?
I'm trying to train a neural network with keras and tesorflow. As usual, I replace -np.inf and np.inf values with np.nan to later run a dropna sequence and clear all that wrong data such as:
Data....