No announcement yet.

Autoplay Calculates expressions wrong

  • Filter
  • Time
  • Show
Clear All
new posts

  • Autoplay Calculates expressions wrong

    Example: in Autoplay 8003203833 * 12 = 1549165484
    But in reality 8003203833 * 12 = 96038445996

    Seems there is a cap in the size it can process. I was playing with NIC Addresses when I came across this.

  • #2
    Re: Autoplay Calculates expressions wrong

    I believe this was mentioned before when someone made a calculator. [img]/ubbthreads/images/icons/smile.gif[/img]
    = Derek
    ["All glory comes from daring to begin" - fortune cookie]


    • #3
      Re: Autoplay Calculates expressions wrong

      It isn't calculating the expression wrong, exactly. [img]/ubbthreads/images/icons/smile.gif[/img] It's just limited to signed integers, which forces all the values to fit inside 32 bits, with 1 bit used for a sign flag (+/-). The largest value it can handle is plus or minus 2 the the 31st power, i.e. –2,147,483,648 to 2,147,483,647.

      Looking at the code, I see a workaround, however. (Kind of odd that I didn't realize this before.) Real numbers are handled as doubles internally, which gives you 1.7E +/- 308. (Technically I think it gives you 15 digits of precision, but in any case it's a really big number.)

      So, if you add a decimal place to one of your numbers, you can perform calculations on much larger values. I just tested it, and yep:

      8003203833 * 12.0 = 96038445996.0

      You can use a "Variable - Format Number" action to remove the decimal place from the result. (Just set the decimal places to 0 and set it to Truncate.)
      --[[ Indigo Rose Software Developer ]]


      • #4
        Re: Autoplay Calculates expressions wrong

        Thanks Lorne,
        I found that out while playing with Truncate & Rounding.