Handling Zeros or NaNs in a Pandas DataFrame operations












0















I have a DataFrame (df) like shown below where each column is sorted from largest to smallest for frequency analysis. That leaves some values either zeros or NaN values as each column has a different length. 



   08FB006 08FC001 08FC003 08FC005 08GD004

----------------------------------------------

0   253      872    256      11.80    2660

1   250      850    255      10.60    2510

2   246      850    241      10.30    2130

3   241      827    235      9.32     1970

4   241      821    229      9.17     1900

5   232       0     228      8.93     1840

6   231       0     225      8.05     1710

7   0         0     225       0       1610

8   0         0     224       0       1590

9   0         0      0        0       1590

10  0         0      0        0       1550


I need to perform the following calculation as if each column has different lengths or number of records (ignoring zero values). I have tried using NaN but for some reason operations on Nan values are not possible.



Here is what I am trying to do with my df columns : 



shape_list1=

location_list1=

scale_list1=



for column in df.columns:

    shape1, location1, scale1=stats.genpareto.fit(df[column])



    shape_list1.append(shape1)

    location_list1.append(location1)

    scale_list1.append(scale1)





python pandas nan zero







share|improve this question





























    0















    I have a DataFrame (df) like shown below where each column is sorted from largest to smallest for frequency analysis. That leaves some values either zeros or NaN values as each column has a different length. 



       08FB006 08FC001 08FC003 08FC005 08GD004
    
----------------------------------------------
    
0   253      872    256      11.80    2660
    
1   250      850    255      10.60    2510
    
2   246      850    241      10.30    2130
    
3   241      827    235      9.32     1970
    
4   241      821    229      9.17     1900
    
5   232       0     228      8.93     1840
    
6   231       0     225      8.05     1710
    
7   0         0     225       0       1610
    
8   0         0     224       0       1590
    
9   0         0      0        0       1590
    
10  0         0      0        0       1550


    I need to perform the following calculation as if each column has different lengths or number of records (ignoring zero values). I have tried using NaN but for some reason operations on Nan values are not possible.



    Here is what I am trying to do with my df columns : 



    shape_list1=
    
location_list1=
    
scale_list1=
    

    
for column in df.columns:
    
    shape1, location1, scale1=stats.genpareto.fit(df[column])
    

    
    shape_list1.append(shape1)
    
    location_list1.append(location1)
    
    scale_list1.append(scale1)





    python pandas nan zero







    share|improve this question



























      0












      0








      0








      I have a DataFrame (df) like shown below where each column is sorted from largest to smallest for frequency analysis. That leaves some values either zeros or NaN values as each column has a different length. 



         08FB006 08FC001 08FC003 08FC005 08GD004
      
----------------------------------------------
      
0   253      872    256      11.80    2660
      
1   250      850    255      10.60    2510
      
2   246      850    241      10.30    2130
      
3   241      827    235      9.32     1970
      
4   241      821    229      9.17     1900
      
5   232       0     228      8.93     1840
      
6   231       0     225      8.05     1710
      
7   0         0     225       0       1610
      
8   0         0     224       0       1590
      
9   0         0      0        0       1590
      
10  0         0      0        0       1550


      I need to perform the following calculation as if each column has different lengths or number of records (ignoring zero values). I have tried using NaN but for some reason operations on Nan values are not possible.



      Here is what I am trying to do with my df columns : 



      shape_list1=
      
location_list1=
      
scale_list1=
      

      
for column in df.columns:
      
    shape1, location1, scale1=stats.genpareto.fit(df[column])
      

      
    shape_list1.append(shape1)
      
    location_list1.append(location1)
      
    scale_list1.append(scale1)





      python pandas nan zero







      share|improve this question
















      I have a DataFrame (df) like shown below where each column is sorted from largest to smallest for frequency analysis. That leaves some values either zeros or NaN values as each column has a different length. 



         08FB006 08FC001 08FC003 08FC005 08GD004
      
----------------------------------------------
      
0   253      872    256      11.80    2660
      
1   250      850    255      10.60    2510
      
2   246      850    241      10.30    2130
      
3   241      827    235      9.32     1970
      
4   241      821    229      9.17     1900
      
5   232       0     228      8.93     1840
      
6   231       0     225      8.05     1710
      
7   0         0     225       0       1610
      
8   0         0     224       0       1590
      
9   0         0      0        0       1590
      
10  0         0      0        0       1550


      I need to perform the following calculation as if each column has different lengths or number of records (ignoring zero values). I have tried using NaN but for some reason operations on Nan values are not possible.



      Here is what I am trying to do with my df columns : 



      shape_list1=
      
location_list1=
      
scale_list1=
      

      
for column in df.columns:
      
    shape1, location1, scale1=stats.genpareto.fit(df[column])
      

      
    shape_list1.append(shape1)
      
    location_list1.append(location1)
      
    scale_list1.append(scale1)





      python pandas nan zero




      python pandas nan zero






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 22 '18 at 22:00







      Sina Shabani

















      asked Nov 22 '18 at 21:44









      Sina ShabaniSina Shabani

      687




      687
























          2 Answers
          2






          active

          oldest

          votes


















          1














          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])





          share|improve this answer



















          • 2





            Thanks both of these answers worked :)

            – Sina Shabani
            Nov 22 '18 at 22:13













          • @andersource's syntax is a lot cleaner than mine!

            – Peter Leimbigler
            Nov 22 '18 at 22:41



















          0














          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].






          share|improve this answer
























          • Thanks for the explanation :)

            – Sina Shabani
            Nov 22 '18 at 22:13











          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53438296%2fhandling-zeros-or-nans-in-a-pandas-dataframe-operations%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          2 Answers
          2






          active

          oldest

          votes








          2 Answers
          2






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          1














          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])





          share|improve this answer



















          • 2





            Thanks both of these answers worked :)

            – Sina Shabani
            Nov 22 '18 at 22:13













          • @andersource's syntax is a lot cleaner than mine!

            – Peter Leimbigler
            Nov 22 '18 at 22:41
















          1














          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])





          share|improve this answer



















          • 2





            Thanks both of these answers worked :)

            – Sina Shabani
            Nov 22 '18 at 22:13













          • @andersource's syntax is a lot cleaner than mine!

            – Peter Leimbigler
            Nov 22 '18 at 22:41














          1












          1








          1







          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])





          share|improve this answer













          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])






          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Nov 22 '18 at 22:06









          andersourceandersource

          51418




          51418








          • 2





            Thanks both of these answers worked :)

            – Sina Shabani
            Nov 22 '18 at 22:13













          • @andersource's syntax is a lot cleaner than mine!

            – Peter Leimbigler
            Nov 22 '18 at 22:41














          • 2





            Thanks both of these answers worked :)

            – Sina Shabani
            Nov 22 '18 at 22:13













          • @andersource's syntax is a lot cleaner than mine!

            – Peter Leimbigler
            Nov 22 '18 at 22:41








          2




          2





          Thanks both of these answers worked :)

          – Sina Shabani
          Nov 22 '18 at 22:13







          Thanks both of these answers worked :)

          – Sina Shabani
          Nov 22 '18 at 22:13















          @andersource's syntax is a lot cleaner than mine!

          – Peter Leimbigler
          Nov 22 '18 at 22:41





          @andersource's syntax is a lot cleaner than mine!

          – Peter Leimbigler
          Nov 22 '18 at 22:41













          0














          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].






          share|improve this answer
























          • Thanks for the explanation :)

            – Sina Shabani
            Nov 22 '18 at 22:13
















          0














          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].






          share|improve this answer
























          • Thanks for the explanation :)

            – Sina Shabani
            Nov 22 '18 at 22:13














          0












          0








          0







          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].






          share|improve this answer













          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].







          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Nov 22 '18 at 22:06









          Peter LeimbiglerPeter Leimbigler

          3,9081415




          3,9081415













          • Thanks for the explanation :)

            – Sina Shabani
            Nov 22 '18 at 22:13



















          • Thanks for the explanation :)

            – Sina Shabani
            Nov 22 '18 at 22:13

















          Thanks for the explanation :)

          – Sina Shabani
          Nov 22 '18 at 22:13





          Thanks for the explanation :)

          – Sina Shabani
          Nov 22 '18 at 22:13


















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53438296%2fhandling-zeros-or-nans-in-a-pandas-dataframe-operations%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          -

          Popular posts from this blog

          Create new schema in PostgreSQL using DBeaver

          Image
          0 At the beginning of my current job I had limited access to DB but recently my role is changed to superuser, admin, and owner. My colleague who was the previous DB admin is able to create a schema or change the permissions as there is a permission tab in the properties and he can change permissions by clicking easily. Would you please let me know how I can create schema and change permissions as well? Why his dbeaver looks different from me? He can easily right click on the schema folder on top of all schemas on the left side column and select create new schema while I do not have that, like the following picture I found on the internet. postgresql database-schema dbeaver share | improve this question
          Read more

          Deepest pit of an array with Javascript: test on Codility

          Image
          0 $begingroup$ This is the question from a Codility test, as a task in an interview: DeepestPit - problem description A non-empty zero-indexed array A consisting of N integers is given. A pit in this array is any triplet of integers (P, Q, R) such that: ·         0 ≤ P < Q < R < N; ·         sequence [A[P], A[P+1], ..., A[Q]] is strictly decreasing, i.e. A[P] > A[P+1] > ... > A[Q]; ·         sequence A[Q], A[Q+1], ..., A[R] is strictly increasing, i.e. A[Q] < A[Q+1] < ... < A[R]. The depth of a pit (P, Q, R) is the number min {A[P] − A[Q], A[R] − A[Q]}. For example, consider array A consisting of 10 elements such that: A[0] = 0 A[1] = 1 A[2] = 3 A[3] = -2 A[4] = 0 A[5] = 1 A[6] = 0 A[7] = -3 A[8] = 2 A[9] = 3 Triplet (2, 3, 4) is
          Read more

          Costa Masnaga

          Image
          .mw-parser-output .avviso .mbox-text-div>div,.mw-parser-output .avviso .mbox-text-full-div>div{font-size:90%}.mw-parser-output .avviso .mbox-image div{width:52px}.mw-parser-output .avviso .mbox-text-full-div .hide-when-compact{display:block} Questa voce o sezione sull'argomento Lombardia non cita le fonti necessarie o quelle presenti sono insufficienti . Puoi migliorare questa voce aggiungendo citazioni da fonti attendibili secondo le linee guida sull'uso delle fonti. Segui i suggerimenti del progetto di riferimento. Costa Masnaga comune Localizzazione Stato   Italia Regione Lombardia Provincia Lecco Amministrazione Sindaco Sabina Panzeri (Al centro il cittadino) dall'08/06/2009 Territorio Coordinate 45°46′N 9°17′E  /  45.766667°N 9.283333°E 45.766667; 9.283333  ( Costa Masnaga ) Coordinate: 45°46′N 9°17′E  /  45.766667°N 9.283333°E 45.766667; 9.283333  ( Costa Masnaga ) Altitudine 318 m s.l.m
          Read more