Handling Zeros or NaNs in a Pandas DataFrame operations












0















I have a DataFrame (df) like shown below where each column is sorted from largest to smallest for frequency analysis. That leaves some values either zeros or NaN values as each column has a different length. 



   08FB006 08FC001 08FC003 08FC005 08GD004

----------------------------------------------

0   253      872    256      11.80    2660

1   250      850    255      10.60    2510

2   246      850    241      10.30    2130

3   241      827    235      9.32     1970

4   241      821    229      9.17     1900

5   232       0     228      8.93     1840

6   231       0     225      8.05     1710

7   0         0     225       0       1610

8   0         0     224       0       1590

9   0         0      0        0       1590

10  0         0      0        0       1550


I need to perform the following calculation as if each column has different lengths or number of records (ignoring zero values). I have tried using NaN but for some reason operations on Nan values are not possible.



Here is what I am trying to do with my df columns : 



shape_list1=

location_list1=

scale_list1=



for column in df.columns:

    shape1, location1, scale1=stats.genpareto.fit(df[column])



    shape_list1.append(shape1)

    location_list1.append(location1)

    scale_list1.append(scale1)





python pandas nan zero







share|improve this question





























    0















    I have a DataFrame (df) like shown below where each column is sorted from largest to smallest for frequency analysis. That leaves some values either zeros or NaN values as each column has a different length. 



       08FB006 08FC001 08FC003 08FC005 08GD004
    
----------------------------------------------
    
0   253      872    256      11.80    2660
    
1   250      850    255      10.60    2510
    
2   246      850    241      10.30    2130
    
3   241      827    235      9.32     1970
    
4   241      821    229      9.17     1900
    
5   232       0     228      8.93     1840
    
6   231       0     225      8.05     1710
    
7   0         0     225       0       1610
    
8   0         0     224       0       1590
    
9   0         0      0        0       1590
    
10  0         0      0        0       1550


    I need to perform the following calculation as if each column has different lengths or number of records (ignoring zero values). I have tried using NaN but for some reason operations on Nan values are not possible.



    Here is what I am trying to do with my df columns : 



    shape_list1=
    
location_list1=
    
scale_list1=
    

    
for column in df.columns:
    
    shape1, location1, scale1=stats.genpareto.fit(df[column])
    

    
    shape_list1.append(shape1)
    
    location_list1.append(location1)
    
    scale_list1.append(scale1)





    python pandas nan zero







    share|improve this question



























      0












      0








      0








      I have a DataFrame (df) like shown below where each column is sorted from largest to smallest for frequency analysis. That leaves some values either zeros or NaN values as each column has a different length. 



         08FB006 08FC001 08FC003 08FC005 08GD004
      
----------------------------------------------
      
0   253      872    256      11.80    2660
      
1   250      850    255      10.60    2510
      
2   246      850    241      10.30    2130
      
3   241      827    235      9.32     1970
      
4   241      821    229      9.17     1900
      
5   232       0     228      8.93     1840
      
6   231       0     225      8.05     1710
      
7   0         0     225       0       1610
      
8   0         0     224       0       1590
      
9   0         0      0        0       1590
      
10  0         0      0        0       1550


      I need to perform the following calculation as if each column has different lengths or number of records (ignoring zero values). I have tried using NaN but for some reason operations on Nan values are not possible.



      Here is what I am trying to do with my df columns : 



      shape_list1=
      
location_list1=
      
scale_list1=
      

      
for column in df.columns:
      
    shape1, location1, scale1=stats.genpareto.fit(df[column])
      

      
    shape_list1.append(shape1)
      
    location_list1.append(location1)
      
    scale_list1.append(scale1)





      python pandas nan zero







      share|improve this question
















      I have a DataFrame (df) like shown below where each column is sorted from largest to smallest for frequency analysis. That leaves some values either zeros or NaN values as each column has a different length. 



         08FB006 08FC001 08FC003 08FC005 08GD004
      
----------------------------------------------
      
0   253      872    256      11.80    2660
      
1   250      850    255      10.60    2510
      
2   246      850    241      10.30    2130
      
3   241      827    235      9.32     1970
      
4   241      821    229      9.17     1900
      
5   232       0     228      8.93     1840
      
6   231       0     225      8.05     1710
      
7   0         0     225       0       1610
      
8   0         0     224       0       1590
      
9   0         0      0        0       1590
      
10  0         0      0        0       1550


      I need to perform the following calculation as if each column has different lengths or number of records (ignoring zero values). I have tried using NaN but for some reason operations on Nan values are not possible.



      Here is what I am trying to do with my df columns : 



      shape_list1=
      
location_list1=
      
scale_list1=
      

      
for column in df.columns:
      
    shape1, location1, scale1=stats.genpareto.fit(df[column])
      

      
    shape_list1.append(shape1)
      
    location_list1.append(location1)
      
    scale_list1.append(scale1)





      python pandas nan zero




      python pandas nan zero






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Nov 22 '18 at 22:00







      Sina Shabani

















      asked Nov 22 '18 at 21:44









      Sina ShabaniSina Shabani

      687




      687
























          2 Answers
          2






          active

          oldest

          votes


















          1














          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])





          share|improve this answer



















          • 2





            Thanks both of these answers worked :)

            – Sina Shabani
            Nov 22 '18 at 22:13













          • @andersource's syntax is a lot cleaner than mine!

            – Peter Leimbigler
            Nov 22 '18 at 22:41



















          0














          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].






          share|improve this answer
























          • Thanks for the explanation :)

            – Sina Shabani
            Nov 22 '18 at 22:13











          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53438296%2fhandling-zeros-or-nans-in-a-pandas-dataframe-operations%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          2 Answers
          2






          active

          oldest

          votes








          2 Answers
          2






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          1














          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])





          share|improve this answer



















          • 2





            Thanks both of these answers worked :)

            – Sina Shabani
            Nov 22 '18 at 22:13













          • @andersource's syntax is a lot cleaner than mine!

            – Peter Leimbigler
            Nov 22 '18 at 22:41
















          1














          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])





          share|improve this answer



















          • 2





            Thanks both of these answers worked :)

            – Sina Shabani
            Nov 22 '18 at 22:13













          • @andersource's syntax is a lot cleaner than mine!

            – Peter Leimbigler
            Nov 22 '18 at 22:41














          1












          1








          1







          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])





          share|improve this answer













          Assuming all values are positive (as seems from your example and description), try:



          stats.genpareto.fit(df[df[column] > 0][column])


          This filters every column to operate just on the positive values.
          Or, if negative values are allowed,



          stats.genpareto.fit(df[df[column] != 0][column])






          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Nov 22 '18 at 22:06









          andersourceandersource

          51418




          51418








          • 2





            Thanks both of these answers worked :)

            – Sina Shabani
            Nov 22 '18 at 22:13













          • @andersource's syntax is a lot cleaner than mine!

            – Peter Leimbigler
            Nov 22 '18 at 22:41














          • 2





            Thanks both of these answers worked :)

            – Sina Shabani
            Nov 22 '18 at 22:13













          • @andersource's syntax is a lot cleaner than mine!

            – Peter Leimbigler
            Nov 22 '18 at 22:41








          2




          2





          Thanks both of these answers worked :)

          – Sina Shabani
          Nov 22 '18 at 22:13







          Thanks both of these answers worked :)

          – Sina Shabani
          Nov 22 '18 at 22:13















          @andersource's syntax is a lot cleaner than mine!

          – Peter Leimbigler
          Nov 22 '18 at 22:41





          @andersource's syntax is a lot cleaner than mine!

          – Peter Leimbigler
          Nov 22 '18 at 22:41













          0














          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].






          share|improve this answer
























          • Thanks for the explanation :)

            – Sina Shabani
            Nov 22 '18 at 22:13
















          0














          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].






          share|improve this answer
























          • Thanks for the explanation :)

            – Sina Shabani
            Nov 22 '18 at 22:13














          0












          0








          0







          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].






          share|improve this answer













          The syntax is messy, but change



          shape1, location1, scale1=stats.genpareto.fit(df[column])


          to



          shape1, location1, scale1=stats.genpareto.fit(df[column][df[column].nonzero()[0]])


          Explanation: df[column].nonzero() returns a tuple of size (1,) whose only element, element [0], is a numpy array that holds the index labels where df is nonzero. To index df[column] by these nonzero labels, you can use df[column][df[column].nonzero()[0]].







          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Nov 22 '18 at 22:06









          Peter LeimbiglerPeter Leimbigler

          3,9081415




          3,9081415













          • Thanks for the explanation :)

            – Sina Shabani
            Nov 22 '18 at 22:13



















          • Thanks for the explanation :)

            – Sina Shabani
            Nov 22 '18 at 22:13

















          Thanks for the explanation :)

          – Sina Shabani
          Nov 22 '18 at 22:13





          Thanks for the explanation :)

          – Sina Shabani
          Nov 22 '18 at 22:13


















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53438296%2fhandling-zeros-or-nans-in-a-pandas-dataframe-operations%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          -

          Popular posts from this blog

          Costa Masnaga

          Image
          .mw-parser-output .avviso .mbox-text-div>div,.mw-parser-output .avviso .mbox-text-full-div>div{font-size:90%}.mw-parser-output .avviso .mbox-image div{width:52px}.mw-parser-output .avviso .mbox-text-full-div .hide-when-compact{display:block} Questa voce o sezione sull'argomento Lombardia non cita le fonti necessarie o quelle presenti sono insufficienti . Puoi migliorare questa voce aggiungendo citazioni da fonti attendibili secondo le linee guida sull'uso delle fonti. Segui i suggerimenti del progetto di riferimento. Costa Masnaga comune Localizzazione Stato   Italia Regione Lombardia Provincia Lecco Amministrazione Sindaco Sabina Panzeri (Al centro il cittadino) dall'08/06/2009 Territorio Coordinate 45°46′N 9°17′E  /  45.766667°N 9.283333°E 45.766667; 9.283333  ( Costa Masnaga ) Coordinate: 45°46′N 9°17′E  /  45.766667°N 9.283333°E 45.766667; 9.283333  ( Costa Masnaga ) Altitudine 318 m s.l.m...
          Read more

          Fotorealismo

          Image
          Guerrino Boatto, Venice's St. Barnaba Church and canal Il fotorealismo è un genere di pittura basato sull'uso di una o più fotografie dalle quali l'artista prende le informazioni necessarie da utilizzare poi nel processo della creazione dell'opera, in genere ad olio o ad acrilico, al fine di avere un aspetto finale il più simile possibile alla fotografia. Indice 1 Storia 2 Lista dei fotorealisti 3 Note 4 Voci correlate 5 Altri progetti Storia | Il fotorealismo è un movimento che nasce negli Stati Uniti d'America alla fine degli anni sessanta [1] sulla scia della Pop Art [2] [3] [4] , e insieme condividono il carattere reazionario verso la travolgente ascesa dei media, che alla metà del ventesimo secolo si trasforma in un fenomeno di massa talmente imponente da minacciare il valore dell'immagine nell'arte. [5] [6] Ne consegue così che, colto il gesto d'inizio che fu della Pop Art, i fotorealisti aprono ad un ...
          Read more

          Sidney Franklin

          Image
          Questa voce sull'argomento registi statunitensi è solo un abbozzo . Contribuisci a migliorarla secondo le convenzioni di Wikipedia. Sidney Franklin, foto di Carl Van Vechten Oscar alla memoria Irving G. Thalberg 1943 Sidney Franklin , all'anagrafe Sidney Arnold Franklin , (San Francisco, 21 marzo 1893 – Santa Monica, 18 maggio 1972), è stato un regista, produttore cinematografico e attore statunitense. Il 20 giugno 1963, si sposò con l'attrice australiana Enid Bennett. Il matrimonio durò fino alla morte dell'attrice, il 14 maggio 1969 [1] Indice 1 Filmografia 1.1 Regista 1.2 Aiuto regia (parziale) 1.3 Produttore 1.4 Attore (parziale) 2 Note 3 Altri progetti 4 Collegamenti esterni Filmografia | Regista | The Baby , co-regia Chester M. Franklin (1915) The Rivals , co-regia Chester M. Franklin (1915) Little Dick's First Case , co-regia Chester M. Franklin (1915) Her Filmland Hero , co-...
          Read more