Compare an array with a file and form groups from elements of an array












0














I have a text file with letters (tab delimited), and a numpy array (obj) with a few letters (single row). The text file has rows with different numbers of columns. Some rows in the text file may have multiple copies of same letters (I will like to consider only a single copy of a letter in each row). Letters in the same row of the text file are assumed to be similar to each other. Also, each letter of the numpy array obj is present in one or more rows of the text file.



Here is an example of the text file (you can download the file from here):



b   q   a   i   m   l   r

j   n   o   r   o

e   i   k   u   i   s


In the example, the letter o is mentioned two times in the second row, and the letter i is denoted two times in the third row. I will like to consider single copies of letters rows of the text file.



This is an example of obj:



obj = np.asarray(['a', 'e', 'i', 'o', 'u'])


I want to compare obj with rows of the text file and form clusters from elements in obj.



This is how I want to do it. Corresponding to each row of the text file, I want to have a list which denotes a cluster (In the above example we will have three clusters since the text file has three rows). For every given element of obj, I want to find rows of the text file where the element is present. Then, I will like to assign index of that element of obj to the cluster which corresponds to the row with maximum length (the lengths of rows are decided with all rows having single copies of letters).



import pandas as pd

import numpy as np



data = pd.read_csv('file.txt', sep=r't+', header=None, engine='python').values[:,:].astype('<U1000')

obj = np.asarray(['a', 'e', 'i', 'o', 'u'])



for i in range(data.shape[0]):

    globals()['data_row' + str(i).zfill(3)] = 

    globals()['clust' + str(i).zfill(3)] = 

    for j in range(len(obj)):

        if obj[j] in set(data[i, :]): globals()['data_row' + str(i).zfill(3)] += [j]



for i in range(len(obj)):

    globals()['obj_lst' + str(i).zfill(3)] = [0]*data.shape[0]  



    for j in range(data.shape[0]):

        if i in globals()['data_row' + str(j).zfill(3)]:

            globals()['obj_lst' + str(i).zfill(3)][j] = len(globals()['data_row' + str(j).zfill(3)])



    indx_max = globals()['obj_lst' + str(i).zfill(3)].index( max(globals()['obj_lst' + str(i).zfill(3)]) )

    globals()['clust' + str(indx_max).zfill(3)] += [i]  



for i in range(data.shape[0]): print globals()['clust' + str(i).zfill(3)]



>> [0]

>> [3]

>> [1, 2, 4]


The code gives me the right answer. But, in my actual work, the text file has tens of thousands of rows, and the numpy array has hundreds of thousands of elements. And, the above given code is not very fast. So, I want to know if there is a better (faster) way to implement the above functionality and aim (using Python).






python array numpy pandas







share|improve this question









New contributor




Siddharth Satpathy is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.




















  • What do you mean by this statement: "Letters in the same row of the text file are assumed to be similar to each other."
    – l0b0
    16 mins ago
















0














I have a text file with letters (tab delimited), and a numpy array (obj) with a few letters (single row). The text file has rows with different numbers of columns. Some rows in the text file may have multiple copies of same letters (I will like to consider only a single copy of a letter in each row). Letters in the same row of the text file are assumed to be similar to each other. Also, each letter of the numpy array obj is present in one or more rows of the text file.



Here is an example of the text file (you can download the file from here):



b   q   a   i   m   l   r

j   n   o   r   o

e   i   k   u   i   s


In the example, the letter o is mentioned two times in the second row, and the letter i is denoted two times in the third row. I will like to consider single copies of letters rows of the text file.



This is an example of obj:



obj = np.asarray(['a', 'e', 'i', 'o', 'u'])


I want to compare obj with rows of the text file and form clusters from elements in obj.



This is how I want to do it. Corresponding to each row of the text file, I want to have a list which denotes a cluster (In the above example we will have three clusters since the text file has three rows). For every given element of obj, I want to find rows of the text file where the element is present. Then, I will like to assign index of that element of obj to the cluster which corresponds to the row with maximum length (the lengths of rows are decided with all rows having single copies of letters).



import pandas as pd

import numpy as np



data = pd.read_csv('file.txt', sep=r't+', header=None, engine='python').values[:,:].astype('<U1000')

obj = np.asarray(['a', 'e', 'i', 'o', 'u'])



for i in range(data.shape[0]):

    globals()['data_row' + str(i).zfill(3)] = 

    globals()['clust' + str(i).zfill(3)] = 

    for j in range(len(obj)):

        if obj[j] in set(data[i, :]): globals()['data_row' + str(i).zfill(3)] += [j]



for i in range(len(obj)):

    globals()['obj_lst' + str(i).zfill(3)] = [0]*data.shape[0]  



    for j in range(data.shape[0]):

        if i in globals()['data_row' + str(j).zfill(3)]:

            globals()['obj_lst' + str(i).zfill(3)][j] = len(globals()['data_row' + str(j).zfill(3)])



    indx_max = globals()['obj_lst' + str(i).zfill(3)].index( max(globals()['obj_lst' + str(i).zfill(3)]) )

    globals()['clust' + str(indx_max).zfill(3)] += [i]  



for i in range(data.shape[0]): print globals()['clust' + str(i).zfill(3)]



>> [0]

>> [3]

>> [1, 2, 4]


The code gives me the right answer. But, in my actual work, the text file has tens of thousands of rows, and the numpy array has hundreds of thousands of elements. And, the above given code is not very fast. So, I want to know if there is a better (faster) way to implement the above functionality and aim (using Python).






python array numpy pandas







share|improve this question









New contributor




Siddharth Satpathy is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.




















  • What do you mean by this statement: "Letters in the same row of the text file are assumed to be similar to each other."
    – l0b0
    16 mins ago














0












0








0







I have a text file with letters (tab delimited), and a numpy array (obj) with a few letters (single row). The text file has rows with different numbers of columns. Some rows in the text file may have multiple copies of same letters (I will like to consider only a single copy of a letter in each row). Letters in the same row of the text file are assumed to be similar to each other. Also, each letter of the numpy array obj is present in one or more rows of the text file.



Here is an example of the text file (you can download the file from here):



b   q   a   i   m   l   r

j   n   o   r   o

e   i   k   u   i   s


In the example, the letter o is mentioned two times in the second row, and the letter i is denoted two times in the third row. I will like to consider single copies of letters rows of the text file.



This is an example of obj:



obj = np.asarray(['a', 'e', 'i', 'o', 'u'])


I want to compare obj with rows of the text file and form clusters from elements in obj.



This is how I want to do it. Corresponding to each row of the text file, I want to have a list which denotes a cluster (In the above example we will have three clusters since the text file has three rows). For every given element of obj, I want to find rows of the text file where the element is present. Then, I will like to assign index of that element of obj to the cluster which corresponds to the row with maximum length (the lengths of rows are decided with all rows having single copies of letters).



import pandas as pd

import numpy as np



data = pd.read_csv('file.txt', sep=r't+', header=None, engine='python').values[:,:].astype('<U1000')

obj = np.asarray(['a', 'e', 'i', 'o', 'u'])



for i in range(data.shape[0]):

    globals()['data_row' + str(i).zfill(3)] = 

    globals()['clust' + str(i).zfill(3)] = 

    for j in range(len(obj)):

        if obj[j] in set(data[i, :]): globals()['data_row' + str(i).zfill(3)] += [j]



for i in range(len(obj)):

    globals()['obj_lst' + str(i).zfill(3)] = [0]*data.shape[0]  



    for j in range(data.shape[0]):

        if i in globals()['data_row' + str(j).zfill(3)]:

            globals()['obj_lst' + str(i).zfill(3)][j] = len(globals()['data_row' + str(j).zfill(3)])



    indx_max = globals()['obj_lst' + str(i).zfill(3)].index( max(globals()['obj_lst' + str(i).zfill(3)]) )

    globals()['clust' + str(indx_max).zfill(3)] += [i]  



for i in range(data.shape[0]): print globals()['clust' + str(i).zfill(3)]



>> [0]

>> [3]

>> [1, 2, 4]


The code gives me the right answer. But, in my actual work, the text file has tens of thousands of rows, and the numpy array has hundreds of thousands of elements. And, the above given code is not very fast. So, I want to know if there is a better (faster) way to implement the above functionality and aim (using Python).






python array numpy pandas







share|improve this question









New contributor




Siddharth Satpathy is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











I have a text file with letters (tab delimited), and a numpy array (obj) with a few letters (single row). The text file has rows with different numbers of columns. Some rows in the text file may have multiple copies of same letters (I will like to consider only a single copy of a letter in each row). Letters in the same row of the text file are assumed to be similar to each other. Also, each letter of the numpy array obj is present in one or more rows of the text file.



Here is an example of the text file (you can download the file from here):



b   q   a   i   m   l   r

j   n   o   r   o

e   i   k   u   i   s


In the example, the letter o is mentioned two times in the second row, and the letter i is denoted two times in the third row. I will like to consider single copies of letters rows of the text file.



This is an example of obj:



obj = np.asarray(['a', 'e', 'i', 'o', 'u'])


I want to compare obj with rows of the text file and form clusters from elements in obj.



This is how I want to do it. Corresponding to each row of the text file, I want to have a list which denotes a cluster (In the above example we will have three clusters since the text file has three rows). For every given element of obj, I want to find rows of the text file where the element is present. Then, I will like to assign index of that element of obj to the cluster which corresponds to the row with maximum length (the lengths of rows are decided with all rows having single copies of letters).



import pandas as pd

import numpy as np



data = pd.read_csv('file.txt', sep=r't+', header=None, engine='python').values[:,:].astype('<U1000')

obj = np.asarray(['a', 'e', 'i', 'o', 'u'])



for i in range(data.shape[0]):

    globals()['data_row' + str(i).zfill(3)] = 

    globals()['clust' + str(i).zfill(3)] = 

    for j in range(len(obj)):

        if obj[j] in set(data[i, :]): globals()['data_row' + str(i).zfill(3)] += [j]



for i in range(len(obj)):

    globals()['obj_lst' + str(i).zfill(3)] = [0]*data.shape[0]  



    for j in range(data.shape[0]):

        if i in globals()['data_row' + str(j).zfill(3)]:

            globals()['obj_lst' + str(i).zfill(3)][j] = len(globals()['data_row' + str(j).zfill(3)])



    indx_max = globals()['obj_lst' + str(i).zfill(3)].index( max(globals()['obj_lst' + str(i).zfill(3)]) )

    globals()['clust' + str(indx_max).zfill(3)] += [i]  



for i in range(data.shape[0]): print globals()['clust' + str(i).zfill(3)]



>> [0]

>> [3]

>> [1, 2, 4]


The code gives me the right answer. But, in my actual work, the text file has tens of thousands of rows, and the numpy array has hundreds of thousands of elements. And, the above given code is not very fast. So, I want to know if there is a better (faster) way to implement the above functionality and aim (using Python).






python array numpy pandas




python array numpy pandas






share|improve this question









New contributor




Siddharth Satpathy is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











share|improve this question









New contributor




Siddharth Satpathy is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









share|improve this question




share|improve this question








edited 9 mins ago









Jamal♦

30.3k11116226




30.3k11116226






New contributor




Siddharth Satpathy is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









asked 4 hours ago









Siddharth Satpathy

1012




1012




New contributor




Siddharth Satpathy is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.





New contributor





Siddharth Satpathy is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.






Siddharth Satpathy is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.












  • What do you mean by this statement: "Letters in the same row of the text file are assumed to be similar to each other."
    – l0b0
    16 mins ago


















  • What do you mean by this statement: "Letters in the same row of the text file are assumed to be similar to each other."
    – l0b0
    16 mins ago
















What do you mean by this statement: "Letters in the same row of the text file are assumed to be similar to each other."
– l0b0
16 mins ago




What do you mean by this statement: "Letters in the same row of the text file are assumed to be similar to each other."
– l0b0
16 mins ago










0






active

oldest

votes











Your Answer





StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["\$", "\$"]]);
});
});
}, "mathjax-editing");

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "196"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});






Siddharth Satpathy is a new contributor. Be nice, and check out our Code of Conduct.










draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f210784%2fcompare-an-array-with-a-file-and-form-groups-from-elements-of-an-array%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























0






active

oldest

votes








0






active

oldest

votes









active

oldest

votes






active

oldest

votes








Siddharth Satpathy is a new contributor. Be nice, and check out our Code of Conduct.










draft saved

draft discarded


















Siddharth Satpathy is a new contributor. Be nice, and check out our Code of Conduct.













Siddharth Satpathy is a new contributor. Be nice, and check out our Code of Conduct.












Siddharth Satpathy is a new contributor. Be nice, and check out our Code of Conduct.
















Thanks for contributing an answer to Code Review Stack Exchange!


  • Please be sure to answer the question. Provide details and share your research!

But avoid …



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


Use MathJax to format equations. MathJax reference.


To learn more, see our tips on writing great answers.





Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


Please pay close attention to the following guidance:


  • Please be sure to answer the question. Provide details and share your research!

But avoid …



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f210784%2fcompare-an-array-with-a-file-and-form-groups-from-elements-of-an-array%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







-

Popular posts from this blog

Create new schema in PostgreSQL using DBeaver

Image
0 At the beginning of my current job I had limited access to DB but recently my role is changed to superuser, admin, and owner. My colleague who was the previous DB admin is able to create a schema or change the permissions as there is a permission tab in the properties and he can change permissions by clicking easily. Would you please let me know how I can create schema and change permissions as well? Why his dbeaver looks different from me? He can easily right click on the schema folder on top of all schemas on the left side column and select create new schema while I do not have that, like the following picture I found on the internet. postgresql database-schema dbeaver share | improve this question
Read more

Deepest pit of an array with Javascript: test on Codility

Image
0 $begingroup$ This is the question from a Codility test, as a task in an interview: DeepestPit - problem description A non-empty zero-indexed array A consisting of N integers is given. A pit in this array is any triplet of integers (P, Q, R) such that: ·         0 ≤ P < Q < R < N; ·         sequence [A[P], A[P+1], ..., A[Q]] is strictly decreasing, i.e. A[P] > A[P+1] > ... > A[Q]; ·         sequence A[Q], A[Q+1], ..., A[R] is strictly increasing, i.e. A[Q] < A[Q+1] < ... < A[R]. The depth of a pit (P, Q, R) is the number min {A[P] − A[Q], A[R] − A[Q]}. For example, consider array A consisting of 10 elements such that: A[0] = 0 A[1] = 1 A[2] = 3 A[3] = -2 A[4] = 0 A[5] = 1 A[6] = 0 A[7] = -3 A[8] = 2 A[9] = 3 Triplet (2, 3, 4) is
Read more

Costa Masnaga

Image
.mw-parser-output .avviso .mbox-text-div>div,.mw-parser-output .avviso .mbox-text-full-div>div{font-size:90%}.mw-parser-output .avviso .mbox-image div{width:52px}.mw-parser-output .avviso .mbox-text-full-div .hide-when-compact{display:block} Questa voce o sezione sull'argomento Lombardia non cita le fonti necessarie o quelle presenti sono insufficienti . Puoi migliorare questa voce aggiungendo citazioni da fonti attendibili secondo le linee guida sull'uso delle fonti. Segui i suggerimenti del progetto di riferimento. Costa Masnaga comune Localizzazione Stato   Italia Regione Lombardia Provincia Lecco Amministrazione Sindaco Sabina Panzeri (Al centro il cittadino) dall'08/06/2009 Territorio Coordinate 45°46′N 9°17′E  /  45.766667°N 9.283333°E 45.766667; 9.283333  ( Costa Masnaga ) Coordinate: 45°46′N 9°17′E  /  45.766667°N 9.283333°E 45.766667; 9.283333  ( Costa Masnaga ) Altitudine 318 m s.l.m
Read more