Efficient way to Convert UTC time to Epoch in millisecond in Python in a Pandas DataFrame
0
How to Convert UTC time to epoch in a python DataFrame in an efficient manner ?
Here is what I have:
df = pd.DataFrame(['2017-12-12T00:59:54', '2017-12-12T01:09:52', '2017-12-12T01:19:51'], columns = ['time'])
Here is what I did:
import datetime as dt
epoch =
for x in df['time']:
dt = datetime.strptime(x, '%Y-%m-%dT%H:%M:%S')
epoch.append(dt.replace(tzinfo=timezone.utc).timestamp())
df['epoch'] = epoch
Here is my Issue:
This for loop seems to be very slow in a large pandas data frame, is there a more efficient way to do it?
Thank you! :)
python pandas datetime
asked Nov 21 '18 at 14:06
MatthewMatthew
79113
add a comment |
0
How to Convert UTC time to epoch in a python DataFrame in an efficient manner ?
Here is what I have:
df = pd.DataFrame(['2017-12-12T00:59:54', '2017-12-12T01:09:52', '2017-12-12T01:19:51'], columns = ['time'])
Here is what I did:
import datetime as dt
epoch =
for x in df['time']:
dt = datetime.strptime(x, '%Y-%m-%dT%H:%M:%S')
epoch.append(dt.replace(tzinfo=timezone.utc).timestamp())
df['epoch'] = epoch
Here is my Issue:
This for loop seems to be very slow in a large pandas data frame, is there a more efficient way to do it?
Thank you! :)
python pandas datetime
asked Nov 21 '18 at 14:06
MatthewMatthew
79113
add a comment |
0
0
0
How to Convert UTC time to epoch in a python DataFrame in an efficient manner ?
Here is what I have:
df = pd.DataFrame(['2017-12-12T00:59:54', '2017-12-12T01:09:52', '2017-12-12T01:19:51'], columns = ['time'])
Here is what I did:
import datetime as dt
epoch =
for x in df['time']:
dt = datetime.strptime(x, '%Y-%m-%dT%H:%M:%S')
epoch.append(dt.replace(tzinfo=timezone.utc).timestamp())
df['epoch'] = epoch
Here is my Issue:
This for loop seems to be very slow in a large pandas data frame, is there a more efficient way to do it?
Thank you! :)
python pandas datetime
asked Nov 21 '18 at 14:06
MatthewMatthew
79113
How to Convert UTC time to epoch in a python DataFrame in an efficient manner ?
Here is what I have:
df = pd.DataFrame(['2017-12-12T00:59:54', '2017-12-12T01:09:52', '2017-12-12T01:19:51'], columns = ['time'])
Here is what I did:
import datetime as dt
epoch =
for x in df['time']:
dt = datetime.strptime(x, '%Y-%m-%dT%H:%M:%S')
epoch.append(dt.replace(tzinfo=timezone.utc).timestamp())
df['epoch'] = epoch
Here is my Issue:
This for loop seems to be very slow in a large pandas data frame, is there a more efficient way to do it?
Thank you! :)
python pandas datetime
python pandas datetime
asked Nov 21 '18 at 14:06
MatthewMatthew
79113
asked Nov 21 '18 at 14:06
MatthewMatthew
79113
edited Nov 21 '18 at 14:17
Matthew
asked Nov 21 '18 at 14:06
MatthewMatthew
79113
asked Nov 21 '18 at 14:06
MatthewMatthew
79113
asked Nov 21 '18 at 14:06
MatthewMatthew
79113
79113
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
1
For me working:
df['epoch'] = pd.to_datetime(df['time']).astype(np.int64) // 10**9
print (df)
time epoch
0 2017-12-12T00:59:54 1513040394
1 2017-12-12T01:09:52 1513040992
2 2017-12-12T01:19:51 1513041591
answered Nov 21 '18 at 14:11
jezraeljezrael
325k23266342
What is the 10**9 part @jezrael ? :)
– Matthew
Nov 21 '18 at 14:17
@Matthew - data are in nanoseconds, for remove last0need divide by10**9- epoch in seconds
– jezrael
Nov 21 '18 at 14:18
so in milliseconds will be 10 ** 3 ?
– Matthew
Nov 21 '18 at 14:19
@Matthew - I think10**6
– jezrael
Nov 21 '18 at 14:23
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53413876%2fefficient-way-to-convert-utc-time-to-epoch-in-millisecond-in-python-in-a-pandas%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
1
For me working:
df['epoch'] = pd.to_datetime(df['time']).astype(np.int64) // 10**9
print (df)
time epoch
0 2017-12-12T00:59:54 1513040394
1 2017-12-12T01:09:52 1513040992
2 2017-12-12T01:19:51 1513041591
answered Nov 21 '18 at 14:11
jezraeljezrael
325k23266342
What is the 10**9 part @jezrael ? :)
– Matthew
Nov 21 '18 at 14:17
@Matthew - data are in nanoseconds, for remove last0need divide by10**9- epoch in seconds
– jezrael
Nov 21 '18 at 14:18
so in milliseconds will be 10 ** 3 ?
– Matthew
Nov 21 '18 at 14:19
@Matthew - I think10**6
– jezrael
Nov 21 '18 at 14:23
add a comment |
1
For me working:
df['epoch'] = pd.to_datetime(df['time']).astype(np.int64) // 10**9
print (df)
time epoch
0 2017-12-12T00:59:54 1513040394
1 2017-12-12T01:09:52 1513040992
2 2017-12-12T01:19:51 1513041591
answered Nov 21 '18 at 14:11
jezraeljezrael
325k23266342
What is the 10**9 part @jezrael ? :)
– Matthew
Nov 21 '18 at 14:17
@Matthew - data are in nanoseconds, for remove last0need divide by10**9- epoch in seconds
– jezrael
Nov 21 '18 at 14:18
so in milliseconds will be 10 ** 3 ?
– Matthew
Nov 21 '18 at 14:19
@Matthew - I think10**6
– jezrael
Nov 21 '18 at 14:23
add a comment |
1
1
1
For me working:
df['epoch'] = pd.to_datetime(df['time']).astype(np.int64) // 10**9
print (df)
time epoch
0 2017-12-12T00:59:54 1513040394
1 2017-12-12T01:09:52 1513040992
2 2017-12-12T01:19:51 1513041591
answered Nov 21 '18 at 14:11
jezraeljezrael
325k23266342
For me working:
df['epoch'] = pd.to_datetime(df['time']).astype(np.int64) // 10**9
print (df)
time epoch
0 2017-12-12T00:59:54 1513040394
1 2017-12-12T01:09:52 1513040992
2 2017-12-12T01:19:51 1513041591
answered Nov 21 '18 at 14:11
jezraeljezrael
325k23266342
answered Nov 21 '18 at 14:11
jezraeljezrael
325k23266342
answered Nov 21 '18 at 14:11
jezraeljezrael
325k23266342
answered Nov 21 '18 at 14:11
jezraeljezrael
325k23266342
325k23266342
What is the 10**9 part @jezrael ? :)
– Matthew
Nov 21 '18 at 14:17
@Matthew - data are in nanoseconds, for remove last0need divide by10**9- epoch in seconds
– jezrael
Nov 21 '18 at 14:18
so in milliseconds will be 10 ** 3 ?
– Matthew
Nov 21 '18 at 14:19
@Matthew - I think10**6
– jezrael
Nov 21 '18 at 14:23
add a comment |
What is the 10**9 part @jezrael ? :)
– Matthew
Nov 21 '18 at 14:17
@Matthew - data are in nanoseconds, for remove last0need divide by10**9- epoch in seconds
– jezrael
Nov 21 '18 at 14:18
so in milliseconds will be 10 ** 3 ?
– Matthew
Nov 21 '18 at 14:19
@Matthew - I think10**6
– jezrael
Nov 21 '18 at 14:23
What is the 10**9 part @jezrael ? :)
– Matthew
Nov 21 '18 at 14:17
What is the 10**9 part @jezrael ? :)
– Matthew
Nov 21 '18 at 14:17
@Matthew - data are in nanoseconds, for remove last
0 need divide by 10**9 - epoch in seconds– jezrael
Nov 21 '18 at 14:18
@Matthew - data are in nanoseconds, for remove last
0 need divide by 10**9 - epoch in seconds– jezrael
Nov 21 '18 at 14:18
so in milliseconds will be 10 ** 3 ?
– Matthew
Nov 21 '18 at 14:19
so in milliseconds will be 10 ** 3 ?
– Matthew
Nov 21 '18 at 14:19
@Matthew - I think
10**6– jezrael
Nov 21 '18 at 14:23
@Matthew - I think
10**6– jezrael
Nov 21 '18 at 14:23
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53413876%2fefficient-way-to-convert-utc-time-to-epoch-in-millisecond-in-python-in-a-pandas%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
