Python cannot decode XORed string with a key larger than 127
I have a problem with decoding bytes to string in python. I wanted to crack the single-byte xor cipher by brute-forcing it. And I did. The key was 90 (dec). The problem here is, that I actually could XOR every byte with a number > 127 but then I always got UnicodeError, when I wanted to decode it. They all looked the same, just different bytes:
128. 'utf-8' codec can't decode byte 0x98 in position 0: invalid start byte
def main():
d = base64.b64decode(base64string)
for i in range(1, 255):
try:
output_bytes = b''.join([bytes([b ^ i]) for b in d])
decoded = output_bytes.decode('utf-8')
except UnicodeDecodeError as e:
print(f'{i}. {e}')
pass
if __name__ == '__main__':
main()
I either had to write a try/except block to prevent my program from failing, or I had to change range(1, 255) to range(1, 127).
Is it an expected behavior? If no, then what am I doing wrong, and if yes, then why? Shouldn't I be able to do xor and then decode() no matter which byte I use?
python unicode xor
asked Nov 25 '18 at 1:29
dabljuesdabljues
1777
add a comment |
I have a problem with decoding bytes to string in python. I wanted to crack the single-byte xor cipher by brute-forcing it. And I did. The key was 90 (dec). The problem here is, that I actually could XOR every byte with a number > 127 but then I always got UnicodeError, when I wanted to decode it. They all looked the same, just different bytes:
128. 'utf-8' codec can't decode byte 0x98 in position 0: invalid start byte
def main():
d = base64.b64decode(base64string)
for i in range(1, 255):
try:
output_bytes = b''.join([bytes([b ^ i]) for b in d])
decoded = output_bytes.decode('utf-8')
except UnicodeDecodeError as e:
print(f'{i}. {e}')
pass
if __name__ == '__main__':
main()
I either had to write a try/except block to prevent my program from failing, or I had to change range(1, 255) to range(1, 127).
Is it an expected behavior? If no, then what am I doing wrong, and if yes, then why? Shouldn't I be able to do xor and then decode() no matter which byte I use?
python unicode xor
asked Nov 25 '18 at 1:29
dabljuesdabljues
1777
add a comment |
I have a problem with decoding bytes to string in python. I wanted to crack the single-byte xor cipher by brute-forcing it. And I did. The key was 90 (dec). The problem here is, that I actually could XOR every byte with a number > 127 but then I always got UnicodeError, when I wanted to decode it. They all looked the same, just different bytes:
128. 'utf-8' codec can't decode byte 0x98 in position 0: invalid start byte
def main():
d = base64.b64decode(base64string)
for i in range(1, 255):
try:
output_bytes = b''.join([bytes([b ^ i]) for b in d])
decoded = output_bytes.decode('utf-8')
except UnicodeDecodeError as e:
print(f'{i}. {e}')
pass
if __name__ == '__main__':
main()
I either had to write a try/except block to prevent my program from failing, or I had to change range(1, 255) to range(1, 127).
Is it an expected behavior? If no, then what am I doing wrong, and if yes, then why? Shouldn't I be able to do xor and then decode() no matter which byte I use?
python unicode xor
asked Nov 25 '18 at 1:29
dabljuesdabljues
1777
I have a problem with decoding bytes to string in python. I wanted to crack the single-byte xor cipher by brute-forcing it. And I did. The key was 90 (dec). The problem here is, that I actually could XOR every byte with a number > 127 but then I always got UnicodeError, when I wanted to decode it. They all looked the same, just different bytes:
128. 'utf-8' codec can't decode byte 0x98 in position 0: invalid start byte
def main():
d = base64.b64decode(base64string)
for i in range(1, 255):
try:
output_bytes = b''.join([bytes([b ^ i]) for b in d])
decoded = output_bytes.decode('utf-8')
except UnicodeDecodeError as e:
print(f'{i}. {e}')
pass
if __name__ == '__main__':
main()
I either had to write a try/except block to prevent my program from failing, or I had to change range(1, 255) to range(1, 127).
Is it an expected behavior? If no, then what am I doing wrong, and if yes, then why? Shouldn't I be able to do xor and then decode() no matter which byte I use?
python unicode xor
python unicode xor
asked Nov 25 '18 at 1:29
dabljuesdabljues
1777
asked Nov 25 '18 at 1:29
dabljuesdabljues
1777
asked Nov 25 '18 at 1:29
dabljuesdabljues
1777
asked Nov 25 '18 at 1:29
dabljuesdabljues
1777
asked Nov 25 '18 at 1:29
dabljuesdabljues
1777
1777
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
UTF-8 expects bytes to have a specific format. If you don't have the right sequence of bytes, it can generate a UnicodeDecodeError. Unless you have the correct key, it would be expected to fail. This also assumes the original message was UTF-8-encoded to begin with. Using a try/except is the right thing to do if the message doesn't have the correct key.
For example, here's a Unicode string encoded in UTF-8 and encrypted, then decoded with all possible byte XOR keys. Only a few keys are successful decoding UTF-8:
#coding:utf8
def encode(msg,key):
data = msg.encode('utf8')
return bytes(b ^ key for b in data)
def decode(msg,key):
data = bytes(b ^ key for b in msg)
try:
return data.decode('utf8')
except UnicodeDecodeError:
return None
msg = "Hello, 马克!"
enc = encode(msg,0x90)
print(enc)
for i in range(256):
dec = decode(enc,i)
if dec is not None:
print(f'{i:02x}: {dec}')
Output:
b'xd8xf5xfcxfcxffxbcxb0y9<ux15x1bxb1'
90: Hello, 马克!
91: Idmmn-!設䄊
92: Jgnnm."뫮燉#
93: Kfool/#ꪯ憈"
96: Ncjji*&ﯪネ'
97: Obkkh+'₌&
98: @mddg$(ᡤ퍃)
99: Aleef%)ࠥ쌂(
9a: Boffe&*㣦+
9b: Cnggd'+⢧*
9c: Di``c ,奠鉇-
9d: Ehaab!-䤡舆,
9e: Fkbba".秢닅/
9f: Gjcc`#/榣ꊄ.
answered Nov 25 '18 at 7:52
Mark TolonenMark Tolonen
94.9k12114177
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53463926%2fpython-cannot-decode-xored-string-with-a-key-larger-than-127%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
UTF-8 expects bytes to have a specific format. If you don't have the right sequence of bytes, it can generate a UnicodeDecodeError. Unless you have the correct key, it would be expected to fail. This also assumes the original message was UTF-8-encoded to begin with. Using a try/except is the right thing to do if the message doesn't have the correct key.
For example, here's a Unicode string encoded in UTF-8 and encrypted, then decoded with all possible byte XOR keys. Only a few keys are successful decoding UTF-8:
#coding:utf8
def encode(msg,key):
data = msg.encode('utf8')
return bytes(b ^ key for b in data)
def decode(msg,key):
data = bytes(b ^ key for b in msg)
try:
return data.decode('utf8')
except UnicodeDecodeError:
return None
msg = "Hello, 马克!"
enc = encode(msg,0x90)
print(enc)
for i in range(256):
dec = decode(enc,i)
if dec is not None:
print(f'{i:02x}: {dec}')
Output:
b'xd8xf5xfcxfcxffxbcxb0y9<ux15x1bxb1'
90: Hello, 马克!
91: Idmmn-!設䄊
92: Jgnnm."뫮燉#
93: Kfool/#ꪯ憈"
96: Ncjji*&ﯪネ'
97: Obkkh+'₌&
98: @mddg$(ᡤ퍃)
99: Aleef%)ࠥ쌂(
9a: Boffe&*㣦+
9b: Cnggd'+⢧*
9c: Di``c ,奠鉇-
9d: Ehaab!-䤡舆,
9e: Fkbba".秢닅/
9f: Gjcc`#/榣ꊄ.
answered Nov 25 '18 at 7:52
Mark TolonenMark Tolonen
94.9k12114177
add a comment |
UTF-8 expects bytes to have a specific format. If you don't have the right sequence of bytes, it can generate a UnicodeDecodeError. Unless you have the correct key, it would be expected to fail. This also assumes the original message was UTF-8-encoded to begin with. Using a try/except is the right thing to do if the message doesn't have the correct key.
For example, here's a Unicode string encoded in UTF-8 and encrypted, then decoded with all possible byte XOR keys. Only a few keys are successful decoding UTF-8:
#coding:utf8
def encode(msg,key):
data = msg.encode('utf8')
return bytes(b ^ key for b in data)
def decode(msg,key):
data = bytes(b ^ key for b in msg)
try:
return data.decode('utf8')
except UnicodeDecodeError:
return None
msg = "Hello, 马克!"
enc = encode(msg,0x90)
print(enc)
for i in range(256):
dec = decode(enc,i)
if dec is not None:
print(f'{i:02x}: {dec}')
Output:
b'xd8xf5xfcxfcxffxbcxb0y9<ux15x1bxb1'
90: Hello, 马克!
91: Idmmn-!設䄊
92: Jgnnm."뫮燉#
93: Kfool/#ꪯ憈"
96: Ncjji*&ﯪネ'
97: Obkkh+'₌&
98: @mddg$(ᡤ퍃)
99: Aleef%)ࠥ쌂(
9a: Boffe&*㣦+
9b: Cnggd'+⢧*
9c: Di``c ,奠鉇-
9d: Ehaab!-䤡舆,
9e: Fkbba".秢닅/
9f: Gjcc`#/榣ꊄ.
answered Nov 25 '18 at 7:52
Mark TolonenMark Tolonen
94.9k12114177
add a comment |
UTF-8 expects bytes to have a specific format. If you don't have the right sequence of bytes, it can generate a UnicodeDecodeError. Unless you have the correct key, it would be expected to fail. This also assumes the original message was UTF-8-encoded to begin with. Using a try/except is the right thing to do if the message doesn't have the correct key.
For example, here's a Unicode string encoded in UTF-8 and encrypted, then decoded with all possible byte XOR keys. Only a few keys are successful decoding UTF-8:
#coding:utf8
def encode(msg,key):
data = msg.encode('utf8')
return bytes(b ^ key for b in data)
def decode(msg,key):
data = bytes(b ^ key for b in msg)
try:
return data.decode('utf8')
except UnicodeDecodeError:
return None
msg = "Hello, 马克!"
enc = encode(msg,0x90)
print(enc)
for i in range(256):
dec = decode(enc,i)
if dec is not None:
print(f'{i:02x}: {dec}')
Output:
b'xd8xf5xfcxfcxffxbcxb0y9<ux15x1bxb1'
90: Hello, 马克!
91: Idmmn-!設䄊
92: Jgnnm."뫮燉#
93: Kfool/#ꪯ憈"
96: Ncjji*&ﯪネ'
97: Obkkh+'₌&
98: @mddg$(ᡤ퍃)
99: Aleef%)ࠥ쌂(
9a: Boffe&*㣦+
9b: Cnggd'+⢧*
9c: Di``c ,奠鉇-
9d: Ehaab!-䤡舆,
9e: Fkbba".秢닅/
9f: Gjcc`#/榣ꊄ.
answered Nov 25 '18 at 7:52
Mark TolonenMark Tolonen
94.9k12114177
UTF-8 expects bytes to have a specific format. If you don't have the right sequence of bytes, it can generate a UnicodeDecodeError. Unless you have the correct key, it would be expected to fail. This also assumes the original message was UTF-8-encoded to begin with. Using a try/except is the right thing to do if the message doesn't have the correct key.
For example, here's a Unicode string encoded in UTF-8 and encrypted, then decoded with all possible byte XOR keys. Only a few keys are successful decoding UTF-8:
#coding:utf8
def encode(msg,key):
data = msg.encode('utf8')
return bytes(b ^ key for b in data)
def decode(msg,key):
data = bytes(b ^ key for b in msg)
try:
return data.decode('utf8')
except UnicodeDecodeError:
return None
msg = "Hello, 马克!"
enc = encode(msg,0x90)
print(enc)
for i in range(256):
dec = decode(enc,i)
if dec is not None:
print(f'{i:02x}: {dec}')
Output:
b'xd8xf5xfcxfcxffxbcxb0y9<ux15x1bxb1'
90: Hello, 马克!
91: Idmmn-!設䄊
92: Jgnnm."뫮燉#
93: Kfool/#ꪯ憈"
96: Ncjji*&ﯪネ'
97: Obkkh+'₌&
98: @mddg$(ᡤ퍃)
99: Aleef%)ࠥ쌂(
9a: Boffe&*㣦+
9b: Cnggd'+⢧*
9c: Di``c ,奠鉇-
9d: Ehaab!-䤡舆,
9e: Fkbba".秢닅/
9f: Gjcc`#/榣ꊄ.
answered Nov 25 '18 at 7:52
Mark TolonenMark Tolonen
94.9k12114177
edited Nov 25 '18 at 8:12
answered Nov 25 '18 at 7:52
Mark TolonenMark Tolonen
94.9k12114177
answered Nov 25 '18 at 7:52
Mark TolonenMark Tolonen
94.9k12114177
answered Nov 25 '18 at 7:52
Mark TolonenMark Tolonen
94.9k12114177
94.9k12114177
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53463926%2fpython-cannot-decode-xored-string-with-a-key-larger-than-127%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
