[b-hebrew] Hebrew in messages

Karl Randolph kwrandolph at email.com
Wed Jul 13 11:17:02 EDT 2005


Content is exactly what is being lost when character entities are changed to html text strings.

That non-ASCII characters are stored in the archives as character entities means that they can not be quoted without losing content.

The encoding is defined by <META http-equiv="Content-Type" content="text/html; charset=us-ascii"> means that the only way to show non-ASCII characters is to have them in ASCII coding, such as character entities. There is no Windows-1255 or UTF-8 encoding. Therefore, in order not to lose content, such coding must be stored unchanged. It changes them.

At first I was ready to blame your email program. Now I realize it is deeper than that. We are dealing with incompatible systems, some of which have been around for some time. The earliest non-MIME complient standard was developed to show Asian characters, and one of the earliest MIME challenges was how to transmit those messages unchanged. (I’d forgotten about uuencoding, a way to transmit non-ASCII data as ASCII strings.)

You blame the mail program I am using. How it sends messages, i.e. the encoding it uses, I am not sure. However it appears that it sends ASCII strings. For non-ASCII characters, that would mean character entities. For receiving messages, it is designed to be read through a browser such as Firefox, Camino and Mozilla, so it passes on whatever message it receives unchanged: if Windows-1255, I can tell the browser to read it; if UTF-8, a different setting; if character entities, sent to the browser unchanged so that the content could be displayed unlost. Of course, I speculate because I don’t have enough data.

Then we have the ibiblio archiving program that takes only us-ascii. When it receives data, it loses content if that data is already encoded in character entities to be compatible with ASCII, because it changes it so that the data is no longer character entities. That losing content is certainly a bug.

The question is, what are we as a forum going to do about these incompatible standards? Shall we make it a rule that in order to participate, we need to use a certain standard? If so, which? Should the technically savvy among us draw up a list of approved email clients, recommending that only those be used by list members?    ??

Karl W. Randolph.

----- Original Message -----
From: "Peter Kirk" <peterkirk at qaya.org>

> On 12/07/2005 20:05, Karl Randolph wrote:
> As a message passes through a series of computer systems, typically 
> from a human writer to a human reader, what needs to be preserved 
> is the CONTENT of the message, which consists (for a text message) 
> of a series of characters (sometimes known as abstract characters). 
> These characters may be encoded in several different ways, and the 
> encodings used may change several times on the way. But this does 
> not matter, because the content is unchanged. The underlying 
> encoding is generally hidden from the actual human user.
> But programs do often provide ways for expert users to peek behind 
> the encoding and manipulate the rules. This is done so that people 
> like you and me can find out what is happening when something goes 
> wrong, as has happened in this case. But we should really look at 
> what happens when we are not peeking behind the encoding. When you 
> look at the b-hebrew archives in the normal way, do you see 
> anything wrong? I think you will see nothing wrong with messages 
> sent by me and most other list members. You may see things wrong 
> with messages you have sent, but the bug which causes this is in 
> your mail program, not in the archiving software.
> The software is changing the encoding of the messages, which is 
> permissible, but it is not changing the content.
> > What I expect now is that if I go back in the archives to one of 
> > your messages with some Hebrew in it, and quote it, that the 
> > Hebrew will come out as unintelligeable html text strings. Is 
> > that not a bug?
> >
> >
> >
> Can you show me an example of this, with one of my messages? Try 
> with this one, into which I will now insert some Hebrew in UTF-8 
> encoding:
> אבגד
> And you can look back to 
> http://lists.ibiblio.org/pipermail/b-hebrew/2005-July/023703.html, 
> my posting on 5th July, for Hebrew in Windows-1255 encoding - look 
> only at the "properly encoded Hebrew characters", the next lines 
> are deliberately not properly encoded.
> By the way, there is another clear bug in your mail software that 
> it does not send information about which message is being replied 
> to, which is why the archiving software takes each of your messages 
> as the start of a new thread.
> -- Peter Kirk
> peter at qaya.org (personal)
> peterkirk at qaya.org (work)
> http://www.qaya.org/

More information about the b-hebrew mailing list