|
Open letter to MIDP standard mantainers and hardware manufacturers from Anfy Team Wireless Java™ development section.
MIDP 1.1+ SUGGESTED IMPROVEMENTS by Anfy Team
document version 1.3, August 6, 2001
Introduction:
Anfy Team produce multimedia and graphically intensite Java applets
since early 1996, and have previous experience on old 8bit computers
with constrained resources (64kb ram, 2Mhz processors).
We found the MIDP 1.0 specifications is missing important features
for videogaming and multimedia: in hour hopinion, with different gfx
API interface we may 2x up to 10x speed-up a wide range of multimedia
applications and videogames, using same hardware (and same costs).
Here's Anfy Team tech suggestion, for future midp enhancements,
and/or for com.* extra packages from manufacturers.
********************************************************
Image Transparency support
********************************************************
1) Image transparency must be supported, following the PNG
transparency, or ALPHA informations in 0xAARRGGBB int.
Alpha byte AA, where at least 2 states are supported:
FF for solid and 00 for transparent, or use bit 31
(1 for an opaque pixel, 0 for a transparent pixel),
with bits 24-30 ignored (00 - 7F is transparent,
80 - FF is opaque.)
In japanese NTT-docomo i-appli's the drawImage could
paint with transparency and this permits "sprites"
with circular contours rather than the overlapping
rectangles of MIDP 1.0 non-transparency.
Results similiar to transparent image drawing can be
obtained with multiple clipping but this is quite
a complex to program and not optimized operation.
This could be implemented adding transparency supp.
in standard drawImage(Image). Or to mantain full
compatibility with MIDP 1.0 (standard Image object
does not consider transparency even if it's present),
adding a special com.manufacturer.Image object able
to retain transparency mask information, and then add
a new draw method variant such as:
drawImage(com.manufacturer.Image)
which accepts such transparent-enabled image and
plots it eventually using the transparency(alpha).
Separated Image/drawImage() implementations for
transparent-non transparent plotting may let
optimize memory/KVM usage (alpha mask stored and
parsed only if special Image/drawImage() is used).
It's important to implement such method in native
code, and to store image internally in native
format, in order to plot it with transparency
(or even without it) as fast as possible, to
permit fluid animation of many moving images.
********************************************************
RGB Image content access in both Read and Write
********************************************************
2) A method to create images from non gif/png encoded
int[] arrays with raw 0x00RRGGBB data is missing.
This is foundamental for paint programs, some
video games effects and optimizations, and other.
We suggest adding such method:
public static Image createImage(int[] rgbData, int offset,
int width, int height) throws ArrayIndexOutOfBoundsException
Explanation of parameters:
int[] rgbData: an array with 0x00RRGGBB pixel informations.
If image transparency is supported, 0xAARRGGBB.
int offset: offset to start reading data from within rgbData[]
int width, int height: width and height of returned Image.
If (width*height)>(rgbData[]lenght-offset): ArrayIndexOutOfBoundsException
Better would be to have a variant (or just this):
public static Image createImage(int[] rgbData, int offset,
int width, int height, int scanline)
throws ArrayIndexOutOfBoundsException
Why add the "scanline" parameter too?
The method will behave that way: will read "width" pixels
starting from "offset", then rather than increment by "width"
again, it will increment by "scanline".
Function will have same behaviour if width=scanline, but
may help in tricks like scrolling or copying part of images.
This, because we can have in the array (memory) a screen
bigger than the real one, but create images from
a (variable/scrollable) horizontal area, leaving
the extra horiz part (width-scanline) hidden.
Another method which may help while dealing
with generation of (multiple) images starting from
a single one with only parts changed, could be:
public static void editImage(Image img, int[] rgbData, int imgOffset,
int dataOffset) throws ArrayIndexOutOfBoundsException
This will edit an already instantiated Image, overwriting
parts (or whole) of it's contents with RGB pixels described
in rgbData[] array. If a version with w,h,scanline is
also provided, this may permit easier update of rectangles
which are not 100% image wide.
Also, it's possible to only make available an editImage()
method passing a previously created Image, rather than
having both createImage() and editImage() variants.
Having only a createImage() which return a new Image
is not suitable instead to make multiple editing of an
image, because we would create a new Image object all
the times and this consume memory.
- Note on animation:
On Java2 the methods are run in multithread and
the end of method execution should be tracked
with mediaTracker or imageProducers; that's a
too much complex system; however, it should be
possible to know if Image.setArray() "completed",
before use the image again, or we can see on
video the image half completed.
Especially, in animation, the dynamic update
of the Image contents should be "synchronized" so,
we see only completed frames.
On Java, there's the setAnimated(true) method,
to set on an Image, then a newPixels() is called
each time.
Double buffer is also useful but it's another matter.
We should make sure createImage() or editImage()
does not return until the operation is done, if
no other mediatracker-like methods are available.
So, if the array[] with image contents may be changed
(updated for animation), we don't have to create a
new image or array, garbage collect the old ones
and such things: we simply overwrite the image
array, and call the Image.editImage() specifying
the same array all the times, with an implicit
"newPixels()" which make sure all the times we
plot the image, latest content is 100% done and
displayable, still using only 1 Image object
and 1 rgb array for maximum memory and System.gc()
optimization.
An example of how the code could look like
to make a dynamically animated image:
// Create the array
int[] rgbarray = new int[w*h];
// We write in rgbarray the first image RGB's
writeSomethingIn(rgbarray);
Image img=null;
// Create the image from rgbarray data:
try {
img = Image createImage(rgbarray, 0,w,h,w);
}
catch( ArrayIndexOutOfBoundsException e ) { }
while(looping) { // Show an animation
// Each frame we change the array contents to make
// some kind of animation
writeSomethingIn(rgbarray);
// We write in rgbarray the updated image RGB's
editImage(img, rgbarray, 0, 0); // Refresh: image = latest rgbarray content.
// this include a newPixels()-like call,
// so even if we specify the same array
// previously specified, it's contents
// are copied into the image.
Graphics.drawImage(img); // At this point, we should make sure the
// Image painted is the latest and it's
// completed. In Java J2SE the Image updates
// include Imageproducers in multithreaded
// environments and may be risky to plot
// without mediaTrackers or similiar to
// wait the image update completion, in
// MIDP this does not exist.
}
3) A method to retrieve RGB array from an Image object,
either loaded from a PNG or created from an array,
like the pixelGrabber() method in Java2:
public void getImageData(int[] rgbData, int offset,
int x, int y, int width, int height, int scanline)
throws ArrayIndexOutOfBoundsException
int[] rgbData: the array where image data will be written;
such image data may be not the same as passed
to same image with a createImage() in case the
device display is not 24bit: lower bits may
be lost (set to zero) if native image format
does not store them to save memory.
int offset: offset from start of rgbData, where start writing
int x, int y: coordinate from where start copying inside Image
int width, int height: amount of pixels to copy=w*h
int scanline: if == width copy continuously, otherwise this
may be useful to copy parts of horizontal image
width (copied int[] rgb rectangle w < image w).
********************************************************
NATIVE Colors, and Image content access
********************************************************
4) As MIDP 1.0 is now, you can detect if colors or grayscales
are present, and how much colors or grays are available:
myDisplay = Display.getDisplay(this);
// Check if color is supported or not:
boolean colors = myDisplay.isColor();
numcolors = myDisplay.numColors();
if(colors) {
System.out.println("colors number: "+numcolors);
}
else {
System.out.println("grays number: "+numcolors);
}
That system seems to belong to an indexed mode with
a palette, whole instead all methods in Graphics class
accepts only RGB values 0x00rrggbb, apparently 24bit depth.
But, most device displays does not support natively 0xAARRGGBB
format (supported natively mostly by PC graphic cards).
LCD displays may have native modes with 2, 256, 4096, 65535
colors as 1, 12, 15, 16 bits, indexed modes with fixed or
dynamic palette, or other.
Could be possible to obtain the "native" color value
at startup with extra methods like:
int Graphics.getColorOfRGB(0xrrggbb);
int Graphics.getColorOfRGB(int r,int g,int b);
int[] getNativeImageData()
If native image format require only a short
or a byte for each pixel:
short Graphics.getColorOfRGB(0xrrggbb);
byte[] getNativeImageData()
This would even save memory.
Then, proceed directly operating with native color
data obtained with methods like:
- Graphics.setNativeColor(int col);
- Image.editImageNative()
Since some devices will finally operate natively as bitmasks
(2 colors), indexed (4-4096 colors), 15-16 bit RGB etc.,
all the graphic methods will have to execute a slow converter
algorithm for each operation. Passing instead the data
in the final format (or very similiar) will remove all
unnecessary conversions, reads/writes, extra memory needs.
Of course supporting different native formats in a
platform independent API would require dozens of extra methods
just for detecting, and entering/retrieving data's in all
the possible available configurations.
Native color/image access it's more likely to be a feature
for the various com.manufacturer.* API added to MIDP sets,
with just the native methods for their particular device.
For paletted systems some dedicated methods may be added:
boolean Display.isDynamicPalette()
int[] Display.getPalette()
Display.setPalette(int[])
An example usage of native methods could be to
load a PNG image, copy it's contents into
an RGB array with getImageData(), or place in
such array a serie of colors we want to use,
then convert all:
for(int i=0; i<w*h; i++) {
nativearray[i] = Graphics.getColorOfRGB(rgbarray[i]);
}
Or add a method to directly handle this job like
Image.getNativeImageData().
A problem in whole native image processing at once
may arise, in the case the image format is not
ordered like CRT ones:
0 1 2 3 4 5
6 7 8 9 10 11
12 13 14 15 16 17
In the case native image format is differently
ordered, the getNativeColor()/setNativeColor()
does not have problems, but get/set Image may
be not easily editable if you don't know how
pixels will be ordered.
In this case we may use a semi-native format
where the pixels ordering should be at least
assured. Or, use a com.manufacturer.* non
standard API for each vendor with specific
standard and documentation.
Some examples of native format methods to add:
int Display.getRGBformat();
For 24 bit displays, it should return "888"
For 15 bit displays,
It may return "555", meaning color bit-mask:
111111
5432109876543210
XRRRRRGGGGGBBBBB
"565" for color bit-mask:
111111
5432109876543210
RRRRRXGGGGGBBBBB
// Detect raw/native pixels system:
int bitsRGB = Display.bitsRGB();
int RGBformat = Display.getRGBformat();
if(bitsRGB!=-1) { // RGB mode?
if(bitsRGB==24) {
PixelCol = colRed<<16 | colGreen<<8 | colBlue);
}
else if(bitsRGB==15|bitsRGB==16) {
if (RGBformat==565) {
PixelCol = (colRed<<11 | colGreen<<5 | colBlue);
} else if (RGBformat==555) {
PixelCol = (colRed<<10 | colGreen<<5 | colBlue);
}
}
else {
// indexed mode
}
Finally, for the phones with simply 2 colors, we
may add the following methods:
Image.setBitmap(byte[] bitmask)
byte[] Image.getBitmap()
So we specify 1 bit for each pixel: 001001110...
This is much more efficient and specific if Display.numColors()
results = 2, than using the other methods.
********************************************************
fillPolygon() or at least fillTriangle() supported
********************************************************
5) Have a Graphics.fillPolygon(), or at least a
Graphics.fillTriangle() method added. It's
important to optimize the code for triangles plotting.
In fact, most of the 3d is plotted via triangles and there
are special drawing algorithms to speed-up the special
case of polygons with 3 vertexes. This could also
permit better 2D vectorial graphics like the "macromedia
Flash" technology for html pages.
********************************************************
Audio support (at least a "beep"!)
********************************************************
6) Pure MIDP 1.0 midlets have to be mute, silent...
Some vendors, such as ntt-docomo added an API able to play
a .MID modified subset, up to 16 voices, called .MLD.
These files are very short, usually less than 1Kb per
melody, and are sufficient to add special effects and
background musics for games and applications.
Considering MIDP resources constraints supposed for
graphics, I think playing PC formats like .WAV or MP3
definitely out of these limits.
Even if some MIDP devices will be able to play MP3
or WAV files using their special hardware, this is not
supposed to be a generic available capability.
Instead, I supposed when some audio capabilities are
present on the devices, these should all include option
of generating different tones with different play
times and pauses between tones, at least one voice.
The best formats for this are subsets of .MID in my
hopinion, similiar to the ones downloaded often from
the net to change cell phones rings. I would not require
a conversion to .MLD or other custom format however, since
.MID is a standard and could be easily subsetted by
providing a simple limitations list (voices, instruments,
and so on). This would require new Midi object and 3
methods like:
Midi audio1 = loadMidi("123.mid"); // Load midi
audio1.play(); // Play midi
audio1.stop(); // Stop midi
I would also add the option of creating beeps/tones and
melodies in custom way:
Midi audio2 = new Midi(); // Create empty midi
audio2.addTone(freq1(+freq2?)-or:instrument,volume,duration_in_ms)
audio2.addSilence(duration_in_ms)
audio2.addTone(freq1(+freq2?)-or:instrument,volume,duration_in_ms)
...
This could allow to create music/audio FX's from algorithms
too, and permits to make "piano" or audio editor midlets.
Having the opposite function (get tones+silences from .MID)
would even allow to load MID's and edit them or compose
longer MID's from smaller MID pieces and custom tones:
// Load 2 mid
Midi audio1 = loadMidi("123.mid");
Midi audio2 = loadMidi("456.mid");
// get the 2 mid tones into int[] arrays
int[] tones1 = audio1.getTones()
int[] tones2 = audio2.getTones()
Midi audio3 = new Midi(); // Create empty midi
// Copy all or part of audio1 melody into audio3
for(int i=0; i<tones1.length; i+=4) {
audio3.addTone(tones1[i],tones1[i+1],tones1[i+2]):
audio3.addSilence(tones1[i+3]):
}
// Add custom tones to audio3
audio3.addTone(freq1(+freq2?)-or:instrument,volume,duration_in_ms)
audio3.addSilence(duration_in_ms)
// Copy all or part of audio2 melody into audio3
for(int i=0; i<tones2.length; i+=4) {
audio3.addTone(tones2[i],tones2[i+1],tones2[i+2]):
audio3.addSilence(tones2[i+3]):
}
We have now a playable audio3 melody made by part
of audio1, audio2 and custom added tones.
Additionally, some detect methods are required:
boolean hasAudio(); // false = silent device
int audioVoices(); // How much concurrent voices are playable
int synthQuality() // Some parameter to detect max
capabilities of audio wave generation,
Maybe min/max volume defaults of various devices may
vary, so a "middle" volume setting in MID's for a game
melody may be too high or noisy on some devices; setting
the "general" midlet volume should be possible too:
setVolume(int volume)
This should multiply or divide the volumes of all the
tones in all the MID's.
If some devices can't even play single voice, tone based
beeps defined in a .MID subset, but instead for example
only 4 "fixed" predefined tones, we may add these methods:
boolean onlyDefTones(); // return true if no custom
tones can be played
playOk(); // Play the tone for "OK" operations
playCancel(); // Play the tone for "CANCEL" operations
playAlarm(); // Play the alarm ringing
stopAlarm(); // Stop the alarm ringing
... // Others
These generic playSomething() methods will be of course
available also if onlyDefTones()==false, as long as
hasAudio()==true.
When coming to playing waveforms (recorded audio), things
become more complex.
Audio waveforms have different bit depths (4,8,16 or more),
and Hz frequency, and may be mono or stereo.
While the CD quality is 16 bit, 44.1 KHz, stereo, we have
probably to target lower quality audio, mostly due to memory
space occupied by such formats (despite replay quality which
could be obtained).
I suggest 8bit, 8Khz mono, like the Sun .au which applets
can play, with addition of 4bit, 4Khz subset, or intermediate
8bit, 4Khz and/or 4bit/8Khz. The .au format looks ok, it's
a political matter if support .WAV, .AIFF or other.
We may support RAW 4bit/8Khz audio files too: audio is not
much compressed (except loss-quality algorithms such as
mp3, real media or wma), and so we may simply place raw
audio waveforms in the .jar file and benefit from the .jar
compression which is only a bit less efficient than .au
or .wav over audio type binaries. This save the extra
buffer memory and decompression work in exchange of a
small % of extra space of jar file (may be 5%? Someone
can make tests on this).
Methods to add may be:
AudioClip clip1 = loadAClip("audio1.raw");
clip1.play();
clip1.stop();
Extra detection methods may be:
boolean playClips(); // false = no audioclip play capability
Replay of different bit depths, Khz, and stereo support
may be covered and detected at startup with extra methods,
if required.
Fabio Ciucci
Anfy Team company CEO
Andrea Fasce
Anfy Team company CTO
www.anfyteam.com
Copyright © 2000/2001 Anfy Team di Fabio Ciucci. Java™ and all Java-based marks are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. |