Performance enhancement
Hi All,
i'm accessing data from the server through HttpConnection to Blackberry mobile unit in the following format -
| 123 | abc | newyork | xxxx | 111 | dfgr |....|....|....|...............
| 234 | aaa | rock | gggg |... ....| ..... | .......
each pipe separated data elements represents a field.
I've to access the data in the above format, cause there is no other way that we can change the format. So there is question of changing the data format.
Each row represents a record.
Now the demo build that we've written is working fine with the few records and our client is happy with the performance of the system.
But when this build run on the real environment it 'll take lots of time to access the record, since there is record of every person in the United States.
Can anybody suggest me some idea to enhance the performance of this system.
[913 byte] By [
dave123a] at [2007-11-26 12:47:26]
# 1
please remove my record
# 2
> Hi All,
>
> i'm accessing data from the server through
> HttpConnection to Blackberry mobile unit in the
> following format -
>
> | 123 | abc | newyork | xxxx | 111 | dfgr
> |....|....|....|...............
> | 234 | aaa | rock | gggg |... ....| ..... | .......
That's the format that the Blackberry sees, but I'm hoping to God you're not storing everything as records on the server.
Ever heard of relational databases? Indexes?
> each pipe separated data elements represents a field.
That's obvious - we can see that.
> I've to access the data in the above format, cause
> there is no other way that we can change the format.
Yes there is. Put it in a relational database. If you've got it in a flat file and can't change it, give up now.
> So there is question of changing the data format.
The data format that the application sees doesn't have to be the same as the storage schema.
> Each row represents a record.
Stop restating the obvious. We get it.
> Now the demo build that we've written is working fine
> with the few records and our client is happy with the
> performance of the system.
Right, that's the problem with demos. If you don't know what you're doing, and don't think about scalability, you can screw yourself. Like you already have.
> But when this build run on the real environment it
> 'll take lots of time to access the record, since
> there is record of every person in the United
> States.
Remove mine, too. 300M records isn't a lot for a relational database. Most companies deal with terabytes of data.
> Can anybody suggest me some idea to enhance the
> performance of this system.
Sure:
(1) Find a way to index your data so you don't have to scan through every record to find an individual. You want an indexed search that is O(1) instead of the O(n) scan you have now.
(2) Only send the information back that is needed for the client. Minimize the bytes on the wire.
Sounds like your first Blackberry app, maybe first of any kind. You did a quick "Hello World" app without thinking about layering or scaling, and now you're screwed.
You might want to hire some professional help to bail you out.
%
# 3
Hi duffy,
I said that we can't change the way the data we are accessing. because the server side is allready developed.
Web Layer is developed using - PHP 4.0
DBMS used : mySql.
Here only problem is that the time of accessing the data from the website to the Blackberry unit.
the things that you've said indexing etc. is allready being done. from accesing information through the brower is not the problem. it works fine.
And also i'm not saying that i'm going to load all the 300m records into the unit. I'm not that stupid the way you are thinking about me. we're only loading the required information precinct by precinct.
In the login time the program download the record of 1 or more precinct. After every 3 minutes the data'll be refreshed and new refreshed data 'll loaded into the Blackberry unit. Also the user of the BlackBerry unit can load the record By simply typing "LOAD" command.
No performance problem is there after loading the records. No threading problem etc.
The thing about layering you'd mentioned allready being done.
The Mobile part of the application comes after developing the application as an enhancement work. It was first not realised by the client of ours.
Now I'm going to describe you the way i'd layered the BlackBerry part of the application. Other parts (web layer, dBMS) is not my concern.
response
<--|
request|
-->| Controller | -> | Presentation |
|
|
| Session | > | DAO |
|BlackerBerry Layer
..................................................................|..........................................................
Web Layer| Web Site |
|
....................................................................|...........................................................
| DATABASE |
................................................................................................................................
So, now here my performance concern lies -
| DAO | --> | Web Site |
Now i'm thinking about geting the data in the compressed format the decompressed the data in the mobile program.
But i've to do some R@D for that.
Now i'll come to your answers -
Ever heard of relational databases? Indexes?
> we can't access the database directly from RIM API's that are available
Yes there is. Put it in a relational database. If you've got it in a flat file and can't change it, give up now.
--> Do you think i'm that much stupid.
The data format that the application sees doesn't have to be the same as the storage schema.
> I know that.
Right, that's the problem with demos. If you don't know what you're doing, and don't think about scalability, you can screw yourself. Like you already have.
--> i'm still not screwed till now, cause the demo worked in your councillor election beautifully. But i konw in the president election i'm going to be screwed, if i don't find any sollution.
** Duffy i know about you and i belive I'll get some good idea from you**
-Remove mine, too. 300M records isn't a lot for a relational database. Most companies deal with terabytes of data.
> You are in my ****Most Wanted List ***
You might want to hire some professional help to bail you out.
-> I'm not going hire anybody. If you people (Specially you) not going tto give me any idea then i'll do it by myself. But in the meant time i don't have any idea.
DUFFY I always appreciate your suggestion in this forum. PLZ I request you first try to understand the whole problem then give your remark. If any person doing some stupid sollution for there problems, PLZ Experienced person like you should so them the way they should go to
problem.
AT LAST I ALWAYS LIKE YOUR IDEAS. THANKS
Now i'm going to describe the problem behind
# 4
> Hi duffy,
>
> I said that we can't change the way the data we are
> accessing. because the server side is allready
> developed.
Doesn't mean you can't change it.
>
> Web Layer is developed using - PHP 4.0
> DBMS used : mySql.
PHP? This is a Java forum. What's the Java part?
> Here only problem is that the time of accessing the
> data from the website to the Blackberry unit.
>
> the things that you've said indexing etc. is allready
> being done. from accesing information through the
> brower is not the problem. it works fine.
>
> And also i'm not saying that i'm going to load all
> the 300m records into the unit. I'm not that stupid
> the way you are thinking about me. we're only loading
> the required information precinct by precinct.
So what do you want?
> In the login time the program download the record of
> 1 or more precinct. After every 3 minutes the
> data'll be refreshed and new refreshed data 'll
> loaded into the Blackberry unit. Also the user of
> the BlackBerry unit can load the record By simply
> typing "LOAD" command.
>
> No performance problem is there after loading the
> records. No threading problem etc.
>
> The thing about layering you'd mentioned allready
> being done.
>
> The Mobile part of the application comes after
> developing the application as an enhancement work. It
> was first not realised by the client of ours.
>
> Now I'm going to describe you the way i'd layered the
> BlackBerry part of the application. Other parts (web
> layer, dBMS) is not my concern.
>
>
> esponse
>
> --|
> request
>
>
> -->| Controller | -> |
> Presentation |
> |
> |
>| Session | > | DAO |
> |
>BlackerBerry Layer
> ............................................|.........
> .................................................
> Web Layer|
> Web Site |
>
>
> ......................................................
> ..............|.......................................
> ....................
>
> DATABASE |
> ......................................................
> ......................................................
> ...................
>
>
> So, now here my performance concern lies -
>
> | DAO | --> | Web Site |
> m thinking about geting the data in the compressed
> format the decompressed the data in the mobile
> program.
>
> But i've to do some R@D for that.
>
>
> Now i'll come to your answers -
>
> Ever heard of relational databases? Indexes?
>
> > we can't access the database directly from RIM
> API's that are available
>
> Yes there is. Put it in a relational database. If
> you've got it in a flat file and can't change it,
> give up now.
>
> --> Do you think i'm that much stupid.
>
> The data format that the application sees doesn't
> have to be the same as the storage schema.
>
> > I know that.
>
>
> Right, that's the problem with demos. If you don't
> know what you're doing, and don't think about
> scalability, you can screw yourself. Like you already
> have.
>
> --> i'm still not screwed till now, cause the demo
> worked in your councillor election beautifully. But i
> konw in the president election i'm going to be
> screwed, if i don't find any sollution.
>
> ** Duffy i know about you and i belive I'll get some
> good idea from you**
>
> -Remove mine, too. 300M records isn't a lot for a
> relational database. Most companies deal with
> terabytes of data.
>
> > You are in my ****Most Wanted List ***
>
>
> You might want to hire some professional help to
> bail you out.
>
> -> I'm not going hire anybody. If you people
> (Specially you) not going tto give me any idea then
> i'll do it by myself. But in the meant time i don't
> have any idea.
>
>
> DUFFY I always appreciate your suggestion in this
> forum. PLZ I request you first try to understand the
> whole problem then give your remark.
I can only understand what is posted here. I react based on what I read. If you want me to understand more, please post it. I'm not a mind reader.
If everyone waits for perfect information, you'll never get an answer.
Besides, YOU don't even have a complete understanding. If you did, you'd be able to solve your own problem.
> If any person
> doing some stupid sollution for there problems, PLZ
> Experienced person like you should so them the way
> they should go to
> roblem.
Please spare us the lectures.
I have no idea what you should do, and after all this I've gotta admit that I don't care. Good luck with this.
%
# 5
My best recommendation is to be a scientist and get some real data as to where the problem occurs instead of guessing.Get a profiler onto your code and a network traffic monitor. See why the mobile code is slow with some real, hard-headed data. Anything else is
# 6
>
> Here only problem is that the time of accessing the
> data from the website to the Blackberry unit.
>
So I am guessing that you believe (rather than know?) that there is a performance problem in the following flow
1. Request goes to server
2. Data is returned from server
3. Client (blackberry) must parse the response.
Now exactly what is the maximum size of the response? 100,000? 1,000,000? What are the average sizes? What is the maximum and average size for a single record?
How many clients will be doing this at one time? What is the actual network bandwidth all the way from the server to the client?
As suggested if you have not actually used a profiling tool then that is absolutely the first step.
Other than that I would guess there are several possible bottlenecks
1. Transmission
2. Parsing
3. Memory (parsed result along with original result.)
# 7
**** So I am guessing that you believe (rather than know?) that there is a performance problem in the following flow
1. Request goes to server
2. Data is returned from server
3. Client (blackberry) must parse the response.
-- Yes, you are right
****Now exactly what is the maximum size of the response? 100,000? 1,000,000? What are the average sizes? What is the maximum and average size for a single record?
Maximum size is unlimited. but you can say average size is more then 100 million.
The maximum size of the single record is 5589 characters. average size is 4380 characters.
**** How many clients will be doing this at one time? What is the actual network bandwidth all the way from the server to the client?
more then 2 and less then 10 client is assigned to to work in each precinct. there are two different client
1) BLOCK CAPTAIN
2) POLL WATCHER
Block captain may be assigned more then one precinct and Poll WATCHER 'll be assigned one precinct. So in case of block captain there are more record to accessed in comparison to poll watcher.
# 8
> Hi All,
>
> i'm accessing data from the server through
> HttpConnection to Blackberry mobile unit in the
> following format -
>
> | 123 | abc | newyork | xxxx | 111 | dfgr
> |....|....|....|...............
> | 234 | aaa | rock | gggg |... ....| ..... | .......
>
> each pipe separated data elements represents a
> field.
>
> I've to access the data in the above format, cause
> there is no other way that we can change the format.
> So there is question of changing the data format.
>
>
> Each row represents a record.
>
> Now the demo build that we've written is working fine
> with the few records and our client is happy with the
> performance of the system.
>
> But when this build run on the real environment it
> 'll take lots of time to access the record, since
> there is record of every person in the United
> States.
how do you know? don't ever simply believe that something will under-perform, do some realistic testing. often the bottlenecks are not where you thought they would be. get the thing written, and working, then worry about optimizing it
> Can anybody suggest me some idea to enhance the
> performance of this system.
# 9
>
>
> Maximum size is unlimited. but you can say
> average size is more then 100 million.
>
> The maximum size of the single record is 5589
> characters. average size is 4380 characters.
>
4380 x 100 million = 438,000,000,000 = 438 gigabytes.
> **** How many clients will be doing this at one time?
> What is the actual network bandwidth all the way from
> the server to the client?
>
> more then 2 and less then 10 client is assigned
> to to work in each precinct. there are two different
> client
> 1) BLOCK CAPTAIN
> 2) POLL WATCHER
>
So two peopel request it at one time. If you have a 1 gig pipe all the way from the server to the blackberrys then..
Total transfer ~ 1,000 gigabytes
Transfer rate of 1 gig ethernet ~ 6000meg bytes per second.
Total transfer time = 30 minutes.
Now just doing some basic research the blackberrys I found.
- Memory 32 meg
- Max download speed of 2m bps wirless
I suspect that 500 gig isn't going to fit in 32 meg.
And that download speed means it will take 500 days to download that amount of data.
# 10
Going at that in the slightly opposite way...
Using 32 meg for nothing but data storage (including no structures) with 4380 bytes means you can store about 7300 records. Nothing else (in that 32 meg.)
Now with research it seems to indicate that an average polling station can have from 300 voters up to at least 3000. So certainly no more than 2 polling stations can fit in there.
# 11
Well, if one actually thinks about it, displaying more than 10 entry abstracts does not really make sense on a blackberry (not to mention the amount of data a person will grasp at a time). You will definitely work with paging and filtering, which, even if you'd download complete entries having 4380 bytes, makes about 42kByte of data at a time. Shouldn't give you any problems with memory or transfer speed.
As george said: Make it work. Then think about necessary optimization.
