Pushing the Boundaries - Slight Return :)

I can't stay away can I.....LOL? I've literally been pushing and testing the boundaries of the CFS3 terrain engine this past week. I've also been working on and off, in between times, these last couple of months on new terrain and water polygons for the new CFS3 ETO project. In the last week, with initiatives in the CFS3 forums at Sim-Outhouse, to create an Eastern Front add-on for CFS3 I decided experiment with a mesh for that. As the Eastern front covered a large geographic area I decided to test for myself just what could be achieved as to terrain size.

First up I created a dummy theatre the same size as CFS3 ETO, only moved east, using the paramaters in the terrain SDK examples. Using some test airbases in a simple gsl file, the theatre worked. Ok next up I decided to test a mega terrain. The terrain map stretched from S.E UK in the west, to Iraq in the east, southern Scandinavia in the north and the northern part of North Africa in the south. I fully expected CFS3 to croak, or rather not start. However CFS3 started as normal and I was able to fly from a couple of my test airfields. However optimism and satisfaction were short lived when I created further test airfields in the global scenery layer and attempted to fly from them, the game seemed to clip the theatre size by itself. However at that stage I did not create a new lcf (landclass) file for the appropriate theatre size and in my earlier experiments had made an error with the image and landclass stride parameters. I will redo the experiment later to make sure. I had been in the process of creating a large map of another geographical region before the Eastern Front project came up.

I would also emphasize that for test purposes, to speed things up, I was only compiling 1km resolution mesh down to a 1km stride. It only takes a few minutes to compile terrain at this resolution. Full resolution terrains can take hours (or days) of computer number crunching to compile.

On to my findings - firstly I would say that either that some of the terrain SDK examples are incorrect or there is a bug in the ers2tiff.exe tool. Creating a terrain mesh for CFS3 is relatively easy once you have source dem data. I use a spreadsheet to calculate the approximate bounds of the terrain data. The only numbers that ers2tiff.exe requires is the theatre origin, image pixel dim, image stride, max height, outX1, outY1 (terrain extents) and the number of images in the X and Y axes. The ers2tiff tool calculates the actual extents and theatre boundary from this information. This tool will automatically clip the supplied terrain to the appropriate size given the theatre minimum extent and the theatre origin.

In CFS3 geographical co-ordinates are relative to the theatre origin i.e. the theatre centre. The terrain SDK provides example parameters of minimum co-ordinate extents of the output data, in meters relative to the origin. The SDK further states that extent of the default ETO theatre is the rectangle (-786432,-786432) to (786432, 786432). However the output parameters that ers2tiff produces using appropriate dem data are (-786432,-786432) to (785408, 785408). (785408, 785408) is the north east co-ordinate of the terrain boundary in meters relative to the theatre origin. 785408 is 1024 meters less than 786432. In other words it truncates the data by approximately 1km in the North East. Further tests reveal that regardless of minimum terrain extent or theatre origin, the tool always outputs terrain extent parameters with the the NE parameters being 1024 meters less than the SW. I don't know whether this is a bug or by design. I do vaguely recall reading posts of a 1km srtm terrain bug in FS2004 & maybe FS2002, in forums in the past - maybe it's the SRTM data (or maybe it's my imaginative memory - someone may be able to clarify)?

This may be pertinent to one problem I have discovered. If we move the theatre origin north, or create a theatre larger than the default theatre, and fly into, or create an airfield in the NE corner of the theatre and fly from it, CFS3 crashes. Everything else appears to work correctly. One can only really test the bounds of CFS3 by flying in the game until one encounters the theatre boundary. The sim will soon let you know when you're leaving the theatre.

I cannot see any input in any of the xml files that tells cfs3 how large the theatre is apart from the files in the campaign's folder and mapdat.xml. CFS3 appears to ignore these as one can use any old numbers and the theatre will function ok in quick combat. The lcf file seems to have a bearing on theatre functionality in the sim and that may set boundaries, or CFS3 may determine theatre size from the terrain archives in the mesh zips.

Anyway after exploring boundaries by flying in the simulation, CFS3 appears to calculate its boundaries based on distances calculated from it's centre origin. As we know, the earth is round thus the distance per degree of longitude or latitude differs wherever we are on the planet. A degree of longitude (E/W) is 111.32km at the equator but 55.8km at a latitude of 60°N. CFS3 calculates its longitude boundaries using 1/2 the theatre dimension size east or west along the latitude at its origin, e.g. origin latitude of 50°N (approx 71.7km per °), 30°E gives 15°19′46″E in the west 44° 40' 14"E in the east. CFS3 calculates north and south from the centre in a similar manner (distance for degree of latitude varies from 110.57km to 111.69km). Having created these boundaries CFS3 then uses a rectangle based on these geographic co-ordinates as its theatre boundary, which is ok as long as it's primary reference is geographic latitude and longitude. It's paradoxical to a certain extent given that many of the SDK calculations are based on meters usually based to the power of 2 or divisible by numbers which are to the power of 2.

Anyway the exploration continues. I will continue some tests with ers2tiff to see if it can produce output whose extent boundaries line up with the SDK documentation. Perhaps using a geotiff dem will produce different results...maybe dted0 dems will produce different results? I'll also experiment again with a mega theatre. If the NE corner bug is the only problem then maybe we can put up with that and caution users to stay clear....maybe a giant smiley...danger Will Robinson...danger Lol? I also hope to explore CFS3's landclass system....more on that later.

All this is very addictive...I keep trying to escape CFS3. There's nothing like a puzzle to maintain one's interest.

cheers

Rob

Last post on this blog for a while.

I'm rationalizing my blog activities somewhat. This blog was established to blog some of my activities in exploring CFS3 scenery creation, largely as a from of recreational activity while waiting for cancer treatment and surgery. Hopefully I am now on the recovery trail and I can now again focus on my core hobby activities such as virtual aircraft creation, and of course real life and ongoing cancer treatment.

I ended up creating a semi-functional Mariana Islands theatre (Guam, Saipan etc) but did not get time to create new land class or scenery objects. However I do hope to return to this someday. I'll be monitoring CFS3 PTO threads on the forums. I'm also looking for time to resume a F6F-3 model I started on last year along with a Helldiver. Hopefully I will have time to create versions for both FSX & CFS3...... sometime in 2008..... though don't hold your breath.

I will not close or delete this blog at this time for reasons mentioned in the previous paragraph. I may (probably will) dabble in FSX scenery creation.

Anyway for the time being.... all the best to my very few readers.

Rob

Out of the Loop

I have surgery very soon & will be out of the loop for a while.

cheers

Rob

October update

I posted an update on my activities in the Biff Diaries. I'm mainly working on 3D projects for relaxation at the moment. I'll return to scenery dabbling in coming weeks/months ;)

In the meantime there are some nice pics of CFS3 scenery work by others, being posted around the web.

http://offaerodrome.com/OFFweb/gallery.htm

and

http://www.sim-outhouse.com/sohforums/showthread.php?t=52790
http://www.sim-outhouse.com/sohforums/showthread.php?t=51783

cheers

Rob

Change of activity

I have eased up on scenery creation for a few weeks, mainly given radiotherapy & its effects. I have also switched over to mainly focusing on aircraft modeling for a few weeks. I will be switching between activities until I recover next year. I'm in a wee bit of pain at the moment.

I still intend to work on my scenery with new trees & textures. As for trees it's a matter of motivating myself to take a few photographs of trees (I'm surrounded by them) for the textures & I will composite ground textures from a variety of free sources including hand painting some of my own. For copyright reasons it is best NOT to use and mod FSX textures. It would be illegal to distribute such to others as not every CFS3 user owns FSX.

I will also be continuing work on my landclass system and integrating the above into it.

The problem with lines on water looks as if it will be difficult to get rid of. The CFS3 terrain SDK tools and CFS3 scenery engine was not designed with SRTM & SWBD data in mind. The terrain features a LOD system, unfortunately the water system doesn't. The lo-res terrain lods intersecting with the high-res SWBD data causes the water polys to bend where they intersect with with the terrain causing lines to appear. This can be seen occasionally in the CFS3 PTO Solomons theatre. I think the only way they can be reduced (not eliminated) is to use hand drawn coast line vectors, a massive task. If one takes a look at the coast lines in FSX they are all fairly lo-res coastlines.

I did start to create a terrain map for the Philippines, however creating the coastline vectors is a massive task.

In the meantime I hope to post a bit more on aircraft creation in the Biff Diaries in the next couple of weeks.

Rob

CFS3 Marianas

I can now reveal the primary area that I am working on (there are two more) - The Mariana Islands. To date I have the terrain mesh, ocean polygons, & shorelines working. There are minor issues such as airfield facilities distorting terrain mesh near coastlines - just a matter of playing around with airfield altitude in the global layer. As I mentioned in the previous post I have found that hand editing or hand creating coastlines works best.

Next I move on to landclass & scenery objects, that will take me a couple of weeks. Presently the terrain only has one woodland landclass just to get the terrain & ocean into CFS3.

Here are a couple of pics - one of a rather bare Apra harbour and the other two of Pagan Island, which Japan held until late in the war and used to raid Saipan.



Rob

Update Sunday

My CFS3 terrain experiments still continue. I managed to create a group of islands at an image stride & LOD of 64 meters which gave me a buzz this morning. What amazed me was that frame rates were barely affected. The verdict is out as to whether mesh at this resolution is worthwhile. The 64m LOD took all night to compile and takes about 600mb space. A 128m LOD (at 64m stride) only takes a couple of hours to compile and is about 140mb in size.

As to coastline data, I find that hand digitising of coastlines worked best in my current theatre. Of course such is a bit unweildy in areas with hundreds of islands. The problem with the CFS3 terrain engine & tools is that if a shoreline lines up with terrain at an altitude of more than a couple of meters it produces a sliver like spike radiating from the shoreline across the ocean, like a mini tsunami if you get down close and have a look at it. I notice a few in the Solomons theatre as well as my own experiments.

Anyway on to tidy up my coastlines - there are still a few glitches. The next step will be landclass & scenery, it's all very generic at this stage. This will keep me going maybe a couple of weeks.

Partial Success

I now have a couple of CFS3 theatres partially up & running. At this stage I have only completed the basics to get a map area operational.

Currently I have to commute daily to hospital for radiotherapy, which can take up to five hours out of my day if I have lunch & do a bit of shopping. Transport from my home on the island, to the hospital only takes just over an hour, but the return trip can take 3 hours given timetable gaps in the public transport system. Thus my experiments in terrain creation are somewhat punctuated. I also hope to resume aircraft creation once I am sure I have the main issues in terrain/scenery creation under control. All this is for relaxation LOL.

I have experimented with mixed results. So far I have yet to see any benefit using the CFS3 SDK with hires mesh, though I have more experiments to carry out this week The tiff2msh tool is where things currently fall down as to resolution. I also experienced interesting results with the landclass system which I will fully explore, once I have a nice terrain mesh working.

Also problems with water occur around some of the coastlines causing spikes in the water polys. I am trying to determine whether it is a DEM problem or a problem with some of the SWBD vector data, or whether it is a CFS3 terrain SDK issue or a Global Mapper issue.

I am being a bit circumspect in writing this. I'd like to disclose a bit more, especially some of my explorations as I go along. However with rivalry situations that exist from time to time in various CFS & FS communities I'll keep quiet until I have something to show. I am doing this for relaxation and don't want to be stampeded into competitive situations, which I can see happening.

The other person creating terrains & theatres for CFS3, Ed Wilson, is quietly getting on with his creations just like me, and I have no problems with that and I admire his achievements. After all MAW did inspire me somewhat to experiment with terrain/scenery creation. There are others however, who are prone to creating competitive situations.

One thing I will say though, is that those who liked what I acheived with the CFS3 & FSX Bristol Fighter, can look forward to the same approach in the world of scenery.

However one thing at a time.

cheers

Rob

Update

Following on my last post it appears that I am not the only person working on a Solomons theatre for CFS3 as witness the current news on the Korean Skies page.

In many ways that is good news and I am generally pleased. I don't object to anyone else making terrain for the Solomons or elsewhere. There are heaps of areas I want to create & experiment with and I could have possibly chosen an other area to start with. Anyway I wish Ed well with it.

I have learned a great deal in the last week. The CFS3 SDK tools to convert DEM files to mesh appear to work ok. They even allow the creation of rectangular as opposed to square maps. Unfortunately this is not the case with tool that converts landclass maps to lcf files. It only accepts square maps. Why the rest of the SDK allows for rectangular maps and the landclass tool does not is beyond me... you should have heard me cursing.

I am currently working on water polygons derived from SWBD data. Unfortunately the data contains shapes for many of the islands instead of empty space. I have to convert many small islands to holes in the water by hand....painstaking & I still don't know how the CFS3 water tools will cope with such hi-resolution data......more cussing on the way perhaps? LOL

Anyway back to square one. I will still persevere with the Solomons area as my learning & test area given that I have come so far and carried out much research. I will at least get it into the sim. It will be interesting to see how it compares to the Korean Skies version. Whether I will ever release it..... who knows? I have a great many theatres large & small in many parts of the world from 1914 to 1960's that I would like to attempt. Of course given the contrary way the world operates MS might surprise us all and announce a new CFS.... that would be a long time coming if they did... plenty of time to create for CFS3 ;)

cheers

Rob

Ready to start Solomon Islands theatre

This week I am ready to start moving things into CFS3. My main goal is to get things working using as much of the original CFS3 installation as possible. Later on I can experiment with higher resolution data. However the CFS3 scenery system is difficult enough as it is without trying to improve things from the outset.

The region I am working on is centered at S7.5 , E152.5 and is planned to cover a similar area to CFS3 Europe.

I have been obtaining the appropriate data and checking it out and assembling in my primary GIS application Global Mapper. The CFS3 terrain SDK is certainly making sense now. However I still have to check out all the SDK tools. There's much work in all this & I don't want to overdo things. I also had a nasty flu virus last week which delayed matters. Hopefully I am on the mend now. Radiotherapy begins this week so I'll have less time for creative endeavours.

The data is split into 3 main types -

• topological DEM data - to produce terrain mesh.
  
• vector data - for coastlines, waterbodies, roads & rivers.
• land coverage data - for the landclass system.

The landclass side of things is still WIP at the moment. I want to take a look at the existing CFS3 landclass system & textures before moving ahead with mine. I also have to find a way to convert from USGS data to CFS3 data & line it up on the map. I will add a few new landclass types to CFS3s existing types, but not too many as that will entail a lot of additional work. I will be refering to period maps too, as land use & coverage in the Solomons area differs today from that of the 1940's http://www.lib.utexas.edu/maps/historical/pacific_islands_1943_1945.html Much of the landclass info will be hand edited in a paint program.

As to DEM data, I had originally intended to use the most recent available 3 arc second data. As this is 10 times higher resolution than the data used for CFS3, I am not sure that the CFS3 engine could cope with it over a region as large as the CFS3 European theatre. One has a number of choices : -

• use that data and then use the numbers as in CFS3 & SDK examples. This will leave any resampling downwards to the CFS3 tools. The results of such would not be known. A lot of work just to experiment.
• use SRTM30 data. This is 1km data similar to the resolution used in CFS3. However this data is more recent and accurate than the data used in CFS3.
• use the 3 arc second data & down sample the DEM data in Global Mapper. This is what I have decided on as I will be using the most recent data available.

I intend to further experiment with hi resolution data later and would love to do this also with textures, however for now, using the above approach I will be able to create a theatre the same size as the CFS3 Europe map, using the same figures in the xml files and SDK tool examples.

Vector data was a big chore last week, especially coastline data. Rivers & roads are fairly easy. For this exercise I am using vmap0 data, available from NGA via their rasterroam site. It is easy to extract this data using Global Mapper (the Swiss Knife of GIS applications.). Water body vector data is easy too. I am using SWBD data derived from SRTM data. It is good resolution, though perhaps stepped along the coast lines. I'll have to see how it looks or whether it is noticeable in the theatre.

Even though we have this data we still need vectors for the coastline, this is where the ordeal began (along with the flu), involving many hours of trial and error. I found the vmap0 & NOAA coastline data to be too lo-res & didn't line up with some of the smaller islands. The SWBD data is in area format. The coastline data has to be in lines. Global mapper's convert area to lines function caused some of the smaller islands to disappear, probably due to flaws in the data and we are still left with the problem of deleting the lines comprising the tile borders, which also deletes parts of the coastline in Global Mapper. I needed to find some other way of editing the vector data and extracting and converting to line shape data. The solution - export the vector data to dxf format & import & edit in 3DS Max (this may work too in gmax, though I haven't tried). This was somewhat unwieldy, time consuming and involved much trial & error. Eventually I got there, imported the edited DXF into Global Mapper and then re-exported as a line vector shapefile, now ready for the CFS3 sdk tools. The SDK shoreline tool is reputedly finicky & again much experimentation will be required to find the correct resolution.

A few links

DEM data is obtainable from a number of sources :-

• http://seamless.usgs.gov/ This is not the most recent data, however one can select a region for download from a seamless map intereface.
• http://srtm.csi.cgiar.org/ This is the most recent 3 arc second SRTM data available. The site also provides a link to a Google Earth interface where one can zoom in to download links. http://www.ambiotek.com/topoview (requires GE on your computer). This is the data that I used.
  
• ftp://e0srp01u.ecs.nasa.gov/srtm/version2/SRTM30 - global DEM data at 1km resolution.
• ftp://e0srp01u.ecs.nasa.gov/srtm/version2/SRTM3 - SRTM 3arc second if you know the tiles you want to download.

Other data sources :-

• http://geoengine.nga.mil/ NGA's raster roam site - both dem & vector data including vmap0 vector data
• http://www.ngdc.noaa.gov/mgg/shorelines/shorelines.html - National Geophysical Data Center (NGDC) - coastline data.
• ftp://e0srp01u.ecs.nasa.gov/srtm/version2/SWBD/ - SWBD water body data derived from SRTM data.
• http://landcover.usgs.gov/glcc/index.php - Global Landcover data.
• http://www-gvm.jrc.it/glc2000/ - CGLC2000 - EU global land cover data.
• http://glcf.umiacs.umd.edu/data/landcover/ - GLCF - AVHRR Global Landcover Classification.

Hopefully I'll have a rudimentry theatre up and running in the next 10 days or so.

Rob

Initial research

Much of this week has been involved in aircraft modeling. The rest of the week has been devoted to research.

I have been focusing on the Solomons Islands area as the initial area of interest (for CFS3). Initially I am considering using SRTM 3 arc second DEM data (i.e. 90 metres between grid co-ordinates.) I think CFS3 Europe used the old gtopo30 or dted0 data for its data (30arc seconds), which is at 1km resolution, thus SRTM 3arc data is 10 times the resolution. Until I actually attempt to create the theatre I don't know how the CFS3 engine will handle it. Given the large areas of sea in that locale it may work. Failing that, SRTM30 data will work.
I just want to add that there is even higher resolution terrain data available, SRTM 1 arc second (30 metres). Presently this is only publicly available for the USA and a few other countries. However there is free publicly available 30M data for the Hawaiian Islands. Maybe someone will create a Pearl Harbour theatre at this resolution?

As I mentioned above, much of the week has involved research. This has involved finding as many images as possible of that area, both present and past; researching the WW2 history of that region, indeed the South West Pacific region; and also aircraft used during ww2 in the Solomons. Yep I hope to model a couple of aircraft for the Pacific theatre. I have ordered a book on the Nakajima A6M2-N Rufe and will probably start that soon. Another aircraft has yet to be decided. This afternoon I have been researching the ground cover of the islands which was largely rainforest interspersed with copra plantations.

Creating a new theatre such as this will involve much work, not only terrain, but texture tiles too and re-working of the landclass system. So the theatre will take a bit of time.

The CFS3 terrain SDK is starting to make a little more sense to me now and hopefully I will decide on the initial test area in the next couple of days and start processing.

I still intend to post some links on various resources from history to GIS ... so keep watching (all two of you...if that - LOL).

cheers

Rob

Terrain Creation for CFS3: Starting off & Why

After primarily focusing my simulation related efforts on FS2004 & FSX for 15 months, returning to CFS3 feels a bit strange. I only recently installed CFS3 to allow installation of OFF Phase 2 & MAW. From the scenery perspective both of these add-ons offer considerable improvement over stock CFS3.

I must say after flying FSX for over a year both beta & release, the first thing one notices when flying CFS3 is the low resolution of the terrain textures. Actually I'm finding them not as bad as I first thought. The ground textures in CFS3 are slightly better than the stock terrain textures in FS2004, however FS2004 wins over CFS3 with building autogen. FS2004 terrain textures are more aliased, jaggy, than CFS3. FSX's 1024x1024 terrain tiles offer a considerably sharper experience than the other two MS sims. Aircraft in CFS3 however do look nice & sharp and are well shaded, and while CSF3's material system is not as fully featured as FSX, offers more options than FS2004.

Anyway my first step is to gain an understanding of CSF3's texture rendering methods, which employs a layered technique modulated by xml mos files. This will involve systematic analysis of all stock ground texture files in CFS3 and pouring through the CFS3 Terrain SDK documents.

I'm also trying to decide on an area to test first. Presently I am looking at the Solomon Islands area and have carried out a few days research on the geography, WW2 history of that region, and looking at the available GIS data. I have also had a quick gander at Spain. I may end up doing something totally different, perhaps a small area.

I have already set up a second CFS3 installation to form the basis of my test region. Today I will set up an additional installation basically to experiment with the guts of CFS3. I know a few others have done likewise however I believe in finding out things for myself and also greatly enjoy experimentation.

Why CFS3?

Why am I focusing on CFS3? A number of reasons - There is no indication of another combat flight simulator from Microsoft and given their current projects i.e. DX10 for FSX, Acceleration Pack for FSX and Train Simulator, Aces will be occupied for quite some time. MS/Aces will also have to start developing FS11 soon too, given that it takes a couple of years to produce a version of FS. There are some other interesting things happening at Aces too, as witness advertised jobs.... Montauk, but no sign of a combat simulator. Given their current projects, even if MS decided on a CFS, we wouldn't see one until 2010 or 2011 (unless of course, there is an ultra top secret one on the way). That's a long time to go without.

Aviation history & also military aviation are consuming passions for me, and FSX while being very nice graphicly just does not cut it from the historical perspective. Even if MS do not add combat to Flight Simulator, they have to, at least consider adding the fourth dimension i.e. time & history to the simulator, the ability to have autogen & scenery appear by date, similar to CFS3's xdp files for sim objects.

Thus CFS3, despite its shortcomings offers the only possibly venue, for me at least, as a sand box to explore & satisfy my interest in military aviation history, in the short term. I do hope to create terrain & scenery for FSX at a later date too... especially my local area. My aircraft creation activities will continue too. Now that much of the learning curve for FSX is behind me I can spend more time on scenery creation & CFS3.

I am also inspired by especially by MAW and also OFF.

I expect progress in the coming months to be haphazard given health matters. So don't expect anything too soon.

Some resources -

• CFS3 Terrain SDK
• Martin Wright's tutorial on the scenery SDK. Also CFS3 SceneDB and his multi CFS3 utility.
• Some useful material on the Korean Skies site (on the Maintenance page).

I'll a post progress report later in the week & hopefully some links on GIS resources.

cheers

Rob

An Introduction.

Hi - this blog will focus on my experiments in the world of scenery & terrain creation for Microsoft Combat Flight Simulator 3.0 and Flight Simulator X. I also hope to post on my experiments in the world of GIS.

The intention is to suppliment & compliment my existing flight sim & 3D creation blogs, The Biff Diaries and The Wing Fell Off.

I find the world of GIS & its myrad of data formats & information systems somwhat bewildering & confusing for the beginner. I have dabbled with terrain creation on the past using software such as VistaPro on both the Windows & Amiga platforms. However most of that is at least a decade in the past. Things have become much more sophisticated in the interim. Likewise with scenery/terrain creation for MS simulators - especially CFS3. Terrain creation for CFS3 is very much a 'Black Art' and to date, only one person that I know of, has created new maps & theatres for CFS3.

Anyway my voyage of discovery in these black arts of terrain creation, GIS & scenery design has begun. Hopefully it will bear fruit. ;)

Rob