Development of the CESNET2 Optical Network

Lada Altmanová, Stanislav Šíma

Lada Altmanová, CESNET z.s.p.o., Zikova 4, 160 00 Praha 6, Czech Republic, la@cesnet.cz
Stanislav Šíma, CESNET z.s.p.o., Zikova 4, 160 00 Praha 6, Czech Republic, sima@cesnet.cz

 

ABSTRACT

This article deals mainly with the implementation of the CESNET2 optical network for the purposes of the National Research and Education Network (NREN) of the Czech Republic. Attention is paid mostly to the data and experience that could be used for building optical NRENs in other countries. Increasing the number of devices for STM-16 and buying expensive routers and interface circuits for STM-64 for CESNET2 probably will not be necessary. Instead, new lines will be used for the gigabit Ethernet and that the most essential backbone routes will be converted to 10-Gigabit Ethernet. Last optical mile problem in Czech cities is discussed.

Keywords: Optical Network Development, Transition to Fiber-optic Network, Long-reach Gigabit Ethernet, Long reach 10-Gigabit Ethernet, Czech Education and Scientific Network.

1. Introduction

Current optical technologies allow construction of computer networks providing faster data transfer, lower cost of data transfer, equipment and service personnel and higher reliability. In some cases, new networks must be implemented in several consecutive phases, in which parts of an existing network are being replaced with an optical network. The implementation has its technical and economical aspects, which must be all analyzed to find an optimal solution.

This article deals mainly with the implementation of the CESNET2, a national optical network for research and education (NREN) in Czech Republic. Attention is paid mostly to information and experience, which can be useful for constructions of optical NRENs in other countries, and to problems, for which we would like to gain more relevant experience from abroad. Characteristics of the CESNET2 network are listed in the [1].

2. Transition to Fiber-Optic Networks

The transition to an optical infrastructure is now being performed in many NRENs. A relatively simple solution is to purchase SDH services operating at 2.5 Gbps (STM-16 or OC-48) or 10 Gbps (STM-64 or OC-192). These services are offered by telecommunication operators using fiber optics. However, this solution is rather expensive for NRENs, and from the technical point of view it is quite complex for the given purpose. It is often used as one phase of the NREN development.

When installing long-distance fiber-optic connections, NRENs have some specific equipment requirements. The equipment cost is usually more important than the increased fiber utilization (i.e., the use of WWDM, CWDM or DWDM) or the use of more colors in parallel to increase the total transfer speed. However, many manufacturers provide optical amplifiers, dispersion compensators and opto-electric regenerators only as a part of large expensive devices designed for telecommunication operators. Nevertheless, we expect that the increasing use of Gigabit Ethernet and 10-Gigabit Ethernet technologies and use of fiber optics owned or shared by current users will give rise to widespread requirements similar to those of NRENs and the manufacturers will provide suitable configurations.

For NREN, the lease of a fiber seems to be more convenient than the lease of a wavelength at the same price, for fiber length up to 80 - 100 km (the reason is the possibility to change the way of use and transfer speed without paying more money for the lease). This conclusion is based on experience that it is usually not convenient for the wavelength provider to place its WDM equipment in the NREN node and a "very long reach" transmitter is needed to interconnect the NREN node with the WDM equipment in the provider node. Typical fiber lengths were about 40 km. When comparing costs for distances longer than 80 - 100 km, it is necessary to add depreciations of amplifiers, compensators and regenerators (and to consider whether they can be used only for a specific type of data transfer or whether they are multi-purpose). However, prices of such equipment are usually significantly lower than prices of the lease (for example 10 - 30 %). We expect that in the future the price of the wavelength lease will be lower than the price of the fiber lease, because a provider can lease more wavelengths using one fiber. We also expect that the wavelength lease will be preferred especially for long-distance connections (e.g., for fibers longer than 400 km, which usually means an actual distance of terminal points of about 200 - 350 km, considering curves of the path and its terrain profile). However, as WDM is just emerging as a new service, providers now try to include as much of the price of the installed WDM equipment as possible in the price of the wavelength lease. This can ironically make the wavelength lease more expensive than the fiber lease.

The fiber lease is currently used rather rarely in international or federal research and educational networks. The reason is the lack of supply (many telecommunication providers do not offer the fiber lease) and rather long distances between nodes (such as hundreds of kilometers). Leased fibers are more frequently used in regional and national academic networks.

3. CESNET2 Optical Network

CESNET2, a national gigabit optical network for research and education in the Czech Republic is gradually constructed by converting the TEN-155 CZ network, which used 34 Mbps and 155 Mbps SDH lines. These lines were set up by the provider partly using fiber optics owned by power station and distribution companies and partly using two-point microwave connections..

3.1 First Gigabit Optical Circuit

The circuit between Praha and Brno over a 311 km long fiber was launched in January 2000 (topology is illustrated in Figure 1 from November 2000). CISCO 12008 routers with 2.5 Gbps (OC-48) adapters were installed on both ends of the line.

The very long reach optical transmitters used on the adapters had the range of 80 - 100 km. Three opto-electric regenerators CISCO ONS 15104 were also used on this line, located on provider premises along the line. This intercity gigabit line was one of the first in Europe to be entirely used for production IP traffic. An important decision factor was that the price of the fiber lease including the cost of three regenerators was several times lower than the price of the 155 Mbps service between the same cities (depreciations for the regenerators were approximately CZK 100,000 per month in a four-year depreciation system, 1 EUR is about 33 CZK).

Note: Indicated prices were obtained as results of tenders during the construction. They may be specific for the Czech Republic and contain discounts provided by suppliers exclusively for the CESNET2 (to support the research and education infrastructure of Czech Republic).

Fig.1: TEN-155 CZ Topology (November 2000)

 

3. 2 Building CESNET2 Gigabit Core

A tender was announced in the second half of the year 2000 for other gigabit lines of the CESNET2 network. Candidates could offer a fiber lease, a wavelength lease or an SDH service. There was a relatively small number of offers for individual lines (one to three), and no candidate offered all lines in the required time. Differences of offered prices were about 400%. We have chosen 7 lines from 6 different suppliers. This solution was very demanding in terms of contract preparation, negotiation and testing of interconnections between supplier and NREN equipment (particularly conditions for regenerator housing were an issue). However, there was no alternative solution. The positive side was that we acquired experience in dealing with 6 partners, and we succeeded in all cases. The biggest problem was a delay (up to 6 weeks) in the implementation of some lines that was caused either by difficulties in interconnecting devices from different producers or by inaccurate information about lengths and parameters of fibers before their actual connection and measurement. No operational problems worth noting have occurred.

All circuits were put in service in the first half of the year 2001 and are listed in Table 1, including the existing line (Praha - Brno).

Table 1: Current Gigabit Lines of CESNET2 Network (November 2001)
Line Road Length (km) Fibre Length (km) Service Type Line Type Regeneration Used from
Praha-Brno202323,3fibre2,5G3pc ONS1510410.1.2000
Praha-H.Králové90 STM-162,5Gno5.3.2001
Praha-Č.Budějovice139 wavelength2,5Gno22.2.2001
Praha-Plzeň80 wavelength2,5Gno22.2.2001
Praha-Ostrava386 STM-162,5Gno12.2.2001
Praha-Liberec108160,0fibre2,5G2pc ONS1510415.5.2001
Praha-Pardubice114188,6fibre2,5G1pc ONS151041.3.2001
Brno-Olomouc81124,3fibre2,5G2pc ONS1510424.5.2001

All nodes listed in the table are equipped with CISCO 12016 routers with OC-48 adapters in a very long reach version and ONS 15104 regenerators use the same optical transmitters and receivers as these adapters. The number of installed regenerators depends on possibilities for their placing along the line. For example, the line between Praha and Liberec is divided into three sections - 40 km, 80 km and 40 km - and it is not possible to place a regenerator in the 80 km section (fiber-optic is conducted in a ground cable on the top of high-voltage poles). In this particular case, the regenerators were delivered by the supplier of the line. In other cases, the regenerators belong to the NREN. The same service mode has been negotiated with all suppliers irrespective to these differences in ownership - the local assistance for configuring regenerators or solving their malfunctions is provided by the supplier of the line. The NREN can configure the regenerators remotely. However, no malfunction of regenerators has occurred so far. The supervision of fibers and their potential repairs are also performed by the supplier (one cable was stolen so far, which caused a line failure for approximately two days).

Fig.2: CESNET2 Topology (September 2000)

 

3.3 Long Lines Using Gigabit Ethernet

An alternative (and less expensive) solution for establishing gigabit lines over long distances is to use Gigabit or 10-Gigabit Ethernet, besides the STM-16 (2.5 Gbps) and STM-64 (10 Gbps) circuits. However, no regenerators or optical amplifiers are available today for this solution when using CISCO equipment (ONS 15104 is a single-purpose device for STM-16). We are currently preparing a pilot installation of the following equipment for the Gigabit Ethernet line between Olomouc and Ostrava, which is about 150 km long:

  1. CISCO 3512 switches with GBIC ZX converters providing signal regeneration on the path
  2. Optical amplifiers for increasing the reach to 150 - 250 km

We are also gathering and testing other devices for this purpose and we would like to cooperate with other producers, as well (see Table 2).

Table 2: Expected Gigabit Lines of CESNET2 NEtwork (February 2002)
Line Road Length (km) Fibre Length (km) Service Type Line Type Regeneration Used from
Praha-Brno202323,3fibre2,5G3pc ONS1510410.1.2000
Praha-Pardubice114188,6fibre2,5G1pc ONS151041.3.2001
Praha-Č.Budějovice139 wavelength2,5Gno22.2.2001
Praha-Liberec108 wavelength2,5G2pc ONS1510415.5.2001
Pardubice-H.Králové2230,0fibre1GEno1.1.2002
Ostrava-Olomouc105146,3fibre1GE1pc C35121.1.2002
Praha-Plzeň80176,7fibre1GEprepared1.2.2002
Plzeň-Č.Budějovice139178,0fibre1GEprepared15.1.2002
Praha-Ústí n.L.92156,2fibre1GEprepared1.2.2002
Liberec-H.Králové96 wavelength2,5Gno21.1.2002
Ostrava-H.Králové247 wavelength2,5Gno6.2.2002
Č.Budějovice-Brno226 wavelength2,5Gno28.2.2002

3.4 Network Topology

Base on the acquired experience, a tender was announced in the second half of the year 2001 for additional circuits with the following objectives:

The tender was for the fiber or wavelength lease only.

Because of our experience from the previous year, we requested preliminary offers to find out which lines could be provided by individual suppliers for comparatively low prices. This depends on the topology of their fiber-optic infrastructures, which is not fully known. The tender was then announced after gathering the appropriate data.

An interesting point is the overall price level of a fiber lease or a wavelength lease for intercity lines for research and educational purposes. In the Czech Republic in the year 2001, these lines can be leased for CZK 1.15 to 2.40 for one meter/pair/month. However, there are also much more expensive offers. If a separated urban section is leased as the last mile, the price ranges from CZK 3 to 6 per meter/pair/month and, again, there are more expensive offers.

On the whole, we expect that we will not increase the number of STM-16 devices and we will not need to buy expensive routers and adapters for STM-64. Rather, we suppose that the new lines will use Gigabit Ethernet and the primary backbone lines will be converted to 10-Gigabit Ethernet.

The current topology (September 2001) is illustrated in Figure 2 and the expected topology is illustrated in Figure 3 (February 2002) and Figure 4 (February 2003).

Fig.3: Expected CESNET2 Topology (February 2002)

 

3.5 Hardware cost savings

It was presented [2], that in "all IP" networks is the significant reduction in the capital and operating costs as compared to a Packet over SONET (POS) network or even an IP/ATM/SONET network. Estimates of the capital cost savings range from 50% up to as much as 90% relative to a traditional layered Internet. Moreover, through the elimination of the SONET and ATM layers, the operating cost of an Optical Internet can be reduced up to 60%. Of course, savings of this magnitude are generally only possible in a new network deployment. In such a configuration, the high performance network router replaces traditional ATM and SONET/SDH switching and multiplexing equipment for the multiplexing and forwarding of data streams.

Capital cost savings are achieved in the CESNET2 network by gradually deploying Gigabit Ethernet instead of STM-16 over optical fibre (or 10-Gigabit Ethernet instead of STM-64 in the future). Optical fibres are directly connected to Cisco GSR 12016 routers. Costs depends on price of GSR interface cards mainly:

0.9 mil CZK for GE port instead of 2.9 mil. CZK for STM-16 port,
0.5 mil. CZK for GE regeneration instead of 1,0 mil. CZK for STM-16 regeneration.

Capital savings are around 66 % for one fibre line with one regeneration point (approximately 160 km long line). Moreover, for new GigaPoPs we use cheaper gigabit routers instead of GSR. These cost savings are result of line management simplicity mostly. We note, that network topology and Multi Protocol Label Switching (MPLS) help us to solve line fault problem and that before Gigabit Ethernet lines saturation we will be able to use 10-Gigabit Ethernet lines.

3.6 Last Optical Mile Problem

Obtained offers for gigabit circuits confirm that:

The lack of fibers in cities causes problems also when connecting members to the CESNET2 network and when interconnecting premises of members (especially universities).

Basically, there are two alternative solutions of this problem:

  1. supplier of an intercity line pays for the installation of the necessary last optical mile
  2. member of the CESNET2 network or the CESNET association pays for the optical connection of its nodes.

The alternative A is convenient for the supplier in locations with a sufficient demand for telecommunication services so that it is likely that fibers in a newly installed cable (e.g., 48 pairs) will be easily exploited.

The alternative B may be necessary in peripheral locations, where telecommunication providers do not want to invest into a new cable (because of the limited payoff). Usually, more subjects interested in high-quality (that is optical) Internet access build their own "local optical network of fiber owners" in a joint-effort. This network is connected to the "city interconnection point".

These interconnection points should be preferably owned or operated by impartial partners, who do not provide their own telecommunication services. This should ensure that the connected telecommunication services providers and users do not need to worry that the owner of the interconnection point might use its position to hamper competing providers.

Participation of public, municipal or national institutions seems to be promising in the alternative B. The city is usually interested in interconnecting schools, medical facilities, municipal institutions, companies owned by the city, etc., in a less expensive and higher-quality way and in a shorter time than what can be offered by telecommunication operators.

The progress in this area in Czech Republic has not yet reached the stage, in which the aforementioned alternative B, in which cities participate, could be used as the solution of the last optical mile of the CESNET2 network (only the cities of Plzeň and Ostrava might represent an exception). To conclude, if there are no fiber optics already available in or very close to a given node, it takes quite a long time to install them (about 3 months in the air, and 6 - 12 months underground). We are now checking what additional circuits for the CESNET2 network can be obtained as a fiber lease or a wavelength lease in approximately one year.

Fig.4: Expected CESNET2 Topology (February 2003)

 

4. Acknowledgments

The authors are particularly grateful to Jan Gruntorád, Václav Novák, Jan Radil, Martin Míchal, Václav Fanta, Pavel Vachek, Sven Ubik and other colleagues from CESNET Association for their collaboration.

5. Referencess

[1] TERENA Compendium of National Research and Educational Networks in Europe, www.terena.nl/compendium/.
[2] Bjerring, A. K., Arnaud, B. S.: Optical Internets and their Role in Future Telecommunications Systems.

 

VITAE

Lada Altmanová is a Research Engineer at CESNET Asociation. She is particularly involved in Research and Development concerning economical aspects of optical network implementation and leased fibre and wavelenght tenders including optical last mile.

Stanislav Šíma is Head of Research and Development department at CESNET Asociation. He is particularly involved in Research and Development concerning technical and economical aspects of optical network development including optical last mile and hardware, leased fibre and wavelenght tenders.