Prepared by UK-HEP-GRID-NETWORK group

P.Clarke UCL

R.Cranfield UCL

G.Fayers ICSTM

J.Gordon CLRC-RAL

R.Hughes-Jones U.Manchester

P.Jeffreys CLRC-RAL

P.Kummer CLRC-Daresbury

J.MacAllister U.Oxford

D.Salmon CLRC-RAL

A.Sansum CLRC-RAL

R.Tasker CLRC-Daresbury

M.Parsons EPSS

M.Westhead EPSS

S.Saliah Brunel

The purpose of this document is to identify and detail a programme of GRID related networking workpackages which we believe should be undertaken as part of the UK initiative to develop a GRID capability in the UK.

The document includes all areas of investigation which we believe are necessary for at least the first stage of R&D. An attempt is made to quantify the staff-year resources necessarry to carry out this programme.

It is important to note that this has been prepared without any constraint coming from the known available resources, i.e. it is intended to be complete rather than fitted to a limited number of staff-years. Therefore it follows that at this stage there is no implication that the persons involved in the preparation of this document have the necessary resources to carry out all of this programme. This issue is addressed separately.

There are several clear and distinct GRID networking issues:

1. Sustained high rate, robust bulk data transfer
2. Use of differentiated services for time critical requirements
3. Monitoring and Resource prediction metrics for use by matchmaking service
4. Data flow Modelling

These items form the basis of workpackages described below.

GRID middleware needs to intelligently distribute data across several sites in order to be in a position to efficiently match user requests with resources. This requires reliable transfer of data entities. In the early stages for pragmatic reasons this is likely to be limited to file oriented transfer (as opposed to object oriented transfer) but in either case needs to reliably move/copy files across the network.

Generally this is done in anticipation of user requests, thus is not time critical on the scale of minutes or hours.

Typical raw/ESD file sizes will be 1 – 10 TBytes to start with, going up to 100s of TBytes later. To set the scale to transfer a 1 TByte file "overnight" (i.e. in 10 hours maximum) requires a sustained rate of 300 Mbits/sec over the WAN. To put this in context the ac.uk WAN provision will be 2.4 Gbits/s in 2001 rising to 10 Gbits/s in a few years, thus the GRID requirements represent a substantial fraction of this.

Issues:

1. Knowledge of availbility of bandwidth on bulk network provision. This is a monitoring and prediction issue. We need to use/adapt monitoring tools to predict average sustained capacity.
2. Robusness of protocols for long integral transfers. We need to demonstrate that large data sets can be reliably transferred, that protocols are resilient against interruption, congestion and failure. This requires phased testing of large transfers using dedicated PVCs where the background conditions can be controlled.
3. Currently we have little experience of sustained sourcing, transfer and sinking of data at very high rates (> 600 Mbits/s). It is essential to gain experience in this core requirement of GRID operation. Partnerships with industry will be the only way to access the required bandwidth.

Accordingly we propose the following workpackages:

For relevant information obtained so far see appendix.

Requirements for QoS

1. Sustained large scale bulk file transfers are a requirement of (HEP) GRID activities. The evidence availaable suggests that such transfers do not consistently run to completion
2. In a "best effort" environment sustained large scale bulk transfers will constantly compete with other traffic exhibiting different traffic profiles/characteristics. Depending on total load this could/will result in the non-optimal operation of these other protocols and their applications. Within the network, packets from all protocols are currently dealt with identically.
3. During normal GRID operations it will from time to time be essential for a task to obtain small amounts of data from a remote site or to cause procesing to be initiated remotely and obtain results. This may happpen, for instance, if a particular entity does not exist at the point of request. Such demands have a time-bound constraint and must be serviced at high priority.

Existing Capabilities

The mechanisms to meet the above requirements are not currently in place within the network .

1. Basic network resiliance is currently not sufficient to meet the demands of sustained data transfer.
2. Recent experience with CAR+WRED within JANET suggest even that the basic congestion avoidance strategies and tools are at an early developmental stage of deployment.
3. Differentiated networking services (diffserv) are widely seen as the way forward to provide a mechanism to deal appropriately with data flows which have differing time-bound constraints. Diffserv is expected to scale across the Internet but in the first instance will appear as single provider solutions and later through bilateral agreements as multi-provider solutions. In the long term the bandwidth broker will provide such services and as such has been identified as a critical element within Internet2.

Issues and Futures

1. Either the reliability within the network must be improved, or applications are developed that are resilient to such service breaks. The latter seems the most likely as the network is typically a complex association of service providers.
2. A test bed for traffic management technique testing needs to be established in association with UKERNA and the router vendors. [This is a problematic area as most large service providers run their networks at 50 - 70% utilisation and have no such issue with congestion etc, further they view traffic management techniques as additional, and largely unwelcomed, complexity. Only by convincing these providers that the GRID will introduce such bandwidth demands will traffic management be raised on their development agenda]
3. Current deployment of diffserv is limited and services making use of diffserv are not yet available. Proprietary implementations exist which operate only on a limited set of network hardware. Coordination/collaboration with Internet2 is vital to ensure relevant experiences and expertise can be intoduced with SuperJANET4.
4. The development of the bandwidth broker is crucial for the future - more complex - deployment of diffserv.
5. GRID Matchmaking services need to be able to match the priority requirements of applications to network capability, for example diffserv utilisation based upon the prevailing state of the network. In the general sense the need to determine specific patterns of GRID QoS requirments exists as this will allow engineered solutions to be developed in the network.

Many tools already exist worldwide for the purposes of monitoring the state of the international networks. It is evident that the successful long term operation of a GRID will rely crucially upon maintaining and evolving these services.

On a long timescale it is necessary to identify long standing bottlenecks and routing,peering problems, and where necessary lobby or take direct action to rectify the situation. This can only be done based upon a detailed knowledge of the history of performance indicators

On a much shorter timescale, the middleware needs to be aware of available resources on the scale of hours in order to decide how best to service user requests. This requires the development of metrics to predict prevailing bandwidth conditions.

It is envisaged that data flow modelling (computer modelling) may be a very important tool relevant to:

1. Specification of GRID components
2. Optimisation of static characteristics: Network Topology choice, resource location
3. Optimisation of dynamic characteristics: eg: replication and cacheing strategy
4. Planning for evolution

Much work in this area has already taken place within the Monarch project, and UK groups have direct experience of more detailed modelling of LHC experiment data flow.

The initial requirement will be to determine the applicability of such modelling to the GRID problem. This should be done in parallel with work to appraise the use of Monarch tools. The ensuing work will then require the implementaion and evolution of a suitable model to provide input to GRID design.