Solver Applications in Optimization and Network Modeling

The provided code constitutes a comprehensive framework for research and development in the fields of artificial intelligence and computational biology. Its architecture consists of specialized modules addressing key aspects of modeling, machine learning, and optimization, with an emphasis on code robustness and reusability.

From a technical perspective, the code includes:

Training Data Generation: Functionality for generating synthetic data and manipulating real datasets, with advanced options for controlled noise introduction and simulation of specific scenarios.

Machine Learning Model Development: Tools for constructing and customizing machine learning models, including deep neural network architectures and supervised and unsupervised learning algorithms.

Performance Evaluation: Capabilities for systematically evaluating model performance under various conditions, using statistical metrics and visualizations to analyze prediction quality and generalization ability.

Optimization Solver Comparison: Functionality for comparing different optimization methods used in solving specific problems, with tools for measuring convergence, computational efficiency, and scalability.

The underlying technical approach is based on the efficient implementation of algorithms and data structures, leveraging high-performance software libraries, and adopting software engineering practices such as modularity, encapsulation, and detailed documentation.

class optimization.model_real_data_.CustomLossFuntion(*args: Any, **kwargs: Any)

Bases: Loss

A custom loss function that computes a loss based on specific criteria and includes a penalty term for certain conditions.

Kt

A coefficient used in the loss computation.

Type:

float

i

Index used in gathering elements from predictions.

Type:

int

j

Another index used in gathering elements from predictions.

Type:

int

Bc

A boundary condition used in the penalty term.

Type:

float

ic

Index for penalty computation.

Type:

int

jc

Another index for penalty computation.

Type:

int

find

A fixed index to slice the predictions for certain operations.

Type:

int, optional

__init__(Kt, i, j, Bc, ic, jc, find=6)

Initializes the object with given parameters.

Parameters:

• Kt (float) – Constant value.

• i (int) – Integer value.

• j (int) – Integer value.

• Bc (float) – Boundary condition value.

• ic (int) – Integer value.

• jc (int) – Integer value.

• find (int, optional) – An integer defining something, by default 6.

Return type:

None

Notes

This function initializes the object with the provided parameters and sets up initial values for some internal variables such as bounds.

call(y_true, y_pred)

Computes the custom loss between true labels and predicted labels.

Parameters:

• y_true (Tensor) – True labels.

• y_pred (Tensor) – Predicted labels.

Returns:

Computed loss value.

Return type:

Tensor

get_config()

Returns the configuration of the custom loss function.

Returns:

Configuration dictionary.

Return type:

dict

class optimization.model_real_data_.CustomSigmoidActivation(lower_bound: float, upper_bound: float)

Bases: object

__call__(x: tensorflow.Tensor) → tensorflow.Tensor

Apply the custom sigmoid activation function to the input.

Parameters:

x (tf.Tensor) – The input tensor to apply the activation function.

Returns:

The output tensor after applying the activation function.

Return type:

tf.Tensor

__init__(lower_bound: float, upper_bound: float)

Initializes the CustomSigmoidActivation with specified bounds.

Parameters:

• lower_bound (float) – The lower limit for the activation function.

• upper_bound (float) – The upper limit for the activation function.

get_config() → dict

Returns the configuration of the CustomSigmoidActivation function.

Returns:

A dictionary containing the configuration: lower and upper bounds.

Return type:

dict

class optimization.model_real_data_.LearningRateReducer(*args: Any, **kwargs: Any)

Bases: Callback

A custom callback to reduce the learning rate based on epoch thresholds.

epoch_threshold

Epoch numbers at which the learning rate should be reduced.

Type:

list of int

on_epoch_begin(epoch, logs=None):

Reduces the learning rate if the current epoch is in the epoch_threshold.

__init__(epoch_threshold)

Initializes the LRReducer callback with specified epoch thresholds.

Parameters:

epoch_threshold (list of int) – Epoch numbers at which the learning rate should be reduced.

on_epoch_begin(epoch, logs=None)

Called at the beginning of an epoch during training.

Parameters:

• epoch (int) – The current epoch number.

• logs (dict, optional) – Currently no logs are used in this method.

class optimization.model_real_data_.RestrictedDense(*args: Any, **kwargs: Any)

Bases: Layer

A custom dense layer with constraints on its output.

initializer

The initializer for the kernel weights of the layer.

Type:

Initializer

well

A threshold value used in the layer’s computations.

Type:

tf.Variable

units

The number of units in the dense layer.

Type:

int

dense

The underlying dense layer.

Type:

tf.keras.layers.Dense

call(inputs):

Computes the layer’s output with the given inputs.

get_config():

Returns the layer’s configuration.

__init__(initializer, well, units, **kwargs)

Initializes the RestrictedDense layer.

Parameters:

• initializer (Initializer) – The initializer for the kernel weights of the layer.

• well (float) – A threshold value used in the layer’s computations.

• units (int) – The number of units in the dense layer.

call(inputs)

Computes the output of the layer with given inputs.

Parameters:

inputs (tuple of Tensors) – Inputs to the layer: (x, In, fw).

Returns:

The output tensor of the layer.

Return type:

Tensor

get_config()

Returns the configuration of the RestrictedDense layer.

Returns:

A dictionary containing the configuration.

Return type:

dict

class optimization.model_real_data_.TimeHistoryTest(*args: Any, **kwargs: Any)

Bases: Callback

A custom callback to track the time taken for predictions during model inference.

prediction_times

A list to store the time taken for each prediction batch.

Type:

list

on_predict_begin(self, logs=None):

Called at the beginning of prediction.

on_predict_batch_begin(self, batch, logs=None):

Called at the beginning of a batch during prediction.

on_predict_batch_end(self, batch, logs=None):

Called at the end of a batch during prediction.

on_predict_batch_begin(batch, logs=None)

Records the start time of a batch prediction.

Parameters:

• batch (int) – Index of the batch.

• logs (dict, optional) – Additional information about the batch, by default None.

on_predict_batch_end(batch, logs=None)

Calculate and record the time taken for a batch prediction.

Parameters:

• batch (int) – Index of the batch.

• logs (dict, optional) – Additional information about the batch, by default None.

on_predict_begin(logs=None)

Initializes the prediction_times list at the beginning of prediction.

Parameters:

logs (dict, optional) – Dictionary containing the logs for the current prediction, by default None.

class optimization.model_real_data_.WellDense(*args: Any, **kwargs: Any)

Bases: Layer

A custom Keras layer that applies a threshold to the sum of inputs.

well

The threshold value for the sum of inputs.

Type:

tf.Variable

call(inputs):

Computes the sum of inputs and applies the threshold.

get_config():

Returns the layer’s configuration.

__init__(well=80.0, **kwargs)

Initializes the WellDense layer with a specified threshold.

Parameters:

well (float, optional) – The threshold value for the sum of inputs.

call(inputs)

Computes the sum of inputs and applies the threshold.

Parameters:

inputs (Tensor) – The input tensor to the layer.

Returns:

The output tensor after applying the threshold.

Return type:

Tensor

get_config()

Returns the configuration of the WellDense layer.

Returns:

A dictionary containing the configuration: well.

Return type:

dict

optimization.model_real_data_.bounded(s1='/content/drive/Shareddrives/red_gas_col/', s2='/content/drive/Shareddrives/red_gas_col/Prueba/Data/', s=1)

Generates box plots to visualize and compare performance metric differences across various models.

This function loads performance data of different models from the Weymouth.xlsx, Balance.xlsx, and Costos.xlsx files. Then, box plots are generated for each of these metrics, showing how the performance values are distributed among the different models.

The plots display the Mean Absolute Percentage Error (MAPE) and costs, facilitating visual comparison between the models. A logarithmic scale is used to ease the visualization of data that may have a wide range of values.

Parameters:

• s1 (str, optional) – Path to the directory containing input data files. Default is ‘/content/drive/Shareddrives/red_gas_col/’.

• s2 (str, optional) – Path to the directory containing model data files. Default is ‘/content/drive/Shareddrives/red_gas_col/Prueba/Data/’.

• s (int, optional) – Determines the type of model. If 0, uses one set of models, if 1, uses another set. Default is 1.

Returns:

The results are directly displayed as plots.

Return type:

None

optimization.model_real_data_.create_custom_pres_layer(input_tensor, lower_bound, upper_bound, layer_name, initializer)

Creates a custom dense layer with a CustomSigmoidActivation activation function.

Parameters:

• input_tensor (Tensor) – The input tensor for the dense layer.

• lower_bound (float) – The lower bound for the CustomSigmoidActivation function.

• upper_bound (float) – The upper bound for the CustomSigmoidActivation function.

• layer_name (str) – The name of the dense layer.

• initializer (Initializer) – The initializer for the kernel weights of the dense layer.

Returns:

The output tensor from the dense layer.

Return type:

Tensor

optimization.model_real_data_.create_customized_dense_layer(input_tensor, layer_info, layer_name, activation_function, initializer)

Creates a custom dense layer using the Keras Dense function.

Parameters:

• input_tensor (Tensor) – The input tensor for the dense layer.

• layer_info (list) – A list containing specific information for constructing the dense layer.

• layer_name (str) – The name of the dense layer.

• activation_function (callable) – The activation function to be applied to the output of the dense layer.

• initializer (Initializer) – The initializer for the kernel weights of the dense layer.

Returns:

The output tensor from the dense layer.

Return type:

Tensor

optimization.model_real_data_.dynamic_val(s1='/content/drive/Shareddrives/red_gas_col/', s2='/content/drive/Shareddrives/red_gas_col/Prueba/Data/')

Generates sinusoidal behavior and adjusts random data accordingly.

Parameters:

• s1 (str, optional) – Path for the first directory, by default ‘/content/drive/Shareddrives/red_gas_col/’.

• s2 (str, optional) – Path for the second directory, by default ‘/content/drive/Shareddrives/red_gas_col/Prueba/Data/’.

Returns:

X_test

Return type:

Randomly generated data.

Notes

This function generates sinusoidal behavior and adjusts random data accordingly.

optimization.model_real_data_.evaluate_balance(FT, W, X)

Evaluates the balance of predictions using Mean Absolute Percentage Error (MAPE).

Parameters:

• FT (list) – A list containing two sets of forecasted data.

• W (array-like) – Weight matrix used in the calculation.

• X (array-like) – Actual data for comparison.

Returns:

A list containing two lists of MAPE values for each forecast set in FT.

Return type:

list

Notes

This function computes the Mean Absolute Percentage Error (MAPE) between the actual data (X) and each set of forecasted data in FT, using the provided weight matrix W.

optimization.model_real_data_.evaluate_balance_two(FT, W, X)

Evaluate the Mean Absolute Percentage Error (MAPE) between actual and predicted data.

Parameters:

• FT (list of numpy arrays) – List of predicted flow data from models.

• W (numpy array) – Weight matrix used for the calculation.

• X (list of numpy arrays) – List of actual flow data for comparison.

Returns:

A list containing MAPE values for each corresponding set of predicted and actual data.

Return type:

list

Notes

This function computes the MAPE for each set of predicted data in ‘FT’ against the actual data in ‘X’. The computation involves a dot product of predicted data with the transpose of weight matrix ‘W’. The function iterates over the elements of ‘FT’ and ‘X’ to compute MAPE for each set of data.

optimization.model_real_data_.evaluate_cost(X, Y)

Calculates the difference between predicted costs and actual costs.

Parameters:

• X (list) – A list of actual cost values.

• Y (list) – A list of predicted cost values.

Returns:

A list of differences between predicted and actual costs.

Return type:

list

Notes

This function iterates over each element in X and Y, calculates the difference (Y[i] - X[i]) for each corresponding element, and returns these differences in a list.

optimization.model_real_data_.evaluate_weymouth(X, Y)

Evaluates the Weymouth equation performance by calculating the MAPE (Mean Absolute Percentage Error) between actual values (X) and predicted values (Y).

Parameters:

• X (list of lists) – Actual values for comparison.

• Y (list of lists) – Predicted values to be compared against X.

Returns:

Calculated MAPE values for each set of comparisons.

Return type:

list of lists

Notes

This function calculates the Mean Absolute Percentage Error (MAPE) for two sets of data (indicated by indices 0 and 1) in X and Y.

optimization.model_real_data_.evaluate_weymouth_two(X, Y)

Calculate the Mean Absolute Percentage Error (MAPE) between two sets of flow data.

Parameters:

• X (list of numpy arrays) – The first set of flow data (usually the actual flow data).

• Y (list of numpy arrays) – The second set of flow data (usually the predicted flow data).

Returns:

A list of MAPE values, each corresponding to the MAPE between the elements of X and Y.

Return type:

list

Notes

This function iterates over the corresponding elements of X and Y, calculating the MAPE for each pair of elements. The results are stored and returned in a list.

optimization.model_real_data_.extract_data(times, b)

Extracts a specific subset of data from a multi-dimensional array.

Parameters:

• times (multi-dimensional array) – The source array from which data is extracted.

• b (int) – The specific index in the first dimension to extract data from.

Returns:

A list containing the extracted data.

Return type:

list

Notes

This function iterates over 320 elements in the specified sub-array of ‘times’, extracting a particular value from each element.

optimization.model_real_data_.extract_flows_from_model(model, test_data)

Extracts the output of a specific layer (‘Flujos’) from a given Keras model.

Parameters:

• model (keras.Model) – The Keras model from which to extract the layer output.

• test_data (array-like) – The input data to pass through the model.

Returns:

The output of the ‘Flujos’ layer in the model.

Return type:

numpy.ndarray

optimization.model_real_data_.flow_model(path, fd, seeds=1, s=1)

Builds and compiles a TensorFlow Keras model based on data from an Excel file and additional parameters.

Parameters:

• path (str) – Path to the Excel file containing the model’s data.

• fd (array-like) – Additional flow data for the model.

• seeds (int, optional) – Seed for random number generation for reproducibility, by default 1.

• s (int, optional) – Mode switch for the model configuration, by default 1.

Returns:

The compiled Keras model.

Return type:

tensorflow.keras.Model

Notes

This function constructs and compiles a TensorFlow Keras model based on data extracted from an Excel file specified by path. Additionally, it accepts additional flow data (fd) to incorporate into the model. The seeds parameter is used to set a seed for random number generation, ensuring reproducibility, while s allows for configuration of the model based on different modes.

optimization.model_real_data_.gen_w(Data_inf, Data_W, Data_T, Data_C, Data_Dc, Data_sto)

Generates a matrix combining information from wells, pipes, compressors, users, and storages.

Parameters:

• Data_inf (List) – Information data.

• Data_W (pd.DataFrame) – Data related to wells.

• Data_T (pd.DataFrame) – Data related to pipes.

• Data_C (pd.DataFrame) – Data related to compressors.

• Data_Dc (pd.DataFrame) – Data related to users.

• Data_sto (pd.DataFrame) – Data related to storages.

Returns:

A matrix combining all the information.

Return type:

np.ndarray

Raises:

ValueError – If any input data is not in the expected format or type.

optimization.model_real_data_.identify_atypical_values(files)

Identify and visualize the percentage of atypical values in a list of arrays.

Parameters:

files (list of arrays) – List of arrays. Each array represents a different data set.

Return type:

None

Notes

This function calculates the percentage of values in each array that are greater than 1, indicating atypical values. It then visualizes these percentages using a bar plot.

optimization.model_real_data_.load_data(path, key)

Loads data from a MATLAB file specified by ‘path’ and extracts data based on ‘key’.

Parameters:

• (str) (key) –

• (str) –

Returns:

• list (A list containing the extracted data.)

• The function iterates through eight sets of data within the MATLAB file, extracting

• each set and appending it to a list. The extracted data is expected to be in a

• specific format within the MATLAB file.

optimization.model_real_data_.loss_val()

Visualizes and compares differences in performance metrics of various models using box plots.

Notes

This function loads performance data from different models from the files Weymouth.xlsx, Balance.xlsx, and Costos.xlsx. Then, box plots are generated for each of these metrics, showing how the performance values are distributed among the different models.

The plots display the Mean Absolute Percentage Error (MAPE) and costs, facilitating visual comparison between the models. A logarithmic scale is used to ease the visualization of data that may have a wide range of values.

This function has no input parameters and does not return any value. The results are directly displayed as plots.

optimization.model_real_data_.mish(x)

Mish Activation Function.

Applies the mish function element-wise to the input Tensor.

Parameters:

x (Tensor) – The input tensor.

Returns:

The transformed tensor after applying the mish activation function.

Return type:

Tensor

optimization.model_real_data_.ng_case_evaluate(s1='/content/drive/Shareddrives/red_gas_col/', s2='/content/drive/Shareddrives/red_gas_col/Prueba/Data/', s=1)

Evaluates multiple gas network cases using pre-trained models and generates evaluation metrics.

Parameters:

• s1 (str, optional) – Base directory path where the network case files are stored, by default ‘/content/drive/Shareddrives/red_gas_col/’.

• s2 (str, optional) – Directory path where input and output data files, and model files are stored, by default ‘/content/drive/Shareddrives/red_gas_col/Prueba/Data/’.

• s (int, optional) – Mode indicator (0 or 1) to choose between two sets of model paths, by default 1.

Returns:

A tuple of DataFrames containing evaluation metrics: (Balance, Weymouth, Pj/Pi, Cost).

Return type:

tuple of DataFrames

Notes

This function performs the following steps: - Loads test input and output data. - Based on the mode ‘s’, selects a set of model paths. - Iterates over multiple network cases, loading the necessary data for each case. - For each case, it loads a pre-trained model and performs predictions. - Computes evaluation metrics such as Weymouth, Balance, Cost, and Pressure Ratios (Pj/Pi). - Aggregates the results into DataFrames for each metric. - Finally, calls the plots function to visualize the results.

optimization.model_real_data_.ng_evaluate_atip(s1='/content/drive/Shareddrives/red_gas_col/', s2='/content/drive/Shareddrives/red_gas_col/Prueba/Data/', s=1)

Evaluate different neural network models on various gas network cases and plot their performance metrics.

Parameters:

• s1 (str, optional) – Base directory path for the model and Excel data files, by default ‘/content/drive/Shareddrives/red_gas_col/’.

• s2 (str, optional) – Path for the input data files, by default ‘/content/drive/Shareddrives/red_gas_col/Prueba/Data/’.

• s (int, optional) – Scenario indicator for choosing different sets of model paths, by default 1.

Returns:

A tuple containing lists of evaluation metrics (Balance, Weymouth, PjPi, Costos) for each network case.

Return type:

tuple

Notes

This function processes multiple gas network cases in a loop. For each case, it loads necessary data, constructs and loads a neural network model, and predicts outputs. It then evaluates these predictions using custom functions for different metrics such as Weymouth formula, balance, cost, and pressure ratios. The function visualizes these metrics using boxplot graphs.

optimization.model_real_data_.plot_time(b='fit', s1='/content/drive/Shareddrives/red_gas_col/', s2='/content/drive/Shareddrives/red_gas_col/Prueba/Data/', s=1)

Generates and displays boxplot visualizations for time measurements of different network models.

Parameters:

• b (str, optional) – Indicates the type of operation to perform, ‘fit’ for fitting or ‘predict’ for prediction, by default ‘fit’.

• s1 (str, optional) – Base path for the file location, by default ‘/content/drive/Shareddrives/red_gas_col/’.

• s2 (str, optional) – Secondary path for data and model files, by default ‘/content/drive/Shareddrives/red_gas_col/Prueba/Data/’.

• s (int, optional) – Mode indicator for selecting model paths, by default 1.

Return type:

None

Notes

The function first determines the file paths for the Excel and model files based on the mode ‘s’. It then loads the test data and times data. If ‘b’ is ‘predict’, it iterates over models, performs predictions, measures the time taken for predictions, and stores these times in a DataFrame. This DataFrame is then used to create boxplot visualizations. If ‘b’ is not ‘predict’, it loads time measurements from a CSV file and creates boxplots for these measurements.

The boxplots provide a comparison of time taken by different network models for predictions or fitting.

optimization.model_real_data_.plots(Balance, Wey, re, Costos, s)

Creates a set of boxplot visualizations for various data frames.

This function generates a 2x2 grid of boxplots, each representing data from one of the provided DataFrames. The boxplots for ‘Balance’ and ‘Costos’ are set to a logarithmic scale. The title of the plots changes based on the value of ‘s’.

Parameters:

• Balance (DataFrame) – DataFrame containing balance data.

• Wey (DataFrame) – DataFrame containing Weymouth data.

• re (DataFrame) – DataFrame containing ratio data.

• Costos (DataFrame) – DataFrame containing cost data.

• s (int) – Mode switch for the plot titles.

Return type:

None

Notes

This function generates visualizations for the provided DataFrames in a 2x2 grid of boxplots. The titles of the plots change based on the value of ‘s’. ‘Balance’ and ‘Costos’ DataFrames are displayed in a logarithmic scale.

optimization.model_real_data_.split(feature_data, output_data)

Splits the feature and output data into training and test sets.

Parameters:

• feature_data (array-like) – The input features data.

• output_data (array-like) – The output data corresponding to the input features.

Returns:

A tuple containing split data: (X_train, X_test, y_train, y_test)

Return type:

tuple

optimization.model_real_data_.swish(x)

Swish Activation Function.

Applies the swish function element-wise to the input Tensor.

Parameters:

x (Tensor) – The input tensor.

Returns:

The transformed tensor after applying the swish activation function.

Return type:

Tensor

optimization.model_real_data_.visualize_atipic(dataframe, columns)

Identifies and visualizes the percentage of outlier values in specified columns of a DataFrame.

Parameters:

• dataframe (DataFrame) – The DataFrame in which to identify outliers.

• columns (list) – List of column names in the DataFrame to analyze.

Return type:

None

Notes

This function utilizes the Interquartile Range (IQR) method to identify outliers. A value is considered an outlier if it is below Q1 - 1.5 * IQR or above Q3 + 1.5 * IQR, where Q1 and Q3 are the first and third quartiles, respectively.

The function generates a bar plot displaying the percentage of outliers in each column.

optimization.model_real_data_.visualize_non_convergence(data_path='/content/drive/Shareddrives/red_gas_col/Prueba/Data/inputs_None_', network_range=(8, 17))

Visualizes the percentage of non-convergence cases for different network dimensions.

Parameters:

• data_path (str, optional) – Path to the directory containing input data files, by default ‘/content/drive/Shareddrives/red_gas_col/Prueba/Data/inputs_None_’.

• network_range (tuple, optional) – Range of network files to analyze (inclusive), by default (8, 17).

Return type:

None

Notes

This function loads data from specified files and calculates the percentage of non-convergence cases. It then generates a bar plot to visually represent this data.