add text here
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from tensorflow.keras.layers import Input,Dense, Embedding, Flatten, Subtract, Concatenate
from tensorflow.keras.models import Model
from tensorflow.keras.utils import plot_model
from sklearn.model_selection import train_test_split
REFRESHER OF KERAS MODELS
# Create an input layer of shape 1
input_tensor = Input(shape=(1,))
# Dense layer
output_layer = Dense(1)
# Connect the dense layer to the input_tensor
output_tensor = output_layer(input_tensor)
#above 2 steps can be shortened to output_tensor=Dense(1)(input_tensor)
#build the model
model = Model(input_tensor, output_tensor)
# Compile the model (mean_absolute_error creates something akin to lnr model/alternative is mean_squared_error):m is the weight of dense layer and b is the bias (y=mx+b)
model.compile(optimizer='adam', loss='mean_absolute_error')
# Summarize the model
model.summary()
games_tourney = pd.read_csv('gamestourney.csv')
games_tourney.head()
# Split the dataset into train and test sets
#games_tourney_train, games_tourney_test = train_test_split(games_tourney, test_size=0.2, random_state=42)
# Fit model against the training dataset
#model.fit(games_tourney_train['seed_diff'], games_tourney_train['score_diff'], epochs=1, batch_size=64, validation_split=0.2, verbose=True)
# Load the X variable from the test data
#X_test = games_tourney_test['seed_diff']
# Load the y variable from the test data
#y_test = games_tourney_test['score_diff']
# Evaluate the model on the test data
#print(model.evaluate(X_test, y_test, verbose=False))CHAPTER 2
CATEGORY EMBEDDINGS
Categorical embeddings are an advanced type of layer, only available in deep learning libraries. They are extremely useful for dealing with high cardinality (High cardinality refers to a column that can have many possible values) categorical data. In this dataset, the team ID variable has high cardinality. Embedding layers are also very useful for dealing with text data, such as in Word2vec models, but that is beyond the scope of this course. To model these teams in the basketball data, you'll use a very simple model that learns a "strength" rating for each team and uses those ratings to make predictions. To map the integer team IDs to a decimal rating, we will use an embedding layer. To get started with category embeddings, you will need an input layer. In this case, your input is a single number, ranging from 1 to 10,887, which represents each team's unique ID. Note that this dataset covers about 30 years of data, and has about 400 unique schools, giving us close to 12,000 IDs. We only have about 11,000 of those year/team combinations, because not every school has a basketball team every year. Embedding layers increase the dimensionality of your data. The input CSV has two dimensions (rows and columns), but embedding layers add a third dimension. This third dimension can be useful when dealing with images and text, so it is not as relevant to this course. Therefore, we use the flatten layer to flatten the embeddings from 3D to 2D. The flatten layer is also the output layer for the embedding process. Flatten layers are an advanced layer for deep learning models and can be used to transform data from multiple dimensions back down to two dimensions. They are useful for dealing with time series data, text data, and images. Now you can wrap your embedding layer in a model. This will allow you to reuse the model for multiple inputs in the dataset. You do this by defining an input layer, then an embedding layer, then a flatten layer for the output. Finally, wrap the input tensor and flatten tensor in a model. This model can be treated exactly the same as a layer, and re-used inside of another model.
import pandas as pd
from tensorflow.keras.layers import Embedding
games_tourney = pd.read_csv('gamestourney.csv')
games_season = pd.read_csv('gamesseason.csv')
# Count the unique number of teams
n_teams = games_season['team_1'].nunique()
# Create an embedding layer
team_lookup = Embedding(input_dim=n_teams,
output_dim=1,
input_length=1,
name='Team-Strength')
# Create an input layer for the team ID
teamid_in = Input(shape=(1,))
# Lookup the input in the team strength embedding layer
strength_lookup = team_lookup(teamid_in)
# Flatten the output
strength_lookup_flat = Flatten()(strength_lookup)
# Combine the operations into a single, re-usable model
team_strength_model = Model(teamid_in, strength_lookup_flat, name='Team-Strength-Model')Shared Layers
Shared layers are an advanced deep learning concept, and are only possible with the Keras functional API. They allow you to define an operation and then apply the exact same operation (with the exact same weights) on different inputs. In this model, we will share team rating for both inputs. The learned rating will be the same, whether it applies to team 1 or team 2. To create a shared layer, you must first create two (or more) inputs, each of which will be passed to the shared layer. In this case, you will use two inputs. Once you have two inputs, the magic of the Keras functional API becomes apparent. Recall from chapter 1 that the Dense() function returns a function as its output. This function, which Dense() outputs, takes a tensor as input and produces a tensor as output. You can use the same Dense() function to create a shared layer! Doing so is as simple as calling the function twice, with a different input tensor each time. Recall the category embedding model we made in the previous lesson. This model first embeds an input and then flattens it. You can also share models, not just layers. This is really cool and is part of what makes the functional API so useful. You can define modular components of models and then reuse them. We define an embedding layer and wrap it in a model. We then define 2 input tensors, and pass each one to the same model, producing 2 output tensors. This will use the same model, with the same layers and the same weights, for mapping each input to its corresponding output. In other words, you can take an arbitrary sequence of keras layers, and wrap them up in a model. Once you have a model, you can re-use that model to share that sequence of steps for different input layers.
# Input layer for team 1
team_in_1 = Input(shape=(1,), name="Team-1-In" )
# Separate input layer for team 2
team_in_2 = Input(shape=(1,), name="Team-2-In")
# Lookup team 1 in the team strength model
team_1_strength = team_strength_model(team_in_1)
# Lookup team 2 in the team strength model
team_2_strength = team_strength_model(team_in_2)MERGE LAYER
There are many kinds of merge layers available in Keras. Add, Subtract, and Multiply layers do simple arithmetic operations by element on the input layers, and require them to be the same shape. For example, if we wanted to multiply our team strength ratings together, we could use a Multiply layer. Concatenate layers simply append the 2 layers together, similar to the hstack() function from numpy. Unlike the other merge layers, the Concatenate layer can operate on layers with different numbers of columns. Let's build a simple Keras model that takes in two numbers and adds them together. You accomplish this by defining two input layers and using the Add layer to add them together. If you'd like to add together many inputs, you can pass a list with more than two elements to an Add layer. Note that all of the inputs are required to have the same shape, so they can be combined element-wise. The Subtract and Multiply layers work the same way. Now you can wrap the output from your Add layer inside a Model, which will then allow you to fit it to data. Note that the model takes in a list of inputs because it has more than one input.
# Create a subtract layer using the inputs from the previous exercise
score_diff = Subtract()([team_1_strength, team_2_strength])
# Create the model
model = Model([team_in_1, team_in_2], score_diff)
# Compile the model
model.compile(optimizer='adam', loss='mean_absolute_error')PREDICT THE MODEL
To fit a model with multiple inputs, provide the model a list of inputs. In this case, since you have two inputs, the model needs to have an input list of length 2. You want to use this model to predict a single target, so the target for training is still a single object. While this network is very simple, the concept it illustrates is quite advanced. Later in the course, you will process different inputs to the network in different ways. In other words, multiple inputs let you do data pre-processing as part of the model you learn! To evaluate a model with multiple inputs, simply give it a list of inputs, along with a single output, and the model will return its loss on the new data.
# Get the team_1 column from the regular season data
input_1 = games_season['team_1']
# Get the team_2 column from the regular season data
input_2 = games_season['team_2']
# Fit the model to input 1 and 2, using score diff as a target
model.fit([input_1,input_2],
games_season['score_diff'],
epochs=1,
batch_size=2048,
validation_split=0.1,
verbose=True)
#evaluate the season data vs the tournament data to see how well you can predict the tournament games , based on a model trained with regular season data.
# Get team_1 from the tournament data
input_1_t = games_tourney['team_1']
# Get team_2 from the tournament data
input_2_t = games_tourney['team_2']
# Evaluate the model using these inputs
print(model.evaluate([input_1_t, input_2_t], games_tourney['score_diff'], verbose=False))CHAPTER 3
3 inputs and beyond .summary() table shows parameters and trainable parameters parameters generally = number of inputs plus a bias or intercept more trainable parameters are more flexible but also more prone to overfitting less trainable paremeters are less flexible therefore less prone to overfitting Important to remember embedding layers often add a very large number of trainable paremeters to a model--- remember...embedding layers map integers to floats: each unique value of the embedding input gets a parameter for its output