Jul 16 20:32:41.859449 ip-10-0-136-249 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 20:32:42.268467 ip-10-0-136-249 kubenswrapper[2530]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 20:32:42.268467 ip-10-0-136-249 kubenswrapper[2530]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 20:32:42.268467 ip-10-0-136-249 kubenswrapper[2530]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 20:32:42.268467 ip-10-0-136-249 kubenswrapper[2530]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 20:32:42.268467 ip-10-0-136-249 kubenswrapper[2530]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 20:32:42.269176 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.268527    2530 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 20:32:42.274053 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274027    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 20:32:42.274053 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274051    2530 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 20:32:42.274053 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274056    2530 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274059    2530 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274062    2530 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274065    2530 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274068    2530 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274071    2530 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274074    2530 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274077    2530 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274080    2530 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274083    2530 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274086    2530 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274088    2530 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274112    2530 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274116    2530 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274119    2530 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274122    2530 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274124    2530 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274127    2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274129    2530 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274132    2530 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 20:32:42.274181 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274134    2530 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274137    2530 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274139    2530 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274142    2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274144    2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274147    2530 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274150    2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274152    2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274155    2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274157    2530 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274160    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274163    2530 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274166    2530 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274169    2530 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274173    2530 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274176    2530 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274179    2530 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274182    2530 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274185    2530 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274187    2530 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 20:32:42.274656 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274190    2530 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274193    2530 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274195    2530 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274198    2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274201    2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274203    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274205    2530 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274208    2530 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274210    2530 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274213    2530 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274215    2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274218    2530 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274220    2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274223    2530 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274225    2530 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274228    2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274230    2530 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274232    2530 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274235    2530 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274237    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 20:32:42.275170 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274239    2530 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274243    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274245    2530 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274248    2530 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274250    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274253    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274256    2530 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274259    2530 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274264    2530 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274269    2530 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274272    2530 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274275    2530 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274277    2530 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274280    2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274282    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274285    2530 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274287    2530 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274290    2530 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274292    2530 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 20:32:42.275733 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274295    2530 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274297    2530 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274300    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274302    2530 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274304    2530 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274308    2530 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274312    2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274315    2530 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274318    2530 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274320    2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274323    2530 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274325    2530 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274328    2530 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274331    2530 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274333    2530 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274337    2530 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274340    2530 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.274343    2530 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275078    2530 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 20:32:42.276223 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275086    2530 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275089    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275092    2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275095    2530 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275127    2530 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275130    2530 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275133    2530 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275135    2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275138    2530 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275141    2530 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275144    2530 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275146    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275149    2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275151    2530 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275154    2530 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275158    2530 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275161    2530 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275164    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275166    2530 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275169    2530 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 20:32:42.276684 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275172    2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275174    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275177    2530 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275180    2530 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275183    2530 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275186    2530 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275188    2530 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275191    2530 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275193    2530 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275196    2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275200    2530 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275204    2530 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275207    2530 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275211    2530 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275213    2530 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275217    2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275220    2530 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275223    2530 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275226    2530 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275228    2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 20:32:42.277201 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275231    2530 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275233    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275236    2530 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275238    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275240    2530 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275243    2530 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275245    2530 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275248    2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275251    2530 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275253    2530 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275256    2530 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275258    2530 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275261    2530 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275263    2530 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275266    2530 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275268    2530 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275271    2530 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275274    2530 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275276    2530 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275280    2530 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 20:32:42.277696 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275283    2530 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275286    2530 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275288    2530 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275291    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275294    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275297    2530 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275299    2530 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275302    2530 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275304    2530 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275307    2530 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275309    2530 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275312    2530 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275314    2530 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275317    2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275320    2530 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275323    2530 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275325    2530 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275328    2530 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275332    2530 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 20:32:42.278217 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275335    2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275338    2530 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275341    2530 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275344    2530 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275346    2530 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275349    2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275352    2530 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275354    2530 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275357    2530 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275360    2530 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275362    2530 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275365    2530 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275368    2530 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275371    2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275373    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275376    2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275378    2530 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275381    2530 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275384    2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 20:32:42.278669 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275800    2530 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275805    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275808    2530 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275811    2530 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275814    2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275817    2530 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275820    2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275823    2530 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275825    2530 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275828    2530 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275830    2530 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275833    2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275835    2530 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275838    2530 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275841    2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275843    2530 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275846    2530 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275848    2530 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275851    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275853    2530 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 20:32:42.279156 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275856    2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275858    2530 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275861    2530 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275864    2530 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275866    2530 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275868    2530 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275871    2530 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275874    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275876    2530 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275879    2530 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275882    2530 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275885    2530 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275887    2530 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275890    2530 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275893    2530 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275895    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275898    2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275901    2530 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275903    2530 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275907    2530 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 20:32:42.279654 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275909    2530 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275912    2530 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275914    2530 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275916    2530 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275919    2530 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275921    2530 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275924    2530 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275926    2530 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275929    2530 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275931    2530 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275935    2530 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275938    2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275941    2530 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275944    2530 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275950    2530 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275953    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275956    2530 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275958    2530 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275961    2530 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275963    2530 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 20:32:42.280165 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275966    2530 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275969    2530 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275971    2530 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275974    2530 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275977    2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275979    2530 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275981    2530 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275985    2530 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275989    2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275992    2530 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275995    2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.275998    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276000    2530 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276003    2530 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276005    2530 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276008    2530 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276010    2530 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276012    2530 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276015    2530 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 20:32:42.280665 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276019    2530 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276022    2530 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276025    2530 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276028    2530 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276031    2530 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276033    2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276036    2530 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276040    2530 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276043    2530 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276046    2530 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276048    2530 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276052    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276055    2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276057    2530 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276060    2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276062    2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276065    2530 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276067    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276070    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 20:32:42.281178 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276072    2530 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276167    2530 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276176    2530 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276183    2530 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276188    2530 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276193    2530 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276196    2530 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276201    2530 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276205    2530 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276209    2530 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276212    2530 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276215    2530 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276218    2530 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276221    2530 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276224    2530 flags.go:64] FLAG: --cgroup-root=""
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276227    2530 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276230    2530 flags.go:64] FLAG: --client-ca-file=""
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276233    2530 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276236    2530 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276240    2530 flags.go:64] FLAG: --cluster-domain=""
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276244    2530 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276247    2530 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276250    2530 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276254    2530 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 20:32:42.281647 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276263    2530 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276267    2530 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276270    2530 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276273    2530 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276276    2530 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276279    2530 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276282    2530 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276285    2530 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276288    2530 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276292    2530 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276295    2530 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276298    2530 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276300    2530 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276303    2530 flags.go:64] FLAG: --enable-server="true"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276307    2530 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276313    2530 flags.go:64] FLAG: --event-burst="100"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276316    2530 flags.go:64] FLAG: --event-qps="50"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276319    2530 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276322    2530 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276324    2530 flags.go:64] FLAG: --eviction-hard=""
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276328    2530 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276331    2530 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276335    2530 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276338    2530 flags.go:64] FLAG: --eviction-soft=""
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276341    2530 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 20:32:42.282264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276344    2530 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276346    2530 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276349    2530 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276354    2530 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276356    2530 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276359    2530 flags.go:64] FLAG: --feature-gates=""
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276363    2530 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276367    2530 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276370    2530 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276374    2530 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276376    2530 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276379    2530 flags.go:64] FLAG: --help="false"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276382    2530 flags.go:64] FLAG: --hostname-override="ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276385    2530 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276388    2530 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276391    2530 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276395    2530 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276398    2530 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276401    2530 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276404    2530 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276406    2530 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276410    2530 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276416    2530 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276419    2530 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 20:32:42.282960 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276422    2530 flags.go:64] FLAG: --kube-reserved=""
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276425    2530 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276428    2530 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276431    2530 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276434    2530 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276436    2530 flags.go:64] FLAG: --lock-file=""
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276439    2530 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276443    2530 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276446    2530 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276451    2530 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276454    2530 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276458    2530 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276460    2530 flags.go:64] FLAG: --logging-format="text"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276463    2530 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276467    2530 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276471    2530 flags.go:64] FLAG: --manifest-url=""
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276474    2530 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276478    2530 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276481    2530 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276485    2530 flags.go:64] FLAG: --max-pods="110"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276488    2530 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276491    2530 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276494    2530 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276497    2530 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276500    2530 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 20:32:42.283578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276503    2530 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276506    2530 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276514    2530 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276517    2530 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276520    2530 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276523    2530 flags.go:64] FLAG: --pod-cidr=""
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276526    2530 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276532    2530 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276535    2530 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276538    2530 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276541    2530 flags.go:64] FLAG: --port="10250"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276544    2530 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276547    2530 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f4468ee79b220acb"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276550    2530 flags.go:64] FLAG: --qos-reserved=""
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276553    2530 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276556    2530 flags.go:64] FLAG: --register-node="true"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276559    2530 flags.go:64] FLAG: --register-with-taints=""
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276562    2530 flags.go:64] FLAG: --registry-burst="10"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276566    2530 flags.go:64] FLAG: --registry-qps="5"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276569    2530 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276572    2530 flags.go:64] FLAG: --reserved-memory=""
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276576    2530 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276579    2530 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276585    2530 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276588    2530 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 20:32:42.284210 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276591    2530 flags.go:64] FLAG: --runonce="false"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276594    2530 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276597    2530 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276600    2530 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276603    2530 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276606    2530 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276609    2530 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276612    2530 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276615    2530 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276618    2530 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276621    2530 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276624    2530 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276626    2530 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276629    2530 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276636    2530 flags.go:64] FLAG: --system-cgroups=""
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276639    2530 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276644    2530 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276647    2530 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276650    2530 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276653    2530 flags.go:64] FLAG: --tls-min-version=""
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276656    2530 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276659    2530 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276662    2530 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276665    2530 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276667    2530 flags.go:64] FLAG: --v="2"
Jul 16 20:32:42.284821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276673    2530 flags.go:64] FLAG: --version="false"
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276677    2530 flags.go:64] FLAG: --vmodule=""
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276681    2530 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.276684    2530 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276803    2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276809    2530 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276812    2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276815    2530 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276818    2530 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276821    2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276824    2530 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276827    2530 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276830    2530 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276833    2530 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276836    2530 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276839    2530 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276841    2530 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276844    2530 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276846    2530 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276849    2530 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 20:32:42.285472 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276851    2530 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276854    2530 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276857    2530 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276860    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276862    2530 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276865    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276867    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276870    2530 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276873    2530 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276875    2530 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276878    2530 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276880    2530 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276884    2530 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276887    2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276889    2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276892    2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276894    2530 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276898    2530 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276901    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276904    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 20:32:42.285965 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276906    2530 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276909    2530 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276912    2530 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276916    2530 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276919    2530 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276921    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276924    2530 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276926    2530 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276929    2530 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276931    2530 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276934    2530 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276936    2530 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276939    2530 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276941    2530 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276945    2530 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276955    2530 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276958    2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276961    2530 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276963    2530 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 20:32:42.286626 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276966    2530 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276968    2530 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276971    2530 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276973    2530 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276976    2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276980    2530 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276982    2530 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276985    2530 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276987    2530 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276990    2530 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276994    2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.276997    2530 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277005    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277007    2530 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277010    2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277012    2530 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277016    2530 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277019    2530 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277022    2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277025    2530 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 20:32:42.287124 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277028    2530 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277030    2530 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277033    2530 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277037    2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277039    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277042    2530 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277045    2530 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277047    2530 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277050    2530 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277060    2530 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277063    2530 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277066    2530 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277068    2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277071    2530 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277073    2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277076    2530 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277078    2530 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277082    2530 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277084    2530 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277087    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 20:32:42.287625 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277089    2530 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 20:32:42.288157 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277091    2530 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 20:32:42.288157 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277117    2530 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 20:32:42.288157 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.277122    2530 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 20:32:42.288157 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.277130    2530 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 20:32:42.288157 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.284239    2530 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 20:32:42.288157 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.284262    2530 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 20:32:42.288157 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.285165    2530 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 20:32:42.288157 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.285175    2530 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 20:32:42.288157 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285244    2530 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 20:32:42.288157 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285250    2530 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 20:32:42.288157 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285255    2530 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 20:32:42.288157 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285260    2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 20:32:42.288157 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285263    2530 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 20:32:42.288157 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285267    2530 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285270    2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285273    2530 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285275    2530 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285278    2530 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285281    2530 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285284    2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285286    2530 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285289    2530 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285292    2530 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285295    2530 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285297    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285300    2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285302    2530 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285305    2530 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285307    2530 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285309    2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285312    2530 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285314    2530 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285317    2530 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 20:32:42.288553 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285320    2530 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285322    2530 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285325    2530 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285328    2530 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285332    2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285335    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285338    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285340    2530 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285343    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285346    2530 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285348    2530 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285351    2530 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285353    2530 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285356    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285358    2530 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285360    2530 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285363    2530 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285365    2530 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285368    2530 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285370    2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 20:32:42.289062 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285373    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285375    2530 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285378    2530 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285380    2530 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285383    2530 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285385    2530 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285387    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285390    2530 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285393    2530 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285395    2530 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285398    2530 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285400    2530 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285403    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285405    2530 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285407    2530 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285410    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285414    2530 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285417    2530 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285421    2530 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 20:32:42.289572 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285424    2530 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285427    2530 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285430    2530 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285433    2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285436    2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285439    2530 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285441    2530 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285444    2530 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285447    2530 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285450    2530 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285455    2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285457    2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285460    2530 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285463    2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285465    2530 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285468    2530 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285470    2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285474    2530 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285476    2530 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 20:32:42.290200 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285479    2530 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 20:32:42.290757 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285481    2530 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 20:32:42.290757 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285484    2530 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 20:32:42.290757 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285487    2530 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 20:32:42.290757 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285490    2530 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 20:32:42.290757 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285492    2530 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 20:32:42.290757 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285495    2530 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 20:32:42.290757 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285497    2530 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 20:32:42.290757 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285499    2530 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 20:32:42.290757 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285502    2530 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 20:32:42.290757 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285505    2530 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 20:32:42.290757 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285508    2530 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 20:32:42.290757 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285511    2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 20:32:42.290757 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285514    2530 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 20:32:42.290757 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285516    2530 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 20:32:42.290757 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285519    2530 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 20:32:42.291189 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.285524    2530 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 20:32:42.291189 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285649    2530 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 20:32:42.291189 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285655    2530 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 20:32:42.291189 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285658    2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 20:32:42.291189 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285661    2530 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 20:32:42.291189 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285664    2530 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 20:32:42.291189 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285667    2530 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 20:32:42.291189 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285670    2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 20:32:42.291189 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285672    2530 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 20:32:42.291189 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285675    2530 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 20:32:42.291189 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285677    2530 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 20:32:42.291189 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285680    2530 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 20:32:42.291189 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285682    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 20:32:42.291189 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285685    2530 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 20:32:42.291189 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285687    2530 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285690    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285692    2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285695    2530 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285697    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285699    2530 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285702    2530 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285704    2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285707    2530 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285710    2530 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285712    2530 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285715    2530 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285718    2530 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285721    2530 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285723    2530 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285726    2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285729    2530 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285732    2530 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285734    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 20:32:42.291575 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285738    2530 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 20:32:42.292088 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285742    2530 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 20:32:42.292088 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285745    2530 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 20:32:42.292088 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285749    2530 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 20:32:42.292088 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285752    2530 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 20:32:42.292088 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285756    2530 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 20:32:42.292088 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285759    2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 20:32:42.292088 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285762    2530 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 20:32:42.292088 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285764    2530 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 20:32:42.292088 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285767    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 20:32:42.292088 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285769    2530 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 20:32:42.292088 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285772    2530 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 20:32:42.292088 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285774    2530 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 20:32:42.292088 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285777    2530 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 20:32:42.292088 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285779    2530 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 20:32:42.292088 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285782    2530 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 20:32:42.292088 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285784    2530 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 20:32:42.292088 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285786    2530 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 20:32:42.292088 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285789    2530 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285791    2530 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285795    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285797    2530 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285800    2530 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285802    2530 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285804    2530 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285808    2530 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285810    2530 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285813    2530 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285816    2530 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285819    2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285822    2530 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285824    2530 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285826    2530 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285829    2530 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285831    2530 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285834    2530 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285836    2530 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285839    2530 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 20:32:42.292596 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285841    2530 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285844    2530 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285846    2530 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285849    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285851    2530 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285853    2530 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285856    2530 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285859    2530 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285862    2530 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285864    2530 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285867    2530 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285869    2530 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285871    2530 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285874    2530 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285876    2530 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285879    2530 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285881    2530 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285884    2530 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285886    2530 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285889    2530 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 20:32:42.293136 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285892    2530 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 20:32:42.293905 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285894    2530 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 20:32:42.293905 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285897    2530 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 20:32:42.293905 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285899    2530 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 20:32:42.293905 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285902    2530 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 20:32:42.293905 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285905    2530 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 20:32:42.293905 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285907    2530 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 20:32:42.293905 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285910    2530 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 20:32:42.293905 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:42.285913    2530 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 20:32:42.293905 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.285918    2530 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 20:32:42.293905 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.286056    2530 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 20:32:42.293905 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.288163    2530 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 20:32:42.293905 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.289569    2530 server.go:1019] "Starting client certificate rotation"
Jul 16 20:32:42.293905 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.289700    2530 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 20:32:42.293905 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.290663    2530 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 20:32:42.294550 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.292183    2530 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 20:32:42.295545 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.295499    2530 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 20:32:42.305389 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.305365    2530 log.go:25] "Validated CRI v1 runtime API"
Jul 16 20:32:42.307571 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.307550    2530 log.go:25] "Validated CRI v1 image API"
Jul 16 20:32:42.309314 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.309294    2530 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 20:32:42.312636 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.312607    2530 fs.go:135] Filesystem UUIDs: map[10f6d20a-2b96-4d7f-bc77-b90efb2718c7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 901b5afc-a5dd-45d0-a9fe-6d395748ac9a:/dev/nvme0n1p3]
Jul 16 20:32:42.312717 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.312636    2530 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 20:32:42.318240 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.318090    2530 manager.go:216] Machine: {Timestamp:2026-07-16 20:32:42.316726171 +0000 UTC m=+0.338313947 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099659 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2191cb75c02c65fe2b5c8360dfa384 SystemUUID:ec2191cb-75c0-2c65-fe2b-5c8360dfa384 BootID:a6c09d57-bff5-4692-8d7a-66c3d59bf934 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fe:cc:da:22:45 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fe:cc:da:22:45 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:b8:c4:ce:67:81 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 20:32:42.318240 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.318230    2530 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 20:32:42.318393 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.318322    2530 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 20:32:42.322166 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.322135    2530 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 20:32:42.322346 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.322169    2530 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-249.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 20:32:42.322520 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.322505    2530 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 20:32:42.323010 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.322996    2530 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 20:32:42.323052 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.323014    2530 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 20:32:42.323052 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.323023    2530 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 20:32:42.323939 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.323926    2530 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 20:32:42.325315 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.325303    2530 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 20:32:42.328720 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.328703    2530 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 20:32:42.328865 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.328855    2530 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 20:32:42.328895 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.328888    2530 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 20:32:42.328925 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.328899    2530 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 20:32:42.328925 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.328916    2530 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 20:32:42.328979 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.328926    2530 kubelet.go:398] "Adding apiserver pod source"
Jul 16 20:32:42.328979 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.328936    2530 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 20:32:42.329344 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.329329    2530 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 20:32:42.329344 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.329347    2530 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 20:32:42.332506 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.332484    2530 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 20:32:42.341831 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:42.341805    2530 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-249.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 20:32:42.341831 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:42.341806    2530 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 20:32:42.342336 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.342322    2530 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 20:32:42.342373 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.342354    2530 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 20:32:42.343010 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.342997    2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 20:32:42.343066 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.343014    2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 20:32:42.343066 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.343027    2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 20:32:42.343066 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.343033    2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 20:32:42.343066 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.343042    2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 20:32:42.343066 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.343048    2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 20:32:42.343252 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.343138    2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 20:32:42.343252 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.343145    2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 20:32:42.343252 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.343151    2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 20:32:42.343252 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.343157    2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 20:32:42.343252 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.343169    2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 20:32:42.343252 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.343177    2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 20:32:42.344606 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.344594    2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 20:32:42.344606 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.344606    2530 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 20:32:42.345382 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.345355    2530 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n9mvc"
Jul 16 20:32:42.348840 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.348821    2530 server.go:1268] "Started kubelet"
Jul 16 20:32:42.348956 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.348907    2530 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 20:32:42.349002 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.348898    2530 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 20:32:42.349142 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.349122    2530 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 20:32:42.350090 ip-10-0-136-249 systemd[1]: Started Kubernetes Kubelet.
Jul 16 20:32:42.350673 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.350452    2530 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 20:32:42.351589 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.351574    2530 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 20:32:42.355071 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.355050    2530 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n9mvc"
Jul 16 20:32:42.355197 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.355091    2530 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-249.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 20:32:42.356309 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:42.356286    2530 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 20:32:42.362521 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.362501    2530 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 20:32:42.362521 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.362512    2530 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 20:32:42.362980 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:42.362708    2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-249.ec2.internal\" not found"
Jul 16 20:32:42.363648 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.363226    2530 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 20:32:42.363648 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.363290    2530 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 20:32:42.363648 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.363304    2530 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 20:32:42.363648 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.363414    2530 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 20:32:42.363648 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.363423    2530 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 20:32:42.363919 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.363695    2530 factory.go:55] Registering systemd factory
Jul 16 20:32:42.363919 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.363711    2530 factory.go:223] Registration of the systemd container factory successfully
Jul 16 20:32:42.364018 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.363946    2530 factory.go:153] Registering CRI-O factory
Jul 16 20:32:42.364018 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.363961    2530 factory.go:223] Registration of the crio container factory successfully
Jul 16 20:32:42.364018 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.364014    2530 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 20:32:42.364171 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.364042    2530 factory.go:103] Registering Raw factory
Jul 16 20:32:42.364171 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.364055    2530 manager.go:1196] Started watching for new ooms in manager
Jul 16 20:32:42.364545 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.364520    2530 manager.go:349] Starting recovery of all containers
Jul 16 20:32:42.367973 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.367948    2530 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 20:32:42.371197 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:42.371025    2530 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-249.ec2.internal\" not found" node="ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.374554 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.374531    2530 manager.go:354] Recovery completed
Jul 16 20:32:42.378964 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.378945    2530 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:32:42.380917 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.380892    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:32:42.381022 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.380938    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:32:42.381022 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.380951    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:32:42.381511 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.381497    2530 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 20:32:42.381576 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.381510    2530 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 20:32:42.381576 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.381529    2530 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 20:32:42.381576 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.381530    2530 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:32:42.382877 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.382856    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:32:42.382957 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.382890    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:32:42.382957 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.382910    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:32:42.384422 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.384404    2530 policy_none.go:49] "None policy: Start"
Jul 16 20:32:42.384492 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.384436    2530 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 20:32:42.384492 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.384448    2530 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 20:32:42.385509 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.385494    2530 policy_none.go:47] "Start"
Jul 16 20:32:42.433848 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.433821    2530 manager.go:337] "Starting Device Plugin manager"
Jul 16 20:32:42.434948 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:42.433867    2530 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 20:32:42.434948 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.433879    2530 server.go:87] "Starting device plugin registration server"
Jul 16 20:32:42.434948 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.434202    2530 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 20:32:42.434948 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.434216    2530 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 20:32:42.434948 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.434312    2530 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 20:32:42.434948 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.434427    2530 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 20:32:42.434948 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.434435    2530 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 20:32:42.435261 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:42.435064    2530 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 20:32:42.435261 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:42.435149    2530 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-249.ec2.internal\" not found"
Jul 16 20:32:42.499979 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.499937    2530 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 20:32:42.501325 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.501305    2530 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 20:32:42.501437 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.501343    2530 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 20:32:42.501437 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.501368    2530 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 20:32:42.501437 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:42.501410    2530 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 20:32:42.503863 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.503837    2530 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 20:32:42.535045 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.534960    2530 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:32:42.536670 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.536637    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:32:42.536670 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.536674    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:32:42.536829 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.536686    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:32:42.536829 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.536709    2530 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.546062 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.546037    2530 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.546201 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:42.546067    2530 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-249.ec2.internal\": node \"ip-10-0-136-249.ec2.internal\" not found"
Jul 16 20:32:42.569979 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:42.569946    2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-249.ec2.internal\" not found"
Jul 16 20:32:42.602032 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.601996    2530 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-249.ec2.internal"]
Jul 16 20:32:42.602127 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.602117    2530 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:32:42.603813 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.603792    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:32:42.603906 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.603828    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:32:42.603906 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.603843    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:32:42.605158 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.605143    2530 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:32:42.605204 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.605189    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.605236 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.605216    2530 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:32:42.606756 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.606739    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:32:42.606837 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.606773    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:32:42.606837 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.606783    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:32:42.606837 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.606809    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:32:42.606837 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.606820    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:32:42.606837 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.606787    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:32:42.607993 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.607977    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.608045 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.608009    2530 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 20:32:42.608946 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.608931    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 20:32:42.609041 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.608952    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 20:32:42.609041 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.608962    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeHasSufficientPID"
Jul 16 20:32:42.641975 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:42.641948    2530 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-249.ec2.internal\" not found" node="ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.646393 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:42.646374    2530 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-249.ec2.internal\" not found" node="ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.665014 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.664972    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7535ca3b2d933dbcbdc547452fd525db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal\" (UID: \"7535ca3b2d933dbcbdc547452fd525db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.665014 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.665012    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7535ca3b2d933dbcbdc547452fd525db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal\" (UID: \"7535ca3b2d933dbcbdc547452fd525db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.665212 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.665034    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/19742d85a0eb69c9b3078865a8f5b5d4-config\") pod \"kube-apiserver-proxy-ip-10-0-136-249.ec2.internal\" (UID: \"19742d85a0eb69c9b3078865a8f5b5d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.670045 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:42.670017    2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-249.ec2.internal\" not found"
Jul 16 20:32:42.765308 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.765256    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7535ca3b2d933dbcbdc547452fd525db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal\" (UID: \"7535ca3b2d933dbcbdc547452fd525db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.765308 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.765313    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7535ca3b2d933dbcbdc547452fd525db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal\" (UID: \"7535ca3b2d933dbcbdc547452fd525db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.765537 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.765332    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7535ca3b2d933dbcbdc547452fd525db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal\" (UID: \"7535ca3b2d933dbcbdc547452fd525db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.765537 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.765336    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/19742d85a0eb69c9b3078865a8f5b5d4-config\") pod \"kube-apiserver-proxy-ip-10-0-136-249.ec2.internal\" (UID: \"19742d85a0eb69c9b3078865a8f5b5d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.765537 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.765401    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7535ca3b2d933dbcbdc547452fd525db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal\" (UID: \"7535ca3b2d933dbcbdc547452fd525db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.765537 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.765402    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/19742d85a0eb69c9b3078865a8f5b5d4-config\") pod \"kube-apiserver-proxy-ip-10-0-136-249.ec2.internal\" (UID: \"19742d85a0eb69c9b3078865a8f5b5d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.770394 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:42.770367    2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-249.ec2.internal\" not found"
Jul 16 20:32:42.871379 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:42.871298    2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-249.ec2.internal\" not found"
Jul 16 20:32:42.944530 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.944500    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.950480 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:42.950460    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.ec2.internal"
Jul 16 20:32:42.972438 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:42.972385    2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-249.ec2.internal\" not found"
Jul 16 20:32:43.072952 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:43.072907    2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-249.ec2.internal\" not found"
Jul 16 20:32:43.173594 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:43.173524    2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-249.ec2.internal\" not found"
Jul 16 20:32:43.246526 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:43.246486    2530 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 20:32:43.274001 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:43.273962    2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-249.ec2.internal\" not found"
Jul 16 20:32:43.289195 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:43.289136    2530 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 20:32:43.289389 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:43.289331    2530 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 20:32:43.289389 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:43.289342    2530 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 20:32:43.289389 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:43.289335    2530 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 20:32:43.361249 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:43.361206    2530 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 20:27:42 +0000 UTC" deadline="2028-03-12 00:53:39.630728902 +0000 UTC"
Jul 16 20:32:43.361249 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:43.361241    2530 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14500h20m56.269491684s"
Jul 16 20:32:43.374174 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:43.374142    2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-249.ec2.internal\" not found"
Jul 16 20:32:43.474911 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:43.474830    2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-249.ec2.internal\" not found"
Jul 16 20:32:43.528291 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:43.528251    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19742d85a0eb69c9b3078865a8f5b5d4.slice/crio-93138d5b67d211bca1db2ee28881fa880ad7833a4230ef6ce241a3834933d469 WatchSource:0}: Error finding container 93138d5b67d211bca1db2ee28881fa880ad7833a4230ef6ce241a3834933d469: Status 404 returned error can't find the container with id 93138d5b67d211bca1db2ee28881fa880ad7833a4230ef6ce241a3834933d469
Jul 16 20:32:43.529036 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:43.529015    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7535ca3b2d933dbcbdc547452fd525db.slice/crio-3fc52b9af19ed77f61b086d0ce33f9b76bcbfe99ec58ca024ad2b3319cf6107c WatchSource:0}: Error finding container 3fc52b9af19ed77f61b086d0ce33f9b76bcbfe99ec58ca024ad2b3319cf6107c: Status 404 returned error can't find the container with id 3fc52b9af19ed77f61b086d0ce33f9b76bcbfe99ec58ca024ad2b3319cf6107c
Jul 16 20:32:43.534323 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:43.534305    2530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 20:32:43.574998 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:43.574950    2530 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-249.ec2.internal\" not found"
Jul 16 20:32:43.621500 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:43.621477    2530 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 20:32:43.663309 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:43.663276    2530 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.ec2.internal"
Jul 16 20:32:43.673503 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:43.673482    2530 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 20:32:43.674422 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:43.674408    2530 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal"
Jul 16 20:32:43.683866 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:43.683840    2530 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 20:32:44.330152 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.330068    2530 apiserver.go:52] "Watching apiserver"
Jul 16 20:32:44.337330 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.337291    2530 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 20:32:44.337886 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.337858    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-cwld6","openshift-multus/multus-additional-cni-plugins-xttrm","openshift-multus/multus-hcx7t","openshift-multus/network-metrics-daemon-4mmnt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b","openshift-dns/node-resolver-764h9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal","openshift-network-diagnostics/network-check-target-vxmfl","openshift-network-operator/iptables-alerter-jlmxh","openshift-ovn-kubernetes/ovnkube-node-blwng","kube-system/global-pull-secret-syncer-tjnr2","kube-system/konnectivity-agent-nf9hg","kube-system/kube-apiserver-proxy-ip-10-0-136-249.ec2.internal","openshift-cluster-node-tuning-operator/tuned-9th7p"]
Jul 16 20:32:44.340792 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.340762    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.343527 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.343495    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 20:32:44.343691 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.343498    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 20:32:44.343981 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.343962    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 20:32:44.344399 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.343969    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d2w8n\""
Jul 16 20:32:44.344399 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.344197    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 20:32:44.344399 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.344212    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 20:32:44.344399 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.344217    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 20:32:44.344763 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.344746    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:44.344958 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:44.344923    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:32:44.348645 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.348237    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.350263 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.350244    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 20:32:44.350597 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.350580    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 20:32:44.350996 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.350801    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-djmkv\""
Jul 16 20:32:44.350996 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.350802    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 20:32:44.350996 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.350852    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 20:32:44.351235 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.351130    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:32:44.351235 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:44.351211    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4mmnt" podUID="864a17b7-c612-43d2-923e-8843e0f519b0"
Jul 16 20:32:44.353742 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.353719    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.356421 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.356387    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 20:32:44.356547 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.356436    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 20:32:44.356606 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.356559    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-prnz2\""
Jul 16 20:32:44.356978 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.356958    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-764h9"
Jul 16 20:32:44.357078 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.356172    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 20:32:44.359840 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.359804    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jlmxh"
Jul 16 20:32:44.362421 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.362396    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2frcx\""
Jul 16 20:32:44.362732 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.362718    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 20:32:44.362928 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.362914    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 20:32:44.363103 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.363090    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 20:32:44.363763 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.363742    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-64bd6\""
Jul 16 20:32:44.364157 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.364138    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 20:32:44.364267 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.364166    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 20:32:44.364460 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.364432    2530 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 20:32:44.364578 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.364557    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.367005 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.366974    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.368305 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.368009    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 20:32:44.368305 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.368051    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-df7rt\""
Jul 16 20:32:44.368305 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.368141    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 20:32:44.370743 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.370719    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 20:32:44.370909 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.370892    2530 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 20:32:44.371117 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.371102    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ck495\""
Jul 16 20:32:44.371279 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.371148    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 20:32:44.371279 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.371194    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cwld6"
Jul 16 20:32:44.373567 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.373529    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 20:32:44.373788 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.373769    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 20:32:44.373885 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.373865    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7jngz\""
Jul 16 20:32:44.374156 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.374004    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 20:32:44.374304 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.374226    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:32:44.374374 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:44.374307    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxmfl" podUID="2aed23ed-5e27-4e01-adc6-d8af18c38baa"
Jul 16 20:32:44.374753 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.374721    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-run-ovn-kubernetes\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.374835 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.374762    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.374835 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.374789    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-kubelet-config\") pod \"global-pull-secret-syncer-tjnr2\" (UID: \"7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e\") " pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:44.374835 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.374814    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-host-var-lib-cni-multus\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.375006 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.374839    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.375006 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.374864    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-etc-selinux\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.375006 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.374889    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-run\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.375006 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.374915    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-slash\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.375006 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.374974    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-host-run-netns\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.375290 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375016    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-host-var-lib-cni-bin\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.375290 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375042    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-host-var-lib-kubelet\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.375290 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375086    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-sys-fs\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.375290 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375112    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz5h7\" (UniqueName: \"kubernetes.io/projected/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-kube-api-access-xz5h7\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.375290 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375142    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t8zk\" (UniqueName: \"kubernetes.io/projected/efdaaf6e-f70a-4b31-9bda-0104eb2ef52f-kube-api-access-6t8zk\") pod \"iptables-alerter-jlmxh\" (UID: \"efdaaf6e-f70a-4b31-9bda-0104eb2ef52f\") " pod="openshift-network-operator/iptables-alerter-jlmxh"
Jul 16 20:32:44.375290 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375207    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l96sr\" (UniqueName: \"kubernetes.io/projected/03510e23-51da-4185-9da4-f908281932e6-kube-api-access-l96sr\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.375290 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375258    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-cnibin\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.375290 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375290    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-run-ovn\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.375656 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375314    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-cni-bin\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.375656 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375339    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-multus-socket-dir-parent\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.375656 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375366    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-sys\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.375656 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375393    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.375656 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375419    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-etc-openvswitch\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.375656 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375483    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ff556300-6da3-45d7-9415-c5b0f124cd2f-multus-daemon-config\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.375656 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375513    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/30f9d985-30db-4a7b-9f9d-554db3992a61-ovnkube-script-lib\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.375656 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375528    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-cnibin\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.375656 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375543    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-hostroot\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.375656 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375559    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/efdaaf6e-f70a-4b31-9bda-0104eb2ef52f-iptables-alerter-script\") pod \"iptables-alerter-jlmxh\" (UID: \"efdaaf6e-f70a-4b31-9bda-0104eb2ef52f\") " pod="openshift-network-operator/iptables-alerter-jlmxh"
Jul 16 20:32:44.375656 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375574    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-var-lib-openvswitch\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.375656 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375588    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-host\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.375656 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375603    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6p5m\" (UniqueName: \"kubernetes.io/projected/30f9d985-30db-4a7b-9f9d-554db3992a61-kube-api-access-c6p5m\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.375656 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375617    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-registration-dir\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.375656 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375633    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9388c864-76b1-415e-a3d8-6114d5968730-hosts-file\") pod \"node-resolver-764h9\" (UID: \"9388c864-76b1-415e-a3d8-6114d5968730\") " pod="openshift-dns/node-resolver-764h9"
Jul 16 20:32:44.375656 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375649    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/30f9d985-30db-4a7b-9f9d-554db3992a61-env-overrides\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.376342 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375673    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff556300-6da3-45d7-9415-c5b0f124cd2f-cni-binary-copy\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.376342 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375689    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.376342 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375711    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsqgm\" (UniqueName: \"kubernetes.io/projected/9388c864-76b1-415e-a3d8-6114d5968730-kube-api-access-vsqgm\") pod \"node-resolver-764h9\" (UID: \"9388c864-76b1-415e-a3d8-6114d5968730\") " pod="openshift-dns/node-resolver-764h9"
Jul 16 20:32:44.376342 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375731    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/03510e23-51da-4185-9da4-f908281932e6-etc-tuned\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.376342 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.375990    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-host-run-multus-certs\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.376342 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376018    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-log-socket\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.376342 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376055    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-multus-conf-dir\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.376342 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376070    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-etc-sysctl-d\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.376342 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376113    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-lib-modules\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.376342 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376168    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-cni-binary-copy\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.376342 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376228    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.376342 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376288    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-host-run-k8s-cni-cncf-io\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.376342 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376317    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs\") pod \"network-metrics-daemon-4mmnt\" (UID: \"864a17b7-c612-43d2-923e-8843e0f519b0\") " pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:32:44.376342 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376351    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/efdaaf6e-f70a-4b31-9bda-0104eb2ef52f-host-slash\") pod \"iptables-alerter-jlmxh\" (UID: \"efdaaf6e-f70a-4b31-9bda-0104eb2ef52f\") " pod="openshift-network-operator/iptables-alerter-jlmxh"
Jul 16 20:32:44.377049 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376378    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-etc-systemd\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.377049 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376406    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-multus-cni-dir\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.377049 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376441    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-etc-kubernetes\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.377049 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376466    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-etc-sysctl-conf\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.377049 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376481    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03510e23-51da-4185-9da4-f908281932e6-tmp\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.377049 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376511    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-node-log\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.377049 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376526    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-dbus\") pod \"global-pull-secret-syncer-tjnr2\" (UID: \"7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e\") " pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:44.377049 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376543    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret\") pod \"global-pull-secret-syncer-tjnr2\" (UID: \"7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e\") " pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:44.377049 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376564    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-os-release\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.377049 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376613    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-os-release\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.377049 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376632    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-run-netns\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.377049 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376648    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-run-openvswitch\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.377049 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376663    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-etc-kubernetes\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.377049 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376685    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q28d7\" (UniqueName: \"kubernetes.io/projected/ff556300-6da3-45d7-9415-c5b0f124cd2f-kube-api-access-q28d7\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.377049 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376704    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-socket-dir\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.377049 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376720    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-device-dir\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.377888 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376740    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-system-cni-dir\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.377888 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376799    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jqpf\" (UniqueName: \"kubernetes.io/projected/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-kube-api-access-2jqpf\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.377888 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376836    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/30f9d985-30db-4a7b-9f9d-554db3992a61-ovnkube-config\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.377888 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376853    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-system-cni-dir\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.377888 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376870    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zdsl\" (UniqueName: \"kubernetes.io/projected/864a17b7-c612-43d2-923e-8843e0f519b0-kube-api-access-6zdsl\") pod \"network-metrics-daemon-4mmnt\" (UID: \"864a17b7-c612-43d2-923e-8843e0f519b0\") " pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:32:44.377888 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376886    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/30f9d985-30db-4a7b-9f9d-554db3992a61-ovn-node-metrics-cert\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.377888 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376909    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9388c864-76b1-415e-a3d8-6114d5968730-tmp-dir\") pod \"node-resolver-764h9\" (UID: \"9388c864-76b1-415e-a3d8-6114d5968730\") " pod="openshift-dns/node-resolver-764h9"
Jul 16 20:32:44.377888 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376940    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-etc-modprobe-d\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.377888 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376958    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-etc-sysconfig\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.377888 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.376979    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-var-lib-kubelet\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.377888 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.377001    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-run-systemd\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.377888 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.377017    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-cni-netd\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.377888 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.377036    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-kubelet\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.377888 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.377052    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-systemd-units\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.377888 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.377350    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nf9hg"
Jul 16 20:32:44.380697 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.380671    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wrmmq\""
Jul 16 20:32:44.380922 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.380906    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 20:32:44.381112 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.381097    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 20:32:44.381718 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.381657    2530 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 20:32:44.401807 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.401772    2530 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-j5npl"
Jul 16 20:32:44.410618 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.410541    2530 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j5npl"
Jul 16 20:32:44.464307 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.464263    2530 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 20:32:44.477502 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477462    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-log-socket\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.477719 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477523    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-multus-conf-dir\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.477719 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477557    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-etc-sysctl-d\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.477719 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477582    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-lib-modules\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.477719 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477610    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-cni-binary-copy\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.477719 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477631    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.477719 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477653    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-host-run-k8s-cni-cncf-io\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.477998 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477728    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs\") pod \"network-metrics-daemon-4mmnt\" (UID: \"864a17b7-c612-43d2-923e-8843e0f519b0\") " pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:32:44.477998 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477747    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/efdaaf6e-f70a-4b31-9bda-0104eb2ef52f-host-slash\") pod \"iptables-alerter-jlmxh\" (UID: \"efdaaf6e-f70a-4b31-9bda-0104eb2ef52f\") " pod="openshift-network-operator/iptables-alerter-jlmxh"
Jul 16 20:32:44.477998 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477763    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-etc-systemd\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.477998 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477787    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-multus-cni-dir\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.477998 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477808    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fhrc\" (UniqueName: \"kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc\") pod \"network-check-target-vxmfl\" (UID: \"2aed23ed-5e27-4e01-adc6-d8af18c38baa\") " pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:32:44.477998 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477838    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/205ec3c3-fd73-4c63-9298-2fe9b3f8a625-konnectivity-ca\") pod \"konnectivity-agent-nf9hg\" (UID: \"205ec3c3-fd73-4c63-9298-2fe9b3f8a625\") " pod="kube-system/konnectivity-agent-nf9hg"
Jul 16 20:32:44.477998 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477869    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-etc-kubernetes\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.477998 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477886    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-etc-sysctl-conf\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.477998 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477910    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03510e23-51da-4185-9da4-f908281932e6-tmp\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.477998 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477925    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-node-log\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.477998 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477948    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-dbus\") pod \"global-pull-secret-syncer-tjnr2\" (UID: \"7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e\") " pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:44.477998 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477974    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret\") pod \"global-pull-secret-syncer-tjnr2\" (UID: \"7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e\") " pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:44.477998 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.477996    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-os-release\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.478654 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478013    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-os-release\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.478654 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478028    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-run-netns\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.478654 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478044    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-run-openvswitch\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.478654 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478060    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-etc-kubernetes\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.478654 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478082    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q28d7\" (UniqueName: \"kubernetes.io/projected/ff556300-6da3-45d7-9415-c5b0f124cd2f-kube-api-access-q28d7\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.478654 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478141    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-socket-dir\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.478654 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478164    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-device-dir\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.478654 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478202    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4288933c-4329-4853-8fb3-e3d46c34479f-serviceca\") pod \"node-ca-cwld6\" (UID: \"4288933c-4329-4853-8fb3-e3d46c34479f\") " pod="openshift-image-registry/node-ca-cwld6"
Jul 16 20:32:44.478654 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478225    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-system-cni-dir\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.478654 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478241    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jqpf\" (UniqueName: \"kubernetes.io/projected/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-kube-api-access-2jqpf\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.478654 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478256    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/30f9d985-30db-4a7b-9f9d-554db3992a61-ovnkube-config\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.478654 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478291    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-system-cni-dir\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.478654 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478320    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zdsl\" (UniqueName: \"kubernetes.io/projected/864a17b7-c612-43d2-923e-8843e0f519b0-kube-api-access-6zdsl\") pod \"network-metrics-daemon-4mmnt\" (UID: \"864a17b7-c612-43d2-923e-8843e0f519b0\") " pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:32:44.478654 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478349    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/30f9d985-30db-4a7b-9f9d-554db3992a61-ovn-node-metrics-cert\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.478654 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478371    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9388c864-76b1-415e-a3d8-6114d5968730-tmp-dir\") pod \"node-resolver-764h9\" (UID: \"9388c864-76b1-415e-a3d8-6114d5968730\") " pod="openshift-dns/node-resolver-764h9"
Jul 16 20:32:44.478654 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478387    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-etc-modprobe-d\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.478654 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478411    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-etc-sysconfig\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.479387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478435    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-var-lib-kubelet\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.479387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478451    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-run-systemd\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.479387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478466    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-cni-netd\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.479387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478486    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-kubelet\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.479387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478546    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-systemd-units\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.479387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478563    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-run-ovn-kubernetes\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.479387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478589    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.479387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478625    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-kubelet-config\") pod \"global-pull-secret-syncer-tjnr2\" (UID: \"7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e\") " pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:44.479387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478642    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-host-var-lib-cni-multus\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.479387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478658    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.479387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478682    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-etc-selinux\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.479387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478704    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-run\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.479387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478731    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-slash\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.479387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478746    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-host-run-netns\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.479387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478761    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-host-var-lib-cni-bin\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.479387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478777    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-host-var-lib-kubelet\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.479387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478793    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-sys-fs\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.480200 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478830    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz5h7\" (UniqueName: \"kubernetes.io/projected/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-kube-api-access-xz5h7\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.480200 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478862    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4288933c-4329-4853-8fb3-e3d46c34479f-host\") pod \"node-ca-cwld6\" (UID: \"4288933c-4329-4853-8fb3-e3d46c34479f\") " pod="openshift-image-registry/node-ca-cwld6"
Jul 16 20:32:44.480200 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.478893    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t8zk\" (UniqueName: \"kubernetes.io/projected/efdaaf6e-f70a-4b31-9bda-0104eb2ef52f-kube-api-access-6t8zk\") pod \"iptables-alerter-jlmxh\" (UID: \"efdaaf6e-f70a-4b31-9bda-0104eb2ef52f\") " pod="openshift-network-operator/iptables-alerter-jlmxh"
Jul 16 20:32:44.480200 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479404    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l96sr\" (UniqueName: \"kubernetes.io/projected/03510e23-51da-4185-9da4-f908281932e6-kube-api-access-l96sr\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.480200 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479427    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-cnibin\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.480200 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479444    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-run-ovn\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.480200 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479470    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-cni-bin\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.480200 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479495    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-multus-socket-dir-parent\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.480200 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479513    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-sys\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.480200 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479540    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.480200 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479574    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-etc-openvswitch\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.480200 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479598    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ff556300-6da3-45d7-9415-c5b0f124cd2f-multus-daemon-config\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.480200 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479615    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/30f9d985-30db-4a7b-9f9d-554db3992a61-ovnkube-script-lib\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.480200 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479632    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-cnibin\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.480200 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479652    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-hostroot\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.480200 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479672    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24vxq\" (UniqueName: \"kubernetes.io/projected/4288933c-4329-4853-8fb3-e3d46c34479f-kube-api-access-24vxq\") pod \"node-ca-cwld6\" (UID: \"4288933c-4329-4853-8fb3-e3d46c34479f\") " pod="openshift-image-registry/node-ca-cwld6"
Jul 16 20:32:44.480200 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479689    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/efdaaf6e-f70a-4b31-9bda-0104eb2ef52f-iptables-alerter-script\") pod \"iptables-alerter-jlmxh\" (UID: \"efdaaf6e-f70a-4b31-9bda-0104eb2ef52f\") " pod="openshift-network-operator/iptables-alerter-jlmxh"
Jul 16 20:32:44.480708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479706    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-var-lib-openvswitch\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.480708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479722    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/205ec3c3-fd73-4c63-9298-2fe9b3f8a625-agent-certs\") pod \"konnectivity-agent-nf9hg\" (UID: \"205ec3c3-fd73-4c63-9298-2fe9b3f8a625\") " pod="kube-system/konnectivity-agent-nf9hg"
Jul 16 20:32:44.480708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479748    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-host\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.480708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479774    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6p5m\" (UniqueName: \"kubernetes.io/projected/30f9d985-30db-4a7b-9f9d-554db3992a61-kube-api-access-c6p5m\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.480708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479795    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-registration-dir\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.480708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479812    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9388c864-76b1-415e-a3d8-6114d5968730-hosts-file\") pod \"node-resolver-764h9\" (UID: \"9388c864-76b1-415e-a3d8-6114d5968730\") " pod="openshift-dns/node-resolver-764h9"
Jul 16 20:32:44.480708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479828    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/30f9d985-30db-4a7b-9f9d-554db3992a61-env-overrides\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.480708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479843    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff556300-6da3-45d7-9415-c5b0f124cd2f-cni-binary-copy\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.480708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479860    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.480708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479889    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqgm\" (UniqueName: \"kubernetes.io/projected/9388c864-76b1-415e-a3d8-6114d5968730-kube-api-access-vsqgm\") pod \"node-resolver-764h9\" (UID: \"9388c864-76b1-415e-a3d8-6114d5968730\") " pod="openshift-dns/node-resolver-764h9"
Jul 16 20:32:44.480708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479915    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/03510e23-51da-4185-9da4-f908281932e6-etc-tuned\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.480708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.479932    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-host-run-multus-certs\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.480708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.480029    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-host-run-multus-certs\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.480708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.480073    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-log-socket\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.480708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.480101    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-multus-conf-dir\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.480708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.480232    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-etc-sysctl-d\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.480708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.480348    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-lib-modules\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.481436 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.481016    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-cni-binary-copy\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.481566 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.481535    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.481708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.481616    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-host-run-k8s-cni-cncf-io\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.481777 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:44.481735    2530 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:32:44.481845 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:44.481802    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs podName:864a17b7-c612-43d2-923e-8843e0f519b0 nodeName:}" failed. No retries permitted until 2026-07-16 20:32:44.981780523 +0000 UTC m=+3.003368282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs") pod "network-metrics-daemon-4mmnt" (UID: "864a17b7-c612-43d2-923e-8843e0f519b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:32:44.481920 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.481855    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/efdaaf6e-f70a-4b31-9bda-0104eb2ef52f-host-slash\") pod \"iptables-alerter-jlmxh\" (UID: \"efdaaf6e-f70a-4b31-9bda-0104eb2ef52f\") " pod="openshift-network-operator/iptables-alerter-jlmxh"
Jul 16 20:32:44.481920 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.481904    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-etc-systemd\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.482025 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.481950    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-multus-cni-dir\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.482025 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.482021    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-etc-kubernetes\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.482173 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.482106    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-etc-sysctl-conf\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.482496 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.482424    2530 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 20:32:44.483778 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.483281    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-slash\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.483778 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.483405    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-node-log\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.483778 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.483542    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-dbus\") pod \"global-pull-secret-syncer-tjnr2\" (UID: \"7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e\") " pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:44.483778 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.483699    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-socket-dir\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.483778 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.483756    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-device-dir\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.484125 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.483812    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-system-cni-dir\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.484125 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.483891    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-os-release\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.484125 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:44.484011    2530 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:32:44.484125 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:44.484067    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret podName:7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e nodeName:}" failed. No retries permitted until 2026-07-16 20:32:44.984048976 +0000 UTC m=+3.005636738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret") pod "global-pull-secret-syncer-tjnr2" (UID: "7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:32:44.484125 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484088    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/30f9d985-30db-4a7b-9f9d-554db3992a61-ovnkube-script-lib\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.484125 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484126    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-system-cni-dir\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.484474 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484135    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-os-release\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.484474 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484164    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-systemd-units\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.484474 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484396    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-host-run-netns\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.484474 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484450    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-host-var-lib-cni-bin\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.484733 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484484    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9388c864-76b1-415e-a3d8-6114d5968730-tmp-dir\") pod \"node-resolver-764h9\" (UID: \"9388c864-76b1-415e-a3d8-6114d5968730\") " pod="openshift-dns/node-resolver-764h9"
Jul 16 20:32:44.484733 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484499    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-host-var-lib-kubelet\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.484733 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484565    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-sys-fs\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.484733 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484636    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-etc-modprobe-d\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.484733 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484658    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/30f9d985-30db-4a7b-9f9d-554db3992a61-ovnkube-config\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.484733 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484693    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-etc-sysconfig\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.484733 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484726    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-run-netns\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.485133 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484743    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-var-lib-kubelet\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.485133 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484774    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-run-openvswitch\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.485133 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484789    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-run-systemd\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.485133 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484825    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-etc-kubernetes\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.485133 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484861    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-kubelet\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.485133 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484919    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-host-var-lib-cni-multus\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.485133 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484965    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-run-ovn-kubernetes\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.485133 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.484974    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.485133 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.485016    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-kubelet-config\") pod \"global-pull-secret-syncer-tjnr2\" (UID: \"7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e\") " pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:44.486230 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.485909    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-cnibin\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.486230 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.486124    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-run\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.486230 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.486174    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-etc-selinux\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.486442 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.486327    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-hostroot\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.486442 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.486418    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ff556300-6da3-45d7-9415-c5b0f124cd2f-multus-socket-dir-parent\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.486552 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.486476    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-host\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.486552 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.486493    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-var-lib-openvswitch\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.486651 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.486591    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-cnibin\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.486708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.486666    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-run-ovn\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.486763 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.486730    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-cni-bin\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.488124 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.485085    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.488124 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.487161    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff556300-6da3-45d7-9415-c5b0f124cd2f-cni-binary-copy\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.488124 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.487167    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/efdaaf6e-f70a-4b31-9bda-0104eb2ef52f-iptables-alerter-script\") pod \"iptables-alerter-jlmxh\" (UID: \"efdaaf6e-f70a-4b31-9bda-0104eb2ef52f\") " pod="openshift-network-operator/iptables-alerter-jlmxh"
Jul 16 20:32:44.488124 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.487383    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-registration-dir\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.488124 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.487510    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9388c864-76b1-415e-a3d8-6114d5968730-hosts-file\") pod \"node-resolver-764h9\" (UID: \"9388c864-76b1-415e-a3d8-6114d5968730\") " pod="openshift-dns/node-resolver-764h9"
Jul 16 20:32:44.488124 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.487573    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-etc-openvswitch\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.488124 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.487660    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03510e23-51da-4185-9da4-f908281932e6-sys\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.488124 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.487802    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.488124 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.487957    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/30f9d985-30db-4a7b-9f9d-554db3992a61-env-overrides\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.488124 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.488117    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.488827 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.488220    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/30f9d985-30db-4a7b-9f9d-554db3992a61-host-cni-netd\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.488827 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.488290    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/30f9d985-30db-4a7b-9f9d-554db3992a61-ovn-node-metrics-cert\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.488827 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.488385    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ff556300-6da3-45d7-9415-c5b0f124cd2f-multus-daemon-config\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.488827 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.488645    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03510e23-51da-4185-9da4-f908281932e6-tmp\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.490472 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.490446    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/03510e23-51da-4185-9da4-f908281932e6-etc-tuned\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.494037 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.494003    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz5h7\" (UniqueName: \"kubernetes.io/projected/c51a898b-93f5-4a94-80d0-6aa2f3c19caa-kube-api-access-xz5h7\") pod \"aws-ebs-csi-driver-node-2m28b\" (UID: \"c51a898b-93f5-4a94-80d0-6aa2f3c19caa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.494553 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.494521    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q28d7\" (UniqueName: \"kubernetes.io/projected/ff556300-6da3-45d7-9415-c5b0f124cd2f-kube-api-access-q28d7\") pod \"multus-hcx7t\" (UID: \"ff556300-6da3-45d7-9415-c5b0f124cd2f\") " pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.495359 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.495004    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jqpf\" (UniqueName: \"kubernetes.io/projected/586e85c8-fa49-4b8a-bc17-1d57c31b1c79-kube-api-access-2jqpf\") pod \"multus-additional-cni-plugins-xttrm\" (UID: \"586e85c8-fa49-4b8a-bc17-1d57c31b1c79\") " pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.495682 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.495653    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t8zk\" (UniqueName: \"kubernetes.io/projected/efdaaf6e-f70a-4b31-9bda-0104eb2ef52f-kube-api-access-6t8zk\") pod \"iptables-alerter-jlmxh\" (UID: \"efdaaf6e-f70a-4b31-9bda-0104eb2ef52f\") " pod="openshift-network-operator/iptables-alerter-jlmxh"
Jul 16 20:32:44.495944 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.495879    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zdsl\" (UniqueName: \"kubernetes.io/projected/864a17b7-c612-43d2-923e-8843e0f519b0-kube-api-access-6zdsl\") pod \"network-metrics-daemon-4mmnt\" (UID: \"864a17b7-c612-43d2-923e-8843e0f519b0\") " pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:32:44.497487 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.497459    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsqgm\" (UniqueName: \"kubernetes.io/projected/9388c864-76b1-415e-a3d8-6114d5968730-kube-api-access-vsqgm\") pod \"node-resolver-764h9\" (UID: \"9388c864-76b1-415e-a3d8-6114d5968730\") " pod="openshift-dns/node-resolver-764h9"
Jul 16 20:32:44.497985 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.497962    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6p5m\" (UniqueName: \"kubernetes.io/projected/30f9d985-30db-4a7b-9f9d-554db3992a61-kube-api-access-c6p5m\") pod \"ovnkube-node-blwng\" (UID: \"30f9d985-30db-4a7b-9f9d-554db3992a61\") " pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.498711 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.498690    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l96sr\" (UniqueName: \"kubernetes.io/projected/03510e23-51da-4185-9da4-f908281932e6-kube-api-access-l96sr\") pod \"tuned-9th7p\" (UID: \"03510e23-51da-4185-9da4-f908281932e6\") " pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.507255 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.507183    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.ec2.internal" event={"ID":"19742d85a0eb69c9b3078865a8f5b5d4","Type":"ContainerStarted","Data":"93138d5b67d211bca1db2ee28881fa880ad7833a4230ef6ce241a3834933d469"}
Jul 16 20:32:44.508692 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.508662    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal" event={"ID":"7535ca3b2d933dbcbdc547452fd525db","Type":"ContainerStarted","Data":"3fc52b9af19ed77f61b086d0ce33f9b76bcbfe99ec58ca024ad2b3319cf6107c"}
Jul 16 20:32:44.581284 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.581184    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fhrc\" (UniqueName: \"kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc\") pod \"network-check-target-vxmfl\" (UID: \"2aed23ed-5e27-4e01-adc6-d8af18c38baa\") " pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:32:44.581284 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.581244    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/205ec3c3-fd73-4c63-9298-2fe9b3f8a625-konnectivity-ca\") pod \"konnectivity-agent-nf9hg\" (UID: \"205ec3c3-fd73-4c63-9298-2fe9b3f8a625\") " pod="kube-system/konnectivity-agent-nf9hg"
Jul 16 20:32:44.581493 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.581297    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4288933c-4329-4853-8fb3-e3d46c34479f-serviceca\") pod \"node-ca-cwld6\" (UID: \"4288933c-4329-4853-8fb3-e3d46c34479f\") " pod="openshift-image-registry/node-ca-cwld6"
Jul 16 20:32:44.581493 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.581341    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4288933c-4329-4853-8fb3-e3d46c34479f-host\") pod \"node-ca-cwld6\" (UID: \"4288933c-4329-4853-8fb3-e3d46c34479f\") " pod="openshift-image-registry/node-ca-cwld6"
Jul 16 20:32:44.581493 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.581364    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24vxq\" (UniqueName: \"kubernetes.io/projected/4288933c-4329-4853-8fb3-e3d46c34479f-kube-api-access-24vxq\") pod \"node-ca-cwld6\" (UID: \"4288933c-4329-4853-8fb3-e3d46c34479f\") " pod="openshift-image-registry/node-ca-cwld6"
Jul 16 20:32:44.581493 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.581381    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/205ec3c3-fd73-4c63-9298-2fe9b3f8a625-agent-certs\") pod \"konnectivity-agent-nf9hg\" (UID: \"205ec3c3-fd73-4c63-9298-2fe9b3f8a625\") " pod="kube-system/konnectivity-agent-nf9hg"
Jul 16 20:32:44.581713 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.581634    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4288933c-4329-4853-8fb3-e3d46c34479f-host\") pod \"node-ca-cwld6\" (UID: \"4288933c-4329-4853-8fb3-e3d46c34479f\") " pod="openshift-image-registry/node-ca-cwld6"
Jul 16 20:32:44.581981 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.581957    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/205ec3c3-fd73-4c63-9298-2fe9b3f8a625-konnectivity-ca\") pod \"konnectivity-agent-nf9hg\" (UID: \"205ec3c3-fd73-4c63-9298-2fe9b3f8a625\") " pod="kube-system/konnectivity-agent-nf9hg"
Jul 16 20:32:44.582140 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.582097    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4288933c-4329-4853-8fb3-e3d46c34479f-serviceca\") pod \"node-ca-cwld6\" (UID: \"4288933c-4329-4853-8fb3-e3d46c34479f\") " pod="openshift-image-registry/node-ca-cwld6"
Jul 16 20:32:44.588521 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.586877    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/205ec3c3-fd73-4c63-9298-2fe9b3f8a625-agent-certs\") pod \"konnectivity-agent-nf9hg\" (UID: \"205ec3c3-fd73-4c63-9298-2fe9b3f8a625\") " pod="kube-system/konnectivity-agent-nf9hg"
Jul 16 20:32:44.588521 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:44.587862    2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:32:44.588521 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:44.587887    2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:32:44.588521 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:44.587900    2530 projected.go:196] Error preparing data for projected volume kube-api-access-9fhrc for pod openshift-network-diagnostics/network-check-target-vxmfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:32:44.588521 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:44.587969    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc podName:2aed23ed-5e27-4e01-adc6-d8af18c38baa nodeName:}" failed. No retries permitted until 2026-07-16 20:32:45.087951439 +0000 UTC m=+3.109539189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9fhrc" (UniqueName: "kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc") pod "network-check-target-vxmfl" (UID: "2aed23ed-5e27-4e01-adc6-d8af18c38baa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:32:44.590858 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.590816    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24vxq\" (UniqueName: \"kubernetes.io/projected/4288933c-4329-4853-8fb3-e3d46c34479f-kube-api-access-24vxq\") pod \"node-ca-cwld6\" (UID: \"4288933c-4329-4853-8fb3-e3d46c34479f\") " pod="openshift-image-registry/node-ca-cwld6"
Jul 16 20:32:44.657387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.657347    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:32:44.671396 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.671356    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hcx7t"
Jul 16 20:32:44.682693 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.682654    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b"
Jul 16 20:32:44.688057 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.688015    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-764h9"
Jul 16 20:32:44.698570 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.698537    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jlmxh"
Jul 16 20:32:44.705677 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.705647    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9th7p"
Jul 16 20:32:44.714070 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.713841    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xttrm"
Jul 16 20:32:44.723236 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.723204    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cwld6"
Jul 16 20:32:44.730408 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.730376    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nf9hg"
Jul 16 20:32:44.753952 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.753918    2530 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 20:32:44.985589 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.985492    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs\") pod \"network-metrics-daemon-4mmnt\" (UID: \"864a17b7-c612-43d2-923e-8843e0f519b0\") " pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:32:44.985589 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:44.985554    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret\") pod \"global-pull-secret-syncer-tjnr2\" (UID: \"7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e\") " pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:44.985806 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:44.985659    2530 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:32:44.985806 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:44.985726    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs podName:864a17b7-c612-43d2-923e-8843e0f519b0 nodeName:}" failed. No retries permitted until 2026-07-16 20:32:45.985710794 +0000 UTC m=+4.007298538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs") pod "network-metrics-daemon-4mmnt" (UID: "864a17b7-c612-43d2-923e-8843e0f519b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:32:44.985806 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:44.985659    2530 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:32:44.985806 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:44.985789    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret podName:7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e nodeName:}" failed. No retries permitted until 2026-07-16 20:32:45.985776806 +0000 UTC m=+4.007364573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret") pod "global-pull-secret-syncer-tjnr2" (UID: "7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:32:45.186624 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:45.186583    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fhrc\" (UniqueName: \"kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc\") pod \"network-check-target-vxmfl\" (UID: \"2aed23ed-5e27-4e01-adc6-d8af18c38baa\") " pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:32:45.186842 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:45.186757    2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:32:45.186842 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:45.186779    2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:32:45.186842 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:45.186792    2530 projected.go:196] Error preparing data for projected volume kube-api-access-9fhrc for pod openshift-network-diagnostics/network-check-target-vxmfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:32:45.187015 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:45.186866    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc podName:2aed23ed-5e27-4e01-adc6-d8af18c38baa nodeName:}" failed. No retries permitted until 2026-07-16 20:32:46.186846858 +0000 UTC m=+4.208434636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9fhrc" (UniqueName: "kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc") pod "network-check-target-vxmfl" (UID: "2aed23ed-5e27-4e01-adc6-d8af18c38baa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:32:45.414220 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:45.414176    2530 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 20:27:44 +0000 UTC" deadline="2027-12-13 19:49:47.265286286 +0000 UTC"
Jul 16 20:32:45.414220 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:45.414212    2530 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12359h17m1.851078607s"
Jul 16 20:32:45.502331 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:45.502288    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:45.502522 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:45.502450    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:32:45.705249 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:45.705216    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefdaaf6e_f70a_4b31_9bda_0104eb2ef52f.slice/crio-b20d0f3901aefbfc93032368201d28cb5038688594e5b1d35835a40f72c4640c WatchSource:0}: Error finding container b20d0f3901aefbfc93032368201d28cb5038688594e5b1d35835a40f72c4640c: Status 404 returned error can't find the container with id b20d0f3901aefbfc93032368201d28cb5038688594e5b1d35835a40f72c4640c
Jul 16 20:32:45.706183 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:45.706154    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205ec3c3_fd73_4c63_9298_2fe9b3f8a625.slice/crio-12f017b3a222ca179faaf36d94f42946169f0161f446031bfcbc399bb9069472 WatchSource:0}: Error finding container 12f017b3a222ca179faaf36d94f42946169f0161f446031bfcbc399bb9069472: Status 404 returned error can't find the container with id 12f017b3a222ca179faaf36d94f42946169f0161f446031bfcbc399bb9069472
Jul 16 20:32:45.707739 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:45.707676    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4288933c_4329_4853_8fb3_e3d46c34479f.slice/crio-fa5a27dceb4ac63dc81f7dda74797c3ef729b54aa1531d9c32995a9c33650751 WatchSource:0}: Error finding container fa5a27dceb4ac63dc81f7dda74797c3ef729b54aa1531d9c32995a9c33650751: Status 404 returned error can't find the container with id fa5a27dceb4ac63dc81f7dda74797c3ef729b54aa1531d9c32995a9c33650751
Jul 16 20:32:45.711549 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:45.711526    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03510e23_51da_4185_9da4_f908281932e6.slice/crio-a0fdb3bf6ead28d96c1d3a50246e9f445e2817ebd158aded66e729e5972c2b08 WatchSource:0}: Error finding container a0fdb3bf6ead28d96c1d3a50246e9f445e2817ebd158aded66e729e5972c2b08: Status 404 returned error can't find the container with id a0fdb3bf6ead28d96c1d3a50246e9f445e2817ebd158aded66e729e5972c2b08
Jul 16 20:32:45.713001 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:45.712974    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff556300_6da3_45d7_9415_c5b0f124cd2f.slice/crio-9e40796f3bb680b74c5dc69667463fbf86314519cc59e04f85ccd3b61ca9f8d5 WatchSource:0}: Error finding container 9e40796f3bb680b74c5dc69667463fbf86314519cc59e04f85ccd3b61ca9f8d5: Status 404 returned error can't find the container with id 9e40796f3bb680b74c5dc69667463fbf86314519cc59e04f85ccd3b61ca9f8d5
Jul 16 20:32:45.714507 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:45.714480    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f9d985_30db_4a7b_9f9d_554db3992a61.slice/crio-85640840af8744f4ee1edbce0fd8423cda38807f76f859c60ddb9024a0f8a33c WatchSource:0}: Error finding container 85640840af8744f4ee1edbce0fd8423cda38807f76f859c60ddb9024a0f8a33c: Status 404 returned error can't find the container with id 85640840af8744f4ee1edbce0fd8423cda38807f76f859c60ddb9024a0f8a33c
Jul 16 20:32:45.718859 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:45.716327    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9388c864_76b1_415e_a3d8_6114d5968730.slice/crio-6f419b0588da2b6b364b686f5ed1f2eb7b0880aeca87bebd6655f6950b4f04c0 WatchSource:0}: Error finding container 6f419b0588da2b6b364b686f5ed1f2eb7b0880aeca87bebd6655f6950b4f04c0: Status 404 returned error can't find the container with id 6f419b0588da2b6b364b686f5ed1f2eb7b0880aeca87bebd6655f6950b4f04c0
Jul 16 20:32:45.721328 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:32:45.720895    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51a898b_93f5_4a94_80d0_6aa2f3c19caa.slice/crio-d4cfaa884a38b8aff9f5b41670876759b005a822dd13eacd54c97baca843f048 WatchSource:0}: Error finding container d4cfaa884a38b8aff9f5b41670876759b005a822dd13eacd54c97baca843f048: Status 404 returned error can't find the container with id d4cfaa884a38b8aff9f5b41670876759b005a822dd13eacd54c97baca843f048
Jul 16 20:32:45.951558 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:45.951522    2530 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 20:32:45.994505 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:45.994462    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs\") pod \"network-metrics-daemon-4mmnt\" (UID: \"864a17b7-c612-43d2-923e-8843e0f519b0\") " pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:32:45.994687 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:45.994516    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret\") pod \"global-pull-secret-syncer-tjnr2\" (UID: \"7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e\") " pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:45.994687 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:45.994618    2530 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:32:45.994687 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:45.994619    2530 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:32:45.994687 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:45.994680    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret podName:7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e nodeName:}" failed. No retries permitted until 2026-07-16 20:32:47.994664021 +0000 UTC m=+6.016251767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret") pod "global-pull-secret-syncer-tjnr2" (UID: "7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:32:45.994850 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:45.994695    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs podName:864a17b7-c612-43d2-923e-8843e0f519b0 nodeName:}" failed. No retries permitted until 2026-07-16 20:32:47.994688046 +0000 UTC m=+6.016275791 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs") pod "network-metrics-daemon-4mmnt" (UID: "864a17b7-c612-43d2-923e-8843e0f519b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:32:46.196049 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:46.195945    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fhrc\" (UniqueName: \"kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc\") pod \"network-check-target-vxmfl\" (UID: \"2aed23ed-5e27-4e01-adc6-d8af18c38baa\") " pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:32:46.196220 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:46.196176    2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:32:46.196282 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:46.196226    2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:32:46.196282 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:46.196242    2530 projected.go:196] Error preparing data for projected volume kube-api-access-9fhrc for pod openshift-network-diagnostics/network-check-target-vxmfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:32:46.196354 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:46.196313    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc podName:2aed23ed-5e27-4e01-adc6-d8af18c38baa nodeName:}" failed. No retries permitted until 2026-07-16 20:32:48.196291726 +0000 UTC m=+6.217879497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9fhrc" (UniqueName: "kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc") pod "network-check-target-vxmfl" (UID: "2aed23ed-5e27-4e01-adc6-d8af18c38baa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:32:46.414928 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:46.414856    2530 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 20:27:44 +0000 UTC" deadline="2028-04-26 14:29:54.165818558 +0000 UTC"
Jul 16 20:32:46.414928 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:46.414926    2530 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15593h57m7.750897618s"
Jul 16 20:32:46.505725 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:46.505559    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:32:46.505903 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:46.505724    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4mmnt" podUID="864a17b7-c612-43d2-923e-8843e0f519b0"
Jul 16 20:32:46.506163 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:46.506141    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:32:46.506303 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:46.506276    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxmfl" podUID="2aed23ed-5e27-4e01-adc6-d8af18c38baa"
Jul 16 20:32:46.525951 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:46.525853    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-764h9" event={"ID":"9388c864-76b1-415e-a3d8-6114d5968730","Type":"ContainerStarted","Data":"6f419b0588da2b6b364b686f5ed1f2eb7b0880aeca87bebd6655f6950b4f04c0"}
Jul 16 20:32:46.536202 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:46.536085    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9th7p" event={"ID":"03510e23-51da-4185-9da4-f908281932e6","Type":"ContainerStarted","Data":"a0fdb3bf6ead28d96c1d3a50246e9f445e2817ebd158aded66e729e5972c2b08"}
Jul 16 20:32:46.543490 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:46.543441    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cwld6" event={"ID":"4288933c-4329-4853-8fb3-e3d46c34479f","Type":"ContainerStarted","Data":"fa5a27dceb4ac63dc81f7dda74797c3ef729b54aa1531d9c32995a9c33650751"}
Jul 16 20:32:46.554569 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:46.554521    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jlmxh" event={"ID":"efdaaf6e-f70a-4b31-9bda-0104eb2ef52f","Type":"ContainerStarted","Data":"b20d0f3901aefbfc93032368201d28cb5038688594e5b1d35835a40f72c4640c"}
Jul 16 20:32:46.562293 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:46.562252    2530 generic.go:356] "Generic (PLEG): container finished" podID="7535ca3b2d933dbcbdc547452fd525db" containerID="5062684b3139d154402a311ae1229d8737f4500c6ef1379ed14981f3c9d628ae" exitCode=0
Jul 16 20:32:46.562469 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:46.562371    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal" event={"ID":"7535ca3b2d933dbcbdc547452fd525db","Type":"ContainerDied","Data":"5062684b3139d154402a311ae1229d8737f4500c6ef1379ed14981f3c9d628ae"}
Jul 16 20:32:46.567279 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:46.567244    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-blwng" event={"ID":"30f9d985-30db-4a7b-9f9d-554db3992a61","Type":"ContainerStarted","Data":"85640840af8744f4ee1edbce0fd8423cda38807f76f859c60ddb9024a0f8a33c"}
Jul 16 20:32:46.569745 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:46.569624    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hcx7t" event={"ID":"ff556300-6da3-45d7-9415-c5b0f124cd2f","Type":"ContainerStarted","Data":"9e40796f3bb680b74c5dc69667463fbf86314519cc59e04f85ccd3b61ca9f8d5"}
Jul 16 20:32:46.571689 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:46.571657    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nf9hg" event={"ID":"205ec3c3-fd73-4c63-9298-2fe9b3f8a625","Type":"ContainerStarted","Data":"12f017b3a222ca179faaf36d94f42946169f0161f446031bfcbc399bb9069472"}
Jul 16 20:32:46.575938 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:46.575901    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.ec2.internal" event={"ID":"19742d85a0eb69c9b3078865a8f5b5d4","Type":"ContainerStarted","Data":"60b7e5221672eaf866d47436b4185e76fbcda994e55ca57b83c89691ffd78e5f"}
Jul 16 20:32:46.587404 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:46.587359    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b" event={"ID":"c51a898b-93f5-4a94-80d0-6aa2f3c19caa","Type":"ContainerStarted","Data":"d4cfaa884a38b8aff9f5b41670876759b005a822dd13eacd54c97baca843f048"}
Jul 16 20:32:46.595002 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:46.594957    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xttrm" event={"ID":"586e85c8-fa49-4b8a-bc17-1d57c31b1c79","Type":"ContainerStarted","Data":"ecf529f5e1214765bb068aa0e40b01b9ce1320f294b7265b26043b96529a9359"}
Jul 16 20:32:47.502093 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:47.502054    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:47.502624 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:47.502229    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:32:47.628042 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:47.627926    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal" event={"ID":"7535ca3b2d933dbcbdc547452fd525db","Type":"ContainerStarted","Data":"3f274e8cd6c6b8b5eaebfe64d42ccd5f5b5b99d59b3713a7fe8f79dbae1a941a"}
Jul 16 20:32:47.645216 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:47.644101    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.ec2.internal" podStartSLOduration=3.644068691 podStartE2EDuration="3.644068691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:32:46.594496223 +0000 UTC m=+4.616083992" watchObservedRunningTime="2026-07-16 20:32:47.644068691 +0000 UTC m=+5.665656461"
Jul 16 20:32:48.012643 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:48.012601    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs\") pod \"network-metrics-daemon-4mmnt\" (UID: \"864a17b7-c612-43d2-923e-8843e0f519b0\") " pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:32:48.012845 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:48.012664    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret\") pod \"global-pull-secret-syncer-tjnr2\" (UID: \"7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e\") " pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:48.012845 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:48.012815    2530 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:32:48.012969 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:48.012871    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret podName:7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e nodeName:}" failed. No retries permitted until 2026-07-16 20:32:52.012856861 +0000 UTC m=+10.034444628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret") pod "global-pull-secret-syncer-tjnr2" (UID: "7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:32:48.013406 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:48.013292    2530 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:32:48.013406 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:48.013351    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs podName:864a17b7-c612-43d2-923e-8843e0f519b0 nodeName:}" failed. No retries permitted until 2026-07-16 20:32:52.013335224 +0000 UTC m=+10.034922971 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs") pod "network-metrics-daemon-4mmnt" (UID: "864a17b7-c612-43d2-923e-8843e0f519b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:32:48.214900 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:48.214855    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fhrc\" (UniqueName: \"kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc\") pod \"network-check-target-vxmfl\" (UID: \"2aed23ed-5e27-4e01-adc6-d8af18c38baa\") " pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:32:48.215651 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:48.215171    2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:32:48.215651 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:48.215203    2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:32:48.215651 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:48.215218    2530 projected.go:196] Error preparing data for projected volume kube-api-access-9fhrc for pod openshift-network-diagnostics/network-check-target-vxmfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:32:48.215651 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:48.215288    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc podName:2aed23ed-5e27-4e01-adc6-d8af18c38baa nodeName:}" failed. No retries permitted until 2026-07-16 20:32:52.215267971 +0000 UTC m=+10.236855721 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9fhrc" (UniqueName: "kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc") pod "network-check-target-vxmfl" (UID: "2aed23ed-5e27-4e01-adc6-d8af18c38baa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:32:48.505055 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:48.504313    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:32:48.505055 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:48.504532    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxmfl" podUID="2aed23ed-5e27-4e01-adc6-d8af18c38baa"
Jul 16 20:32:48.505603 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:48.505172    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:32:48.505603 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:48.505316    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4mmnt" podUID="864a17b7-c612-43d2-923e-8843e0f519b0"
Jul 16 20:32:49.503272 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:49.502617    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:49.503272 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:49.502769    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:32:50.503258 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:50.503222    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:32:50.503848 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:50.503377    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4mmnt" podUID="864a17b7-c612-43d2-923e-8843e0f519b0"
Jul 16 20:32:50.505634 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:50.505480    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:32:50.505634 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:50.505593    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxmfl" podUID="2aed23ed-5e27-4e01-adc6-d8af18c38baa"
Jul 16 20:32:51.502002 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:51.501910    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:51.502190 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:51.502054    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:32:52.047342 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:52.047299    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret\") pod \"global-pull-secret-syncer-tjnr2\" (UID: \"7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e\") " pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:52.047739 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:52.047377    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs\") pod \"network-metrics-daemon-4mmnt\" (UID: \"864a17b7-c612-43d2-923e-8843e0f519b0\") " pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:32:52.047739 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:52.047480    2530 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:32:52.047739 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:52.047536    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs podName:864a17b7-c612-43d2-923e-8843e0f519b0 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:00.047516489 +0000 UTC m=+18.069104234 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs") pod "network-metrics-daemon-4mmnt" (UID: "864a17b7-c612-43d2-923e-8843e0f519b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:32:52.047860 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:52.047850    2530 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:32:52.047901 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:52.047883    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret podName:7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e nodeName:}" failed. No retries permitted until 2026-07-16 20:33:00.04787343 +0000 UTC m=+18.069461174 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret") pod "global-pull-secret-syncer-tjnr2" (UID: "7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:32:52.248827 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:52.248785    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fhrc\" (UniqueName: \"kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc\") pod \"network-check-target-vxmfl\" (UID: \"2aed23ed-5e27-4e01-adc6-d8af18c38baa\") " pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:32:52.249025 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:52.249000    2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:32:52.249139 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:52.249035    2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:32:52.249139 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:52.249050    2530 projected.go:196] Error preparing data for projected volume kube-api-access-9fhrc for pod openshift-network-diagnostics/network-check-target-vxmfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:32:52.249139 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:52.249136    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc podName:2aed23ed-5e27-4e01-adc6-d8af18c38baa nodeName:}" failed. No retries permitted until 2026-07-16 20:33:00.24911489 +0000 UTC m=+18.270702651 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9fhrc" (UniqueName: "kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc") pod "network-check-target-vxmfl" (UID: "2aed23ed-5e27-4e01-adc6-d8af18c38baa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:32:52.503283 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:52.503182    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:32:52.503464 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:52.503313    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxmfl" podUID="2aed23ed-5e27-4e01-adc6-d8af18c38baa"
Jul 16 20:32:52.503464 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:52.503355    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:32:52.503464 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:52.503448    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4mmnt" podUID="864a17b7-c612-43d2-923e-8843e0f519b0"
Jul 16 20:32:53.502408 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:53.502369    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:53.502895 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:53.502505    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:32:54.502000 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:54.501963    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:32:54.502196 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:54.502017    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:32:54.502196 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:54.502129    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxmfl" podUID="2aed23ed-5e27-4e01-adc6-d8af18c38baa"
Jul 16 20:32:54.502335 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:54.502216    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4mmnt" podUID="864a17b7-c612-43d2-923e-8843e0f519b0"
Jul 16 20:32:55.501827 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:55.501786    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:55.502328 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:55.501924    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:32:56.502451 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:56.502412    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:32:56.502940 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:56.502416    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:32:56.502940 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:56.502565    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4mmnt" podUID="864a17b7-c612-43d2-923e-8843e0f519b0"
Jul 16 20:32:56.502940 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:56.502643    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxmfl" podUID="2aed23ed-5e27-4e01-adc6-d8af18c38baa"
Jul 16 20:32:57.501896 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:57.501858    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:57.502075 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:57.501999    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:32:58.502257 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:58.502223    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:32:58.502691 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:58.502224    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:32:58.502691 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:58.502351    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxmfl" podUID="2aed23ed-5e27-4e01-adc6-d8af18c38baa"
Jul 16 20:32:58.502691 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:58.502433    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4mmnt" podUID="864a17b7-c612-43d2-923e-8843e0f519b0"
Jul 16 20:32:59.501999 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:32:59.501960    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:32:59.502327 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:32:59.502115    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:33:00.110020 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:00.109983    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs\") pod \"network-metrics-daemon-4mmnt\" (UID: \"864a17b7-c612-43d2-923e-8843e0f519b0\") " pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:33:00.110232 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:00.110032    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret\") pod \"global-pull-secret-syncer-tjnr2\" (UID: \"7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e\") " pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:33:00.110232 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:00.110188    2530 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:33:00.110232 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:00.110188    2530 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:33:00.110374 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:00.110258    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret podName:7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e nodeName:}" failed. No retries permitted until 2026-07-16 20:33:16.110239565 +0000 UTC m=+34.131827322 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret") pod "global-pull-secret-syncer-tjnr2" (UID: "7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:33:00.110374 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:00.110280    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs podName:864a17b7-c612-43d2-923e-8843e0f519b0 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:16.110269787 +0000 UTC m=+34.131857537 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs") pod "network-metrics-daemon-4mmnt" (UID: "864a17b7-c612-43d2-923e-8843e0f519b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:33:00.311367 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:00.311315    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fhrc\" (UniqueName: \"kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc\") pod \"network-check-target-vxmfl\" (UID: \"2aed23ed-5e27-4e01-adc6-d8af18c38baa\") " pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:33:00.311564 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:00.311483    2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:33:00.311564 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:00.311510    2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:33:00.311564 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:00.311520    2530 projected.go:196] Error preparing data for projected volume kube-api-access-9fhrc for pod openshift-network-diagnostics/network-check-target-vxmfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:33:00.311742 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:00.311580    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc podName:2aed23ed-5e27-4e01-adc6-d8af18c38baa nodeName:}" failed. No retries permitted until 2026-07-16 20:33:16.311562128 +0000 UTC m=+34.333149893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9fhrc" (UniqueName: "kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc") pod "network-check-target-vxmfl" (UID: "2aed23ed-5e27-4e01-adc6-d8af18c38baa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:33:00.502608 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:00.502567    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:33:00.503058 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:00.502622    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:33:00.503058 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:00.502725    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4mmnt" podUID="864a17b7-c612-43d2-923e-8843e0f519b0"
Jul 16 20:33:00.503058 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:00.502865    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxmfl" podUID="2aed23ed-5e27-4e01-adc6-d8af18c38baa"
Jul 16 20:33:01.502143 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:01.502090    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:33:01.502352 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:01.502227    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:33:02.503071 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:02.503034    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:33:02.503554 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:02.503170    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4mmnt" podUID="864a17b7-c612-43d2-923e-8843e0f519b0"
Jul 16 20:33:02.503554 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:02.503223    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:33:02.503554 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:02.503350    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxmfl" podUID="2aed23ed-5e27-4e01-adc6-d8af18c38baa"
Jul 16 20:33:03.502533 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:03.502497    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:33:03.502704 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:03.502623    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:33:04.501711 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:04.501668    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:33:04.502266 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:04.501787    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4mmnt" podUID="864a17b7-c612-43d2-923e-8843e0f519b0"
Jul 16 20:33:04.502266 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:04.501799    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:33:04.502266 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:04.501903    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxmfl" podUID="2aed23ed-5e27-4e01-adc6-d8af18c38baa"
Jul 16 20:33:05.502787 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.502504    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:33:05.503468 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:05.502879    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:33:05.669240 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.669149    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-blwng" event={"ID":"30f9d985-30db-4a7b-9f9d-554db3992a61","Type":"ContainerStarted","Data":"c900bbb7f049d262248a19b9ee6f022f8aaf02f66803cd40d6ea4975c843f946"}
Jul 16 20:33:05.669536 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.669508    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-blwng" event={"ID":"30f9d985-30db-4a7b-9f9d-554db3992a61","Type":"ContainerStarted","Data":"8e72fb0a05b43ceceddd54ebea710bc5f4ccc6cbdc9f0ed9797cf3dc0fdd2dce"}
Jul 16 20:33:05.669645 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.669573    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-blwng" event={"ID":"30f9d985-30db-4a7b-9f9d-554db3992a61","Type":"ContainerStarted","Data":"0ab47343d2ff6a28bece0e56e15a29d97009dddd889c8ea2c5d2a0a4499cd4f8"}
Jul 16 20:33:05.669645 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.669624    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-blwng" event={"ID":"30f9d985-30db-4a7b-9f9d-554db3992a61","Type":"ContainerStarted","Data":"bad9922a0451a6e33cf27616a8ba589cb492f098b4e5b16130b1fa1274fa6215"}
Jul 16 20:33:05.669736 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.669664    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-blwng" event={"ID":"30f9d985-30db-4a7b-9f9d-554db3992a61","Type":"ContainerStarted","Data":"acf259674469be402e9f858406e14dcbf986c547224b7e6f0fc01b17f857d177"}
Jul 16 20:33:05.669736 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.669703    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-blwng" event={"ID":"30f9d985-30db-4a7b-9f9d-554db3992a61","Type":"ContainerStarted","Data":"cd6b0bb7befe5c7149f3752cb59ab1dfa9036229ac450e4bfd3ba37f0cce32f4"}
Jul 16 20:33:05.670802 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.670777    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hcx7t" event={"ID":"ff556300-6da3-45d7-9415-c5b0f124cd2f","Type":"ContainerStarted","Data":"f048a2a39538fd5c8ee6e8c70f9bac06841f8550b2cf81ae4a9fe03fa48dadfb"}
Jul 16 20:33:05.672828 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.672807    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nf9hg" event={"ID":"205ec3c3-fd73-4c63-9298-2fe9b3f8a625","Type":"ContainerStarted","Data":"ed7e70c93b068177164738ca2ef176fba0b00212678cdd8c664e4b767a55e28a"}
Jul 16 20:33:05.674520 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.674496    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b" event={"ID":"c51a898b-93f5-4a94-80d0-6aa2f3c19caa","Type":"ContainerStarted","Data":"47cf81efb909dab76dfeab014bb532779bb19309953c78a709cd606d5fc1a539"}
Jul 16 20:33:05.676468 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.676448    2530 generic.go:356] "Generic (PLEG): container finished" podID="586e85c8-fa49-4b8a-bc17-1d57c31b1c79" containerID="dfd4067e725a1f42e8ffeb2c2cc068470776b491d3f36b9cdc3883b69a8e235c" exitCode=0
Jul 16 20:33:05.676706 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.676658    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xttrm" event={"ID":"586e85c8-fa49-4b8a-bc17-1d57c31b1c79","Type":"ContainerDied","Data":"dfd4067e725a1f42e8ffeb2c2cc068470776b491d3f36b9cdc3883b69a8e235c"}
Jul 16 20:33:05.678479 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.678443    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-764h9" event={"ID":"9388c864-76b1-415e-a3d8-6114d5968730","Type":"ContainerStarted","Data":"677e29d0c6ef7add3a827a2d0917028f2c3e5d0207ae90eea8442eb0c7ffefe0"}
Jul 16 20:33:05.680284 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.680259    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9th7p" event={"ID":"03510e23-51da-4185-9da4-f908281932e6","Type":"ContainerStarted","Data":"28d8d44bcf6622588a70f13d03b4d9f7aa4a1864365d49e7977b78a3ab146107"}
Jul 16 20:33:05.683129 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.683094    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cwld6" event={"ID":"4288933c-4329-4853-8fb3-e3d46c34479f","Type":"ContainerStarted","Data":"a209434379dac7f7fe518f9ddc0a0ef99ecda335e37137aaede520c41a5e8851"}
Jul 16 20:33:05.686849 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.686794    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.ec2.internal" podStartSLOduration=21.686782775 podStartE2EDuration="21.686782775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:32:47.653539356 +0000 UTC m=+5.675127126" watchObservedRunningTime="2026-07-16 20:33:05.686782775 +0000 UTC m=+23.708370540"
Jul 16 20:33:05.687594 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.687543    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hcx7t" podStartSLOduration=3.478465343 podStartE2EDuration="22.687535357s" totalImagesPullingTime="19.209070014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:05.686676337 +0000 UTC m=+23.708264123" watchObservedRunningTime="2026-07-16 20:33:05.687535357 +0000 UTC m=+23.709123124"
Jul 16 20:33:05.732321 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.732271    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nf9hg" podStartSLOduration=3.5794582139999997 podStartE2EDuration="22.732255058s" totalImagesPullingTime="19.152796844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:05.719632201 +0000 UTC m=+23.741219968" watchObservedRunningTime="2026-07-16 20:33:05.732255058 +0000 UTC m=+23.753842824"
Jul 16 20:33:05.732544 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.732470    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cwld6" podStartSLOduration=3.579674054 podStartE2EDuration="22.732461s" totalImagesPullingTime="19.152786946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:05.731810921 +0000 UTC m=+23.753398689" watchObservedRunningTime="2026-07-16 20:33:05.732461 +0000 UTC m=+23.754048768"
Jul 16 20:33:05.747224 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.747176    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-764h9" podStartSLOduration=3.582710359 podStartE2EDuration="22.747162402s" totalImagesPullingTime="19.164452043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:05.746723472 +0000 UTC m=+23.768311327" watchObservedRunningTime="2026-07-16 20:33:05.747162402 +0000 UTC m=+23.768750169"
Jul 16 20:33:05.763309 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:05.763265    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9th7p" podStartSLOduration=3.614913245 podStartE2EDuration="22.763251604s" totalImagesPullingTime="19.148338359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:05.763152486 +0000 UTC m=+23.784740249" watchObservedRunningTime="2026-07-16 20:33:05.763251604 +0000 UTC m=+23.784839369"
Jul 16 20:33:06.457015 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:06.456616    2530 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 20:33:06.502472 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:06.502393    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:33:06.502648 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:06.502393    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:33:06.502648 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:06.502524    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxmfl" podUID="2aed23ed-5e27-4e01-adc6-d8af18c38baa"
Jul 16 20:33:06.502648 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:06.502584    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4mmnt" podUID="864a17b7-c612-43d2-923e-8843e0f519b0"
Jul 16 20:33:06.687126 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:06.687060    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jlmxh" event={"ID":"efdaaf6e-f70a-4b31-9bda-0104eb2ef52f","Type":"ContainerStarted","Data":"84a5a423496cccc4c63d5e60f9c14534992bdd96f9e8b069a112ba1e74f77bfc"}
Jul 16 20:33:06.688945 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:06.688915    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b" event={"ID":"c51a898b-93f5-4a94-80d0-6aa2f3c19caa","Type":"ContainerStarted","Data":"b73d4111a751c26b74b29b0d417616d64bd1453d5fdc711e92315a119d594484"}
Jul 16 20:33:06.701201 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:06.701150    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jlmxh" podStartSLOduration=4.521717995 podStartE2EDuration="23.701133207s" totalImagesPullingTime="19.179415212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:06.700730839 +0000 UTC m=+24.722318605" watchObservedRunningTime="2026-07-16 20:33:06.701133207 +0000 UTC m=+24.722720969"
Jul 16 20:33:07.445699 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:07.445580    2530 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T20:33:06.456636184Z","UUID":"557dc28b-dc5f-4922-99f9-beec2a53abf5","Handler":null,"Name":"","Endpoint":""}
Jul 16 20:33:07.447305 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:07.447285    2530 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 20:33:07.447435 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:07.447329    2530 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 20:33:07.502362 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:07.502329    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:33:07.502561 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:07.502451    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:33:08.201092 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:08.201005    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nf9hg"
Jul 16 20:33:08.501871 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:08.501825    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:33:08.502066 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:08.501958    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxmfl" podUID="2aed23ed-5e27-4e01-adc6-d8af18c38baa"
Jul 16 20:33:08.502066 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:08.502008    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:33:08.502206 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:08.502126    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4mmnt" podUID="864a17b7-c612-43d2-923e-8843e0f519b0"
Jul 16 20:33:08.698337 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:08.698300    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-blwng" event={"ID":"30f9d985-30db-4a7b-9f9d-554db3992a61","Type":"ContainerStarted","Data":"1171d885e748a048f27c9c1717dcd0e8714adaf24fc14d44d91af101d328f90a"}
Jul 16 20:33:08.700660 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:08.700626    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b" event={"ID":"c51a898b-93f5-4a94-80d0-6aa2f3c19caa","Type":"ContainerStarted","Data":"65d6063ade08fa652cb6a579ac65e52a040ea379e278b448283ff1b33837fb1c"}
Jul 16 20:33:08.716904 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:08.716820    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2m28b" podStartSLOduration=4.204450034 podStartE2EDuration="25.716798621s" totalImagesPullingTime="21.512348587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:32:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:08.715525862 +0000 UTC m=+26.737113630" watchObservedRunningTime="2026-07-16 20:33:08.716798621 +0000 UTC m=+26.738386388"
Jul 16 20:33:09.502010 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:09.501820    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:33:09.502446 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:09.502131    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:33:09.695223 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:09.695186    2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nf9hg"
Jul 16 20:33:09.696298 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:09.696268    2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nf9hg"
Jul 16 20:33:09.704087 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:09.704062    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nf9hg"
Jul 16 20:33:10.501814 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:10.501785    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:33:10.501966 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:10.501899    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxmfl" podUID="2aed23ed-5e27-4e01-adc6-d8af18c38baa"
Jul 16 20:33:10.501966 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:10.501949    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:33:10.502676 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:10.502013    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4mmnt" podUID="864a17b7-c612-43d2-923e-8843e0f519b0"
Jul 16 20:33:10.707171 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:10.707136    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xttrm" event={"ID":"586e85c8-fa49-4b8a-bc17-1d57c31b1c79","Type":"ContainerStarted","Data":"02f25500aa2276f7ef00ebd77ffc13a0685c78ebc2045e9469d0f6254b68a695"}
Jul 16 20:33:10.710621 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:10.710590    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-blwng" event={"ID":"30f9d985-30db-4a7b-9f9d-554db3992a61","Type":"ContainerStarted","Data":"d86d4a268b312e59067a6dadd3df8b0b8c75d174139f18506fbc2c915fc30671"}
Jul 16 20:33:10.754779 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:10.754729    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-blwng" podStartSLOduration=8.515688317 podStartE2EDuration="27.754712425s" totalImagesPullingTime="19.239024108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:10.75400491 +0000 UTC m=+28.775592687" watchObservedRunningTime="2026-07-16 20:33:10.754712425 +0000 UTC m=+28.776300226"
Jul 16 20:33:11.502094 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:11.501909    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:33:11.502634 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:11.502183    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:33:11.715092 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:11.715056    2530 generic.go:356] "Generic (PLEG): container finished" podID="586e85c8-fa49-4b8a-bc17-1d57c31b1c79" containerID="02f25500aa2276f7ef00ebd77ffc13a0685c78ebc2045e9469d0f6254b68a695" exitCode=0
Jul 16 20:33:11.715359 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:11.715139    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xttrm" event={"ID":"586e85c8-fa49-4b8a-bc17-1d57c31b1c79","Type":"ContainerDied","Data":"02f25500aa2276f7ef00ebd77ffc13a0685c78ebc2045e9469d0f6254b68a695"}
Jul 16 20:33:11.715759 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:11.715740    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:33:11.715870 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:11.715794    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:33:11.716434 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:11.715874    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:33:11.733300 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:11.733270    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:33:11.733461 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:11.733398    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:33:12.360150 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:12.360117    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tjnr2"]
Jul 16 20:33:12.360357 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:12.360267    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:33:12.360415 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:12.360369    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:33:12.362776 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:12.362743    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vxmfl"]
Jul 16 20:33:12.362950 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:12.362803    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4mmnt"]
Jul 16 20:33:12.362950 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:12.362902    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:33:12.363061 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:12.363000    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4mmnt" podUID="864a17b7-c612-43d2-923e-8843e0f519b0"
Jul 16 20:33:12.363139 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:12.363064    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:33:12.363139 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:12.363132    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxmfl" podUID="2aed23ed-5e27-4e01-adc6-d8af18c38baa"
Jul 16 20:33:13.721742 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:13.721702    2530 generic.go:356] "Generic (PLEG): container finished" podID="586e85c8-fa49-4b8a-bc17-1d57c31b1c79" containerID="956d832eb308516840608e58b0268d941bb9c1d47e56e1a77797afd18d30096b" exitCode=0
Jul 16 20:33:13.722146 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:13.721784    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xttrm" event={"ID":"586e85c8-fa49-4b8a-bc17-1d57c31b1c79","Type":"ContainerDied","Data":"956d832eb308516840608e58b0268d941bb9c1d47e56e1a77797afd18d30096b"}
Jul 16 20:33:14.502135 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:14.501885    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:33:14.502245 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:14.501885    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:33:14.502287 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:14.501886    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:33:14.502287 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:14.502263    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4mmnt" podUID="864a17b7-c612-43d2-923e-8843e0f519b0"
Jul 16 20:33:14.502372 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:14.502339    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:33:14.502450 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:14.502421    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxmfl" podUID="2aed23ed-5e27-4e01-adc6-d8af18c38baa"
Jul 16 20:33:14.726521 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:14.726486    2530 generic.go:356] "Generic (PLEG): container finished" podID="586e85c8-fa49-4b8a-bc17-1d57c31b1c79" containerID="9ba148c4d23f250149ae230073cd31dcd37930bcaaba8644f5ef0b8dd439a1a2" exitCode=0
Jul 16 20:33:14.726875 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:14.726545    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xttrm" event={"ID":"586e85c8-fa49-4b8a-bc17-1d57c31b1c79","Type":"ContainerDied","Data":"9ba148c4d23f250149ae230073cd31dcd37930bcaaba8644f5ef0b8dd439a1a2"}
Jul 16 20:33:15.585095 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:15.585063    2530 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nf9hg_205ec3c3-fd73-4c63-9298-2fe9b3f8a625/konnectivity-agent/0.log" limit=1024
Jul 16 20:33:15.675919 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:15.675882    2530 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nf9hg_205ec3c3-fd73-4c63-9298-2fe9b3f8a625/konnectivity-agent/0.log" limit=1024
Jul 16 20:33:16.128047 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:16.128014    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs\") pod \"network-metrics-daemon-4mmnt\" (UID: \"864a17b7-c612-43d2-923e-8843e0f519b0\") " pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:33:16.128572 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:16.128068    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret\") pod \"global-pull-secret-syncer-tjnr2\" (UID: \"7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e\") " pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:33:16.128572 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:16.128212    2530 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:33:16.128572 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:16.128222    2530 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 20:33:16.128572 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:16.128288    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs podName:864a17b7-c612-43d2-923e-8843e0f519b0 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:48.128266639 +0000 UTC m=+66.149854397 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs") pod "network-metrics-daemon-4mmnt" (UID: "864a17b7-c612-43d2-923e-8843e0f519b0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 20:33:16.128572 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:16.128309    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret podName:7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e nodeName:}" failed. No retries permitted until 2026-07-16 20:33:48.128298911 +0000 UTC m=+66.149886667 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret") pod "global-pull-secret-syncer-tjnr2" (UID: "7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e") : object "kube-system"/"original-pull-secret" not registered
Jul 16 20:33:16.329243 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:16.329209    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fhrc\" (UniqueName: \"kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc\") pod \"network-check-target-vxmfl\" (UID: \"2aed23ed-5e27-4e01-adc6-d8af18c38baa\") " pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:33:16.329441 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:16.329396    2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 20:33:16.329441 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:16.329425    2530 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 20:33:16.329441 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:16.329436    2530 projected.go:196] Error preparing data for projected volume kube-api-access-9fhrc for pod openshift-network-diagnostics/network-check-target-vxmfl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:33:16.329582 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:16.329483    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc podName:2aed23ed-5e27-4e01-adc6-d8af18c38baa nodeName:}" failed. No retries permitted until 2026-07-16 20:33:48.329469864 +0000 UTC m=+66.351057622 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9fhrc" (UniqueName: "kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc") pod "network-check-target-vxmfl" (UID: "2aed23ed-5e27-4e01-adc6-d8af18c38baa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 20:33:16.482155 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:16.482117    2530 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nf9hg_205ec3c3-fd73-4c63-9298-2fe9b3f8a625/konnectivity-agent/0.log" limit=1024
Jul 16 20:33:16.502436 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:16.502403    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:33:16.502436 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:16.502439    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:33:16.502667 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:16.502403    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:33:16.502667 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:16.502526    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:33:16.502667 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:16.502586    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxmfl" podUID="2aed23ed-5e27-4e01-adc6-d8af18c38baa"
Jul 16 20:33:16.502825 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:16.502665    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4mmnt" podUID="864a17b7-c612-43d2-923e-8843e0f519b0"
Jul 16 20:33:16.583579 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:16.583546    2530 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nf9hg_205ec3c3-fd73-4c63-9298-2fe9b3f8a625/konnectivity-agent/0.log" limit=1024
Jul 16 20:33:18.501961 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.501920    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:33:18.502508 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.501940    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:33:18.502508 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:18.502193    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxmfl" podUID="2aed23ed-5e27-4e01-adc6-d8af18c38baa"
Jul 16 20:33:18.502508 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:18.502061    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4mmnt" podUID="864a17b7-c612-43d2-923e-8843e0f519b0"
Jul 16 20:33:18.502508 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.501920    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:33:18.502508 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:18.502285    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tjnr2" podUID="7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e"
Jul 16 20:33:18.830518 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.830491    2530 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-249.ec2.internal" event="NodeReady"
Jul 16 20:33:18.830675 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.830644    2530 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 20:33:18.866780 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.866749    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-869d7cdb67-bzsnl"]
Jul 16 20:33:18.870663 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.870638    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:18.873381 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.873356    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 16 20:33:18.873381 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.873377    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-68jbx\""
Jul 16 20:33:18.873587 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.873575    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 16 20:33:18.873628 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.873594    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 16 20:33:18.878194 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.878175    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 16 20:33:18.885053 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.884195    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x7pgr"]
Jul 16 20:33:18.888068 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.888046    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-869d7cdb67-bzsnl"]
Jul 16 20:33:18.888068 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.888084    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hmjxf"]
Jul 16 20:33:18.888267 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.888210    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:18.890556 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.890534    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 20:33:18.890666 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.890567    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xlvk5\""
Jul 16 20:33:18.890666 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.890605    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 20:33:18.890971 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.890951    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hmjxf"
Jul 16 20:33:18.893168 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.893150    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dt299\""
Jul 16 20:33:18.893326 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.893309    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 20:33:18.893381 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.893310    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 20:33:18.893381 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.893353    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 20:33:18.896521 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.896500    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x7pgr"]
Jul 16 20:33:18.900984 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.900821    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hmjxf"]
Jul 16 20:33:18.950587 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.950547    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:18.950587 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.950587    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0705ca3b-1ea6-4263-a444-9618d29c2701-ca-trust-extracted\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:18.950788 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.950617    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kf4k\" (UniqueName: \"kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-kube-api-access-8kf4k\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:18.950788 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.950650    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-bound-sa-token\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:18.950788 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.950668    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-certificates\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:18.950788 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.950760    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0705ca3b-1ea6-4263-a444-9618d29c2701-trusted-ca\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:18.951369 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.950898    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0705ca3b-1ea6-4263-a444-9618d29c2701-image-registry-private-configuration\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:18.951459 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:18.951404    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0705ca3b-1ea6-4263-a444-9618d29c2701-installation-pull-secrets\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:19.002084 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.002044    2530 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nf9hg_205ec3c3-fd73-4c63-9298-2fe9b3f8a625/konnectivity-agent/0.log" limit=1024
Jul 16 20:33:19.052139 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.052077    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0705ca3b-1ea6-4263-a444-9618d29c2701-trusted-ca\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:19.052301 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.052187    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzr6v\" (UniqueName: \"kubernetes.io/projected/3589675b-514d-46eb-b879-9c66f98576ae-kube-api-access-zzr6v\") pod \"dns-default-x7pgr\" (UID: \"3589675b-514d-46eb-b879-9c66f98576ae\") " pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:19.052301 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.052208    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a949c15b-79d3-42e2-ba6f-aaa345dd0bcf-cert\") pod \"ingress-canary-hmjxf\" (UID: \"a949c15b-79d3-42e2-ba6f-aaa345dd0bcf\") " pod="openshift-ingress-canary/ingress-canary-hmjxf"
Jul 16 20:33:19.052301 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.052238    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44zfk\" (UniqueName: \"kubernetes.io/projected/a949c15b-79d3-42e2-ba6f-aaa345dd0bcf-kube-api-access-44zfk\") pod \"ingress-canary-hmjxf\" (UID: \"a949c15b-79d3-42e2-ba6f-aaa345dd0bcf\") " pod="openshift-ingress-canary/ingress-canary-hmjxf"
Jul 16 20:33:19.052301 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.052282    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0705ca3b-1ea6-4263-a444-9618d29c2701-image-registry-private-configuration\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:19.052492 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.052334    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0705ca3b-1ea6-4263-a444-9618d29c2701-installation-pull-secrets\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:19.052492 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.052435    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:19.052492 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.052476    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0705ca3b-1ea6-4263-a444-9618d29c2701-ca-trust-extracted\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:19.052645 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.052503    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kf4k\" (UniqueName: \"kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-kube-api-access-8kf4k\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:19.052645 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.052539    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3589675b-514d-46eb-b879-9c66f98576ae-config-volume\") pod \"dns-default-x7pgr\" (UID: \"3589675b-514d-46eb-b879-9c66f98576ae\") " pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:19.052645 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:19.052559    2530 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 20:33:19.052645 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:19.052574    2530 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869d7cdb67-bzsnl: secret "image-registry-tls" not found
Jul 16 20:33:19.052645 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:19.052630    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls podName:0705ca3b-1ea6-4263-a444-9618d29c2701 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:19.55261199 +0000 UTC m=+37.574199734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls") pod "image-registry-869d7cdb67-bzsnl" (UID: "0705ca3b-1ea6-4263-a444-9618d29c2701") : secret "image-registry-tls" not found
Jul 16 20:33:19.052885 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.052571    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-bound-sa-token\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:19.052885 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.052738    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-certificates\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:19.052885 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.052767    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3589675b-514d-46eb-b879-9c66f98576ae-tmp-dir\") pod \"dns-default-x7pgr\" (UID: \"3589675b-514d-46eb-b879-9c66f98576ae\") " pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:19.052885 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.052795    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3589675b-514d-46eb-b879-9c66f98576ae-metrics-tls\") pod \"dns-default-x7pgr\" (UID: \"3589675b-514d-46eb-b879-9c66f98576ae\") " pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:19.052885 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.052798    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0705ca3b-1ea6-4263-a444-9618d29c2701-ca-trust-extracted\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:19.053180 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.053160    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0705ca3b-1ea6-4263-a444-9618d29c2701-trusted-ca\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:19.053254 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.053234    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-certificates\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:19.057666 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.057607    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0705ca3b-1ea6-4263-a444-9618d29c2701-image-registry-private-configuration\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:19.057666 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.057608    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0705ca3b-1ea6-4263-a444-9618d29c2701-installation-pull-secrets\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:19.060809 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.060785    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-bound-sa-token\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:19.061673 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.061647    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kf4k\" (UniqueName: \"kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-kube-api-access-8kf4k\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:19.153780 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.153742    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzr6v\" (UniqueName: \"kubernetes.io/projected/3589675b-514d-46eb-b879-9c66f98576ae-kube-api-access-zzr6v\") pod \"dns-default-x7pgr\" (UID: \"3589675b-514d-46eb-b879-9c66f98576ae\") " pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:19.153780 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.153780    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a949c15b-79d3-42e2-ba6f-aaa345dd0bcf-cert\") pod \"ingress-canary-hmjxf\" (UID: \"a949c15b-79d3-42e2-ba6f-aaa345dd0bcf\") " pod="openshift-ingress-canary/ingress-canary-hmjxf"
Jul 16 20:33:19.154029 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.153812    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44zfk\" (UniqueName: \"kubernetes.io/projected/a949c15b-79d3-42e2-ba6f-aaa345dd0bcf-kube-api-access-44zfk\") pod \"ingress-canary-hmjxf\" (UID: \"a949c15b-79d3-42e2-ba6f-aaa345dd0bcf\") " pod="openshift-ingress-canary/ingress-canary-hmjxf"
Jul 16 20:33:19.154029 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.153960    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3589675b-514d-46eb-b879-9c66f98576ae-config-volume\") pod \"dns-default-x7pgr\" (UID: \"3589675b-514d-46eb-b879-9c66f98576ae\") " pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:19.154029 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.154003    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3589675b-514d-46eb-b879-9c66f98576ae-tmp-dir\") pod \"dns-default-x7pgr\" (UID: \"3589675b-514d-46eb-b879-9c66f98576ae\") " pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:19.154214 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.154031    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3589675b-514d-46eb-b879-9c66f98576ae-metrics-tls\") pod \"dns-default-x7pgr\" (UID: \"3589675b-514d-46eb-b879-9c66f98576ae\") " pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:19.154214 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:19.154162    2530 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 20:33:19.154304 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:19.154233    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3589675b-514d-46eb-b879-9c66f98576ae-metrics-tls podName:3589675b-514d-46eb-b879-9c66f98576ae nodeName:}" failed. No retries permitted until 2026-07-16 20:33:19.654211869 +0000 UTC m=+37.675799614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3589675b-514d-46eb-b879-9c66f98576ae-metrics-tls") pod "dns-default-x7pgr" (UID: "3589675b-514d-46eb-b879-9c66f98576ae") : secret "dns-default-metrics-tls" not found
Jul 16 20:33:19.154425 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.154403    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3589675b-514d-46eb-b879-9c66f98576ae-tmp-dir\") pod \"dns-default-x7pgr\" (UID: \"3589675b-514d-46eb-b879-9c66f98576ae\") " pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:19.154634 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.154617    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3589675b-514d-46eb-b879-9c66f98576ae-config-volume\") pod \"dns-default-x7pgr\" (UID: \"3589675b-514d-46eb-b879-9c66f98576ae\") " pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:19.156370 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.156347    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a949c15b-79d3-42e2-ba6f-aaa345dd0bcf-cert\") pod \"ingress-canary-hmjxf\" (UID: \"a949c15b-79d3-42e2-ba6f-aaa345dd0bcf\") " pod="openshift-ingress-canary/ingress-canary-hmjxf"
Jul 16 20:33:19.162266 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.162244    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzr6v\" (UniqueName: \"kubernetes.io/projected/3589675b-514d-46eb-b879-9c66f98576ae-kube-api-access-zzr6v\") pod \"dns-default-x7pgr\" (UID: \"3589675b-514d-46eb-b879-9c66f98576ae\") " pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:19.162376 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.162345    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44zfk\" (UniqueName: \"kubernetes.io/projected/a949c15b-79d3-42e2-ba6f-aaa345dd0bcf-kube-api-access-44zfk\") pod \"ingress-canary-hmjxf\" (UID: \"a949c15b-79d3-42e2-ba6f-aaa345dd0bcf\") " pod="openshift-ingress-canary/ingress-canary-hmjxf"
Jul 16 20:33:19.211624 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.211578    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hmjxf"
Jul 16 20:33:19.350236 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.349985    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hmjxf"]
Jul 16 20:33:19.353512 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:33:19.353481    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda949c15b_79d3_42e2_ba6f_aaa345dd0bcf.slice/crio-5e50d4d3bba365020787c3e4031731c010a4980cf251d809562c51273c5b44c6 WatchSource:0}: Error finding container 5e50d4d3bba365020787c3e4031731c010a4980cf251d809562c51273c5b44c6: Status 404 returned error can't find the container with id 5e50d4d3bba365020787c3e4031731c010a4980cf251d809562c51273c5b44c6
Jul 16 20:33:19.556988 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.556946    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:19.557594 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:19.557093    2530 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 20:33:19.557594 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:19.557131    2530 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869d7cdb67-bzsnl: secret "image-registry-tls" not found
Jul 16 20:33:19.557594 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:19.557190    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls podName:0705ca3b-1ea6-4263-a444-9618d29c2701 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:20.55717417 +0000 UTC m=+38.578761949 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls") pod "image-registry-869d7cdb67-bzsnl" (UID: "0705ca3b-1ea6-4263-a444-9618d29c2701") : secret "image-registry-tls" not found
Jul 16 20:33:19.657662 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.657562    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3589675b-514d-46eb-b879-9c66f98576ae-metrics-tls\") pod \"dns-default-x7pgr\" (UID: \"3589675b-514d-46eb-b879-9c66f98576ae\") " pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:19.657828 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:19.657734    2530 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 20:33:19.657828 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:19.657818    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3589675b-514d-46eb-b879-9c66f98576ae-metrics-tls podName:3589675b-514d-46eb-b879-9c66f98576ae nodeName:}" failed. No retries permitted until 2026-07-16 20:33:20.657797697 +0000 UTC m=+38.679385455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3589675b-514d-46eb-b879-9c66f98576ae-metrics-tls") pod "dns-default-x7pgr" (UID: "3589675b-514d-46eb-b879-9c66f98576ae") : secret "dns-default-metrics-tls" not found
Jul 16 20:33:19.739362 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.739326    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hmjxf" event={"ID":"a949c15b-79d3-42e2-ba6f-aaa345dd0bcf","Type":"ContainerStarted","Data":"5e50d4d3bba365020787c3e4031731c010a4980cf251d809562c51273c5b44c6"}
Jul 16 20:33:19.742001 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.741977    2530 generic.go:356] "Generic (PLEG): container finished" podID="586e85c8-fa49-4b8a-bc17-1d57c31b1c79" containerID="73028b9187dab740c1e64f880b5aac83ae667c635f43b1293da17a9ed1921bf2" exitCode=0
Jul 16 20:33:19.742143 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:19.742012    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xttrm" event={"ID":"586e85c8-fa49-4b8a-bc17-1d57c31b1c79","Type":"ContainerDied","Data":"73028b9187dab740c1e64f880b5aac83ae667c635f43b1293da17a9ed1921bf2"}
Jul 16 20:33:20.501895 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:20.501859    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:33:20.502240 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:20.501859    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:33:20.502560 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:20.501859    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:33:20.506377 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:20.506358    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 20:33:20.506485 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:20.506384    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pjwwv\""
Jul 16 20:33:20.506485 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:20.506362    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 20:33:20.506485 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:20.506480    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 20:33:20.506638 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:20.506358    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 20:33:20.506638 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:20.506584    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ql74g\""
Jul 16 20:33:20.565463 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:20.565438    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:20.565729 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:20.565537    2530 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 20:33:20.565729 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:20.565549    2530 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869d7cdb67-bzsnl: secret "image-registry-tls" not found
Jul 16 20:33:20.565729 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:20.565594    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls podName:0705ca3b-1ea6-4263-a444-9618d29c2701 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:22.565580565 +0000 UTC m=+40.587168308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls") pod "image-registry-869d7cdb67-bzsnl" (UID: "0705ca3b-1ea6-4263-a444-9618d29c2701") : secret "image-registry-tls" not found
Jul 16 20:33:20.666812 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:20.666779    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3589675b-514d-46eb-b879-9c66f98576ae-metrics-tls\") pod \"dns-default-x7pgr\" (UID: \"3589675b-514d-46eb-b879-9c66f98576ae\") " pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:20.666977 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:20.666949    2530 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 20:33:20.667050 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:20.667027    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3589675b-514d-46eb-b879-9c66f98576ae-metrics-tls podName:3589675b-514d-46eb-b879-9c66f98576ae nodeName:}" failed. No retries permitted until 2026-07-16 20:33:22.66700691 +0000 UTC m=+40.688594674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3589675b-514d-46eb-b879-9c66f98576ae-metrics-tls") pod "dns-default-x7pgr" (UID: "3589675b-514d-46eb-b879-9c66f98576ae") : secret "dns-default-metrics-tls" not found
Jul 16 20:33:20.749648 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:20.749611    2530 generic.go:356] "Generic (PLEG): container finished" podID="586e85c8-fa49-4b8a-bc17-1d57c31b1c79" containerID="68167ce6d7e8d99e2386017af97eed9328c0f459befc4acffd573e618f60db7e" exitCode=0
Jul 16 20:33:20.749792 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:20.749661    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xttrm" event={"ID":"586e85c8-fa49-4b8a-bc17-1d57c31b1c79","Type":"ContainerDied","Data":"68167ce6d7e8d99e2386017af97eed9328c0f459befc4acffd573e618f60db7e"}
Jul 16 20:33:21.754399 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:21.754233    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xttrm" event={"ID":"586e85c8-fa49-4b8a-bc17-1d57c31b1c79","Type":"ContainerStarted","Data":"ca700275829ed196e83698cb824199741b7981b59e7479fd21dab17eae3ce38a"}
Jul 16 20:33:21.776900 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:21.776824    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xttrm" podStartSLOduration=8.807665362 podStartE2EDuration="38.776810419s" totalImagesPullingTime="29.969145057s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:32:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:21.775414674 +0000 UTC m=+39.797002439" watchObservedRunningTime="2026-07-16 20:33:21.776810419 +0000 UTC m=+39.798398178"
Jul 16 20:33:22.582690 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:22.582661    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:22.582861 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:22.582804    2530 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 20:33:22.582861 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:22.582821    2530 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869d7cdb67-bzsnl: secret "image-registry-tls" not found
Jul 16 20:33:22.582943 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:22.582880    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls podName:0705ca3b-1ea6-4263-a444-9618d29c2701 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:26.582866196 +0000 UTC m=+44.604453981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls") pod "image-registry-869d7cdb67-bzsnl" (UID: "0705ca3b-1ea6-4263-a444-9618d29c2701") : secret "image-registry-tls" not found
Jul 16 20:33:22.683612 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:22.683585    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3589675b-514d-46eb-b879-9c66f98576ae-metrics-tls\") pod \"dns-default-x7pgr\" (UID: \"3589675b-514d-46eb-b879-9c66f98576ae\") " pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:22.683731 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:22.683673    2530 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 20:33:22.683731 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:22.683715    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3589675b-514d-46eb-b879-9c66f98576ae-metrics-tls podName:3589675b-514d-46eb-b879-9c66f98576ae nodeName:}" failed. No retries permitted until 2026-07-16 20:33:26.683703649 +0000 UTC m=+44.705291408 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3589675b-514d-46eb-b879-9c66f98576ae-metrics-tls") pod "dns-default-x7pgr" (UID: "3589675b-514d-46eb-b879-9c66f98576ae") : secret "dns-default-metrics-tls" not found
Jul 16 20:33:22.757114 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:22.757074    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hmjxf" event={"ID":"a949c15b-79d3-42e2-ba6f-aaa345dd0bcf","Type":"ContainerStarted","Data":"f301ad9b2a819fbe43e45ac878dbba6591625e8eb5e2fc684f8b6fecbdd5d7ab"}
Jul 16 20:33:24.678382 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.678319    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hmjxf" podStartSLOduration=4.171503128 podStartE2EDuration="6.67830218s" totalImagesPullingTime="2.506799052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:22.7778728 +0000 UTC m=+40.799460556" watchObservedRunningTime="2026-07-16 20:33:24.67830218 +0000 UTC m=+42.699889944"
Jul 16 20:33:24.678956 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.678942    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fbf6t"]
Jul 16 20:33:24.722366 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.722339    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fbf6t"]
Jul 16 20:33:24.722467 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.722433    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fbf6t"
Jul 16 20:33:24.726007 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.725479    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 16 20:33:24.726542 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.726521    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jf2c9\""
Jul 16 20:33:24.726746 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.726669    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 16 20:33:24.786438 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.786411    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-55rbb"]
Jul 16 20:33:24.807817 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.807798    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-55rbb"]
Jul 16 20:33:24.807924 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.807909    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:24.810208 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.810166    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 20:33:24.810426 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.810211    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 20:33:24.810426 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.810281    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 20:33:24.810426 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.810404    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 20:33:24.810590 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.810494    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-khzsm\""
Jul 16 20:33:24.899807 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.899773    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:24.899948 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.899831    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/71aec882-bb2e-4daa-9975-3eb3b1879cfa-crio-socket\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:24.899948 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.899860    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzj2d\" (UniqueName: \"kubernetes.io/projected/210aa416-26e0-48e0-97c2-a04d85c4be5b-kube-api-access-zzj2d\") pod \"migrator-7cc9c7ffd7-fbf6t\" (UID: \"210aa416-26e0-48e0-97c2-a04d85c4be5b\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fbf6t"
Jul 16 20:33:24.899948 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.899930    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfgcj\" (UniqueName: \"kubernetes.io/projected/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-api-access-bfgcj\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:24.900059 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.900013    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:24.900059 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:24.900038    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/71aec882-bb2e-4daa-9975-3eb3b1879cfa-data-volume\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:25.000532 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:25.000505    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgcj\" (UniqueName: \"kubernetes.io/projected/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-api-access-bfgcj\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:25.000646 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:25.000563    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:25.000646 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:25.000593    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/71aec882-bb2e-4daa-9975-3eb3b1879cfa-data-volume\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:25.000646 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:25.000619    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:25.000778 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:25.000653    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/71aec882-bb2e-4daa-9975-3eb3b1879cfa-crio-socket\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:25.000778 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:25.000678    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzj2d\" (UniqueName: \"kubernetes.io/projected/210aa416-26e0-48e0-97c2-a04d85c4be5b-kube-api-access-zzj2d\") pod \"migrator-7cc9c7ffd7-fbf6t\" (UID: \"210aa416-26e0-48e0-97c2-a04d85c4be5b\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fbf6t"
Jul 16 20:33:25.000778 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:25.000712    2530 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 20:33:25.000778 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:25.000753    2530 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 20:33:25.000939 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:25.000786    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-rbac-proxy-cm podName:71aec882-bb2e-4daa-9975-3eb3b1879cfa nodeName:}" failed. No retries permitted until 2026-07-16 20:33:25.500766391 +0000 UTC m=+43.522354155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-55rbb" (UID: "71aec882-bb2e-4daa-9975-3eb3b1879cfa") : configmap "kube-rbac-proxy" not found
Jul 16 20:33:25.000939 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:25.000806    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls podName:71aec882-bb2e-4daa-9975-3eb3b1879cfa nodeName:}" failed. No retries permitted until 2026-07-16 20:33:25.500793818 +0000 UTC m=+43.522381583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-55rbb" (UID: "71aec882-bb2e-4daa-9975-3eb3b1879cfa") : secret "insights-runtime-extractor-tls" not found
Jul 16 20:33:25.000939 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:25.000868    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/71aec882-bb2e-4daa-9975-3eb3b1879cfa-crio-socket\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:25.001038 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:25.000979    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/71aec882-bb2e-4daa-9975-3eb3b1879cfa-data-volume\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:25.010937 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:25.010915    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzj2d\" (UniqueName: \"kubernetes.io/projected/210aa416-26e0-48e0-97c2-a04d85c4be5b-kube-api-access-zzj2d\") pod \"migrator-7cc9c7ffd7-fbf6t\" (UID: \"210aa416-26e0-48e0-97c2-a04d85c4be5b\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fbf6t"
Jul 16 20:33:25.011033 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:25.010955    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgcj\" (UniqueName: \"kubernetes.io/projected/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-api-access-bfgcj\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:25.032723 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:25.032702    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fbf6t"
Jul 16 20:33:25.166195 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:25.166167    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fbf6t"]
Jul 16 20:33:25.168921 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:33:25.168896    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210aa416_26e0_48e0_97c2_a04d85c4be5b.slice/crio-d6df0a7c8894d82de60aba2fef582cbc2b1070b6256a00c08b0b37d91fa31bb8 WatchSource:0}: Error finding container d6df0a7c8894d82de60aba2fef582cbc2b1070b6256a00c08b0b37d91fa31bb8: Status 404 returned error can't find the container with id d6df0a7c8894d82de60aba2fef582cbc2b1070b6256a00c08b0b37d91fa31bb8
Jul 16 20:33:25.503328 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:25.503297    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:25.503455 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:25.503333    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:25.503455 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:25.503417    2530 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 20:33:25.503552 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:25.503457    2530 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 20:33:25.503552 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:25.503483    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-rbac-proxy-cm podName:71aec882-bb2e-4daa-9975-3eb3b1879cfa nodeName:}" failed. No retries permitted until 2026-07-16 20:33:26.50346646 +0000 UTC m=+44.525054204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-55rbb" (UID: "71aec882-bb2e-4daa-9975-3eb3b1879cfa") : configmap "kube-rbac-proxy" not found
Jul 16 20:33:25.503552 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:25.503504    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls podName:71aec882-bb2e-4daa-9975-3eb3b1879cfa nodeName:}" failed. No retries permitted until 2026-07-16 20:33:26.503493096 +0000 UTC m=+44.525080840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-55rbb" (UID: "71aec882-bb2e-4daa-9975-3eb3b1879cfa") : secret "insights-runtime-extractor-tls" not found
Jul 16 20:33:25.764721 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:25.764643    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fbf6t" event={"ID":"210aa416-26e0-48e0-97c2-a04d85c4be5b","Type":"ContainerStarted","Data":"d6df0a7c8894d82de60aba2fef582cbc2b1070b6256a00c08b0b37d91fa31bb8"}
Jul 16 20:33:26.511068 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:26.511034    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:26.511244 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:26.511149    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:26.511244 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:26.511183    2530 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 20:33:26.511244 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:26.511243    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls podName:71aec882-bb2e-4daa-9975-3eb3b1879cfa nodeName:}" failed. No retries permitted until 2026-07-16 20:33:28.51122875 +0000 UTC m=+46.532816495 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-55rbb" (UID: "71aec882-bb2e-4daa-9975-3eb3b1879cfa") : secret "insights-runtime-extractor-tls" not found
Jul 16 20:33:26.511350 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:26.511242    2530 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 20:33:26.511350 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:26.511286    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-rbac-proxy-cm podName:71aec882-bb2e-4daa-9975-3eb3b1879cfa nodeName:}" failed. No retries permitted until 2026-07-16 20:33:28.511274652 +0000 UTC m=+46.532862409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-55rbb" (UID: "71aec882-bb2e-4daa-9975-3eb3b1879cfa") : configmap "kube-rbac-proxy" not found
Jul 16 20:33:26.612519 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:26.612475    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:26.612710 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:26.612610    2530 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 20:33:26.612710 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:26.612626    2530 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869d7cdb67-bzsnl: secret "image-registry-tls" not found
Jul 16 20:33:26.612710 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:26.612700    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls podName:0705ca3b-1ea6-4263-a444-9618d29c2701 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:34.61268174 +0000 UTC m=+52.634269498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls") pod "image-registry-869d7cdb67-bzsnl" (UID: "0705ca3b-1ea6-4263-a444-9618d29c2701") : secret "image-registry-tls" not found
Jul 16 20:33:26.712859 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:26.712819    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3589675b-514d-46eb-b879-9c66f98576ae-metrics-tls\") pod \"dns-default-x7pgr\" (UID: \"3589675b-514d-46eb-b879-9c66f98576ae\") " pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:26.713032 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:26.713008    2530 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 20:33:26.713085 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:26.713074    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3589675b-514d-46eb-b879-9c66f98576ae-metrics-tls podName:3589675b-514d-46eb-b879-9c66f98576ae nodeName:}" failed. No retries permitted until 2026-07-16 20:33:34.71305863 +0000 UTC m=+52.734646389 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3589675b-514d-46eb-b879-9c66f98576ae-metrics-tls") pod "dns-default-x7pgr" (UID: "3589675b-514d-46eb-b879-9c66f98576ae") : secret "dns-default-metrics-tls" not found
Jul 16 20:33:26.929769 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:26.929745    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-764h9_9388c864-76b1-415e-a3d8-6114d5968730/dns-node-resolver/0.log"
Jul 16 20:33:27.051826 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.051745    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-qtn7v"]
Jul 16 20:33:27.062707 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.062680    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-qtn7v"]
Jul 16 20:33:27.062821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.062769    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-qtn7v"
Jul 16 20:33:27.065327 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.065308    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 16 20:33:27.066412 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.066383    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 16 20:33:27.066508 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.066383    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-59vkl\""
Jul 16 20:33:27.066568 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.066388    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 16 20:33:27.066629 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.066466    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 16 20:33:27.216692 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.216658    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq95g\" (UniqueName: \"kubernetes.io/projected/544e5296-7cf4-4593-aaec-5f86bb32cb78-kube-api-access-rq95g\") pod \"service-ca-58598695f7-qtn7v\" (UID: \"544e5296-7cf4-4593-aaec-5f86bb32cb78\") " pod="openshift-service-ca/service-ca-58598695f7-qtn7v"
Jul 16 20:33:27.216879 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.216724    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/544e5296-7cf4-4593-aaec-5f86bb32cb78-signing-cabundle\") pod \"service-ca-58598695f7-qtn7v\" (UID: \"544e5296-7cf4-4593-aaec-5f86bb32cb78\") " pod="openshift-service-ca/service-ca-58598695f7-qtn7v"
Jul 16 20:33:27.216879 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.216766    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/544e5296-7cf4-4593-aaec-5f86bb32cb78-signing-key\") pod \"service-ca-58598695f7-qtn7v\" (UID: \"544e5296-7cf4-4593-aaec-5f86bb32cb78\") " pod="openshift-service-ca/service-ca-58598695f7-qtn7v"
Jul 16 20:33:27.317651 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.317573    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/544e5296-7cf4-4593-aaec-5f86bb32cb78-signing-cabundle\") pod \"service-ca-58598695f7-qtn7v\" (UID: \"544e5296-7cf4-4593-aaec-5f86bb32cb78\") " pod="openshift-service-ca/service-ca-58598695f7-qtn7v"
Jul 16 20:33:27.317651 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.317634    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/544e5296-7cf4-4593-aaec-5f86bb32cb78-signing-key\") pod \"service-ca-58598695f7-qtn7v\" (UID: \"544e5296-7cf4-4593-aaec-5f86bb32cb78\") " pod="openshift-service-ca/service-ca-58598695f7-qtn7v"
Jul 16 20:33:27.317893 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.317678    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq95g\" (UniqueName: \"kubernetes.io/projected/544e5296-7cf4-4593-aaec-5f86bb32cb78-kube-api-access-rq95g\") pod \"service-ca-58598695f7-qtn7v\" (UID: \"544e5296-7cf4-4593-aaec-5f86bb32cb78\") " pod="openshift-service-ca/service-ca-58598695f7-qtn7v"
Jul 16 20:33:27.318258 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.318236    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/544e5296-7cf4-4593-aaec-5f86bb32cb78-signing-cabundle\") pod \"service-ca-58598695f7-qtn7v\" (UID: \"544e5296-7cf4-4593-aaec-5f86bb32cb78\") " pod="openshift-service-ca/service-ca-58598695f7-qtn7v"
Jul 16 20:33:27.320078 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.320056    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/544e5296-7cf4-4593-aaec-5f86bb32cb78-signing-key\") pod \"service-ca-58598695f7-qtn7v\" (UID: \"544e5296-7cf4-4593-aaec-5f86bb32cb78\") " pod="openshift-service-ca/service-ca-58598695f7-qtn7v"
Jul 16 20:33:27.325596 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.325569    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq95g\" (UniqueName: \"kubernetes.io/projected/544e5296-7cf4-4593-aaec-5f86bb32cb78-kube-api-access-rq95g\") pod \"service-ca-58598695f7-qtn7v\" (UID: \"544e5296-7cf4-4593-aaec-5f86bb32cb78\") " pod="openshift-service-ca/service-ca-58598695f7-qtn7v"
Jul 16 20:33:27.371366 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.371331    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-qtn7v"
Jul 16 20:33:27.504548 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.504517    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-qtn7v"]
Jul 16 20:33:27.509162 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:33:27.507770    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544e5296_7cf4_4593_aaec_5f86bb32cb78.slice/crio-cfe7f68a157978770a847b102ab043933a3b20d5df17f51b080d333394360ddd WatchSource:0}: Error finding container cfe7f68a157978770a847b102ab043933a3b20d5df17f51b080d333394360ddd: Status 404 returned error can't find the container with id cfe7f68a157978770a847b102ab043933a3b20d5df17f51b080d333394360ddd
Jul 16 20:33:27.770492 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.770458    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fbf6t" event={"ID":"210aa416-26e0-48e0-97c2-a04d85c4be5b","Type":"ContainerStarted","Data":"076520b69efc57e1922a865f7ab07d4466327f5626c21aa5fa1450fe7b56ab89"}
Jul 16 20:33:27.770679 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.770534    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fbf6t" event={"ID":"210aa416-26e0-48e0-97c2-a04d85c4be5b","Type":"ContainerStarted","Data":"b26711b5bb5a05fb39c3e9a122c4aed6a6355d41e086e8de169e22aaddfd6900"}
Jul 16 20:33:27.771503 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.771480    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-qtn7v" event={"ID":"544e5296-7cf4-4593-aaec-5f86bb32cb78","Type":"ContainerStarted","Data":"cfe7f68a157978770a847b102ab043933a3b20d5df17f51b080d333394360ddd"}
Jul 16 20:33:27.789523 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:27.789476    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fbf6t" podStartSLOduration=2.185760539 podStartE2EDuration="3.789461151s" totalImagesPullingTime="1.603700612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:27.788323498 +0000 UTC m=+45.809911255" watchObservedRunningTime="2026-07-16 20:33:27.789461151 +0000 UTC m=+45.811048917"
Jul 16 20:33:28.131028 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:28.130946    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cwld6_4288933c-4329-4853-8fb3-e3d46c34479f/node-ca/0.log"
Jul 16 20:33:28.527338 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:28.527307    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:28.527338 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:28.527349    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:28.527563 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:28.527440    2530 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 20:33:28.527563 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:28.527469    2530 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 20:33:28.527563 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:28.527512    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls podName:71aec882-bb2e-4daa-9975-3eb3b1879cfa nodeName:}" failed. No retries permitted until 2026-07-16 20:33:32.527499316 +0000 UTC m=+50.549087060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-55rbb" (UID: "71aec882-bb2e-4daa-9975-3eb3b1879cfa") : secret "insights-runtime-extractor-tls" not found
Jul 16 20:33:28.527563 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:28.527530    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-rbac-proxy-cm podName:71aec882-bb2e-4daa-9975-3eb3b1879cfa nodeName:}" failed. No retries permitted until 2026-07-16 20:33:32.527519682 +0000 UTC m=+50.549107426 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-55rbb" (UID: "71aec882-bb2e-4daa-9975-3eb3b1879cfa") : configmap "kube-rbac-proxy" not found
Jul 16 20:33:28.730305 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:28.730273    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hmjxf_a949c15b-79d3-42e2-ba6f-aaa345dd0bcf/serve-healthcheck-canary/0.log"
Jul 16 20:33:29.130034 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:29.130009    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-fbf6t_210aa416-26e0-48e0-97c2-a04d85c4be5b/migrator/0.log"
Jul 16 20:33:29.329938 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:29.329905    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-fbf6t_210aa416-26e0-48e0-97c2-a04d85c4be5b/graceful-termination/0.log"
Jul 16 20:33:30.780116 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:30.780069    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-qtn7v" event={"ID":"544e5296-7cf4-4593-aaec-5f86bb32cb78","Type":"ContainerStarted","Data":"d0576b9876b7d1136b842b294b1458ce2d2c8f6c2f05247a4a5518719d5ce3ce"}
Jul 16 20:33:30.800387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:30.800312    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-qtn7v" podStartSLOduration=1.5959412039999998 podStartE2EDuration="3.800294458s" totalImagesPullingTime="2.204353254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:30.799456804 +0000 UTC m=+48.821044569" watchObservedRunningTime="2026-07-16 20:33:30.800294458 +0000 UTC m=+48.821882225"
Jul 16 20:33:32.561845 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:32.561803    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:32.561845 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:32.561840    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:32.562371 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:32.561915    2530 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 20:33:32.562371 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:32.561985    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-rbac-proxy-cm podName:71aec882-bb2e-4daa-9975-3eb3b1879cfa nodeName:}" failed. No retries permitted until 2026-07-16 20:33:40.561966613 +0000 UTC m=+58.583554359 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-55rbb" (UID: "71aec882-bb2e-4daa-9975-3eb3b1879cfa") : configmap "kube-rbac-proxy" not found
Jul 16 20:33:32.562371 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:32.561984    2530 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 20:33:32.562371 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:32.562029    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls podName:71aec882-bb2e-4daa-9975-3eb3b1879cfa nodeName:}" failed. No retries permitted until 2026-07-16 20:33:40.5620155 +0000 UTC m=+58.583603244 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-55rbb" (UID: "71aec882-bb2e-4daa-9975-3eb3b1879cfa") : secret "insights-runtime-extractor-tls" not found
Jul 16 20:33:34.676015 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:34.675964    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls\") pod \"image-registry-869d7cdb67-bzsnl\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") " pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:34.676598 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:34.676131    2530 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 20:33:34.676598 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:34.676155    2530 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-869d7cdb67-bzsnl: secret "image-registry-tls" not found
Jul 16 20:33:34.676598 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:34.676230    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls podName:0705ca3b-1ea6-4263-a444-9618d29c2701 nodeName:}" failed. No retries permitted until 2026-07-16 20:33:50.67620922 +0000 UTC m=+68.697796964 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls") pod "image-registry-869d7cdb67-bzsnl" (UID: "0705ca3b-1ea6-4263-a444-9618d29c2701") : secret "image-registry-tls" not found
Jul 16 20:33:34.776862 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:34.776822    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3589675b-514d-46eb-b879-9c66f98576ae-metrics-tls\") pod \"dns-default-x7pgr\" (UID: \"3589675b-514d-46eb-b879-9c66f98576ae\") " pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:34.779248 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:34.779225    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3589675b-514d-46eb-b879-9c66f98576ae-metrics-tls\") pod \"dns-default-x7pgr\" (UID: \"3589675b-514d-46eb-b879-9c66f98576ae\") " pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:34.806256 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:34.806220    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:34.954252 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:34.954220    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x7pgr"]
Jul 16 20:33:34.958028 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:33:34.958000    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3589675b_514d_46eb_b879_9c66f98576ae.slice/crio-edcd022202dcac64bbb237fb5582ab88eb248bfce712a82a7146c8949ce5665d WatchSource:0}: Error finding container edcd022202dcac64bbb237fb5582ab88eb248bfce712a82a7146c8949ce5665d: Status 404 returned error can't find the container with id edcd022202dcac64bbb237fb5582ab88eb248bfce712a82a7146c8949ce5665d
Jul 16 20:33:35.794631 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:35.794596    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x7pgr" event={"ID":"3589675b-514d-46eb-b879-9c66f98576ae","Type":"ContainerStarted","Data":"edcd022202dcac64bbb237fb5582ab88eb248bfce712a82a7146c8949ce5665d"}
Jul 16 20:33:37.802241 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:37.802207    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x7pgr" event={"ID":"3589675b-514d-46eb-b879-9c66f98576ae","Type":"ContainerStarted","Data":"52bf792ad4d51302f1e619f72a0cb26650f286b8d2c83122d401bdf33107be9c"}
Jul 16 20:33:37.802619 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:37.802264    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x7pgr" event={"ID":"3589675b-514d-46eb-b879-9c66f98576ae","Type":"ContainerStarted","Data":"d3090aab9408c1d4d43a8a1e6e76150adbc955c210155f7fad5fe729c18356ae"}
Jul 16 20:33:37.802619 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:37.802339    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:37.821043 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:37.820988    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-x7pgr" podStartSLOduration=17.619315875 podStartE2EDuration="19.820970791s" totalImagesPullingTime="2.201654916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:37.819959323 +0000 UTC m=+55.841547089" watchObservedRunningTime="2026-07-16 20:33:37.820970791 +0000 UTC m=+55.842558557"
Jul 16 20:33:40.624764 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:40.624725    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:40.625329 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:40.624821    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:40.625329 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:40.624898    2530 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 20:33:40.625329 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:40.625012    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls podName:71aec882-bb2e-4daa-9975-3eb3b1879cfa nodeName:}" failed. No retries permitted until 2026-07-16 20:33:56.624990849 +0000 UTC m=+74.646578610 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-55rbb" (UID: "71aec882-bb2e-4daa-9975-3eb3b1879cfa") : secret "insights-runtime-extractor-tls" not found
Jul 16 20:33:40.625329 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:40.625246    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71aec882-bb2e-4daa-9975-3eb3b1879cfa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:43.743853 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:43.743820    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-blwng"
Jul 16 20:33:45.952451 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.952414    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8"]
Jul 16 20:33:45.955307 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.955289    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh"]
Jul 16 20:33:45.955445 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.955428    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8"
Jul 16 20:33:45.958690 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.958664    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 16 20:33:45.958905 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.958878    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5zhdc\""
Jul 16 20:33:45.959433 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.959411    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh"
Jul 16 20:33:45.960650 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.960624    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 16 20:33:45.960752 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.960731    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"]
Jul 16 20:33:45.961261 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.961241    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 16 20:33:45.961377 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.961258    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 16 20:33:45.963299 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.963282    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:45.963441 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.963409    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 16 20:33:45.964150 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.964134    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh"]
Jul 16 20:33:45.965012 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.964994    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8"]
Jul 16 20:33:45.965833 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.965811    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 16 20:33:45.966178 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.966159    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 16 20:33:45.966269 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.966180    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 16 20:33:45.966465 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.966433    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 16 20:33:45.978248 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:45.978226    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"]
Jul 16 20:33:46.040303 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.040264    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-869d7cdb67-bzsnl"]
Jul 16 20:33:46.040510 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:33:46.040486    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl" podUID="0705ca3b-1ea6-4263-a444-9618d29c2701"
Jul 16 20:33:46.064173 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.064138    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5bde78e-efad-491e-b9b0-59c856a6ac24-tmp\") pod \"klusterlet-addon-workmgr-67f5949cf8-5mxgh\" (UID: \"d5bde78e-efad-491e-b9b0-59c856a6ac24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh"
Jul 16 20:33:46.064342 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.064178    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbn9q\" (UniqueName: \"kubernetes.io/projected/a9fdcb17-e49a-4e07-8310-4f4915ef5a51-kube-api-access-vbn9q\") pod \"cluster-proxy-proxy-agent-7644978f68-hwk4g\" (UID: \"a9fdcb17-e49a-4e07-8310-4f4915ef5a51\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.064342 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.064255    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g84gh\" (UniqueName: \"kubernetes.io/projected/e40908ed-596b-4ccd-b230-72cf4c41e9a2-kube-api-access-g84gh\") pod \"managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8\" (UID: \"e40908ed-596b-4ccd-b230-72cf4c41e9a2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8"
Jul 16 20:33:46.064342 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.064301    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj8x7\" (UniqueName: \"kubernetes.io/projected/d5bde78e-efad-491e-b9b0-59c856a6ac24-kube-api-access-dj8x7\") pod \"klusterlet-addon-workmgr-67f5949cf8-5mxgh\" (UID: \"d5bde78e-efad-491e-b9b0-59c856a6ac24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh"
Jul 16 20:33:46.064466 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.064391    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d5bde78e-efad-491e-b9b0-59c856a6ac24-klusterlet-config\") pod \"klusterlet-addon-workmgr-67f5949cf8-5mxgh\" (UID: \"d5bde78e-efad-491e-b9b0-59c856a6ac24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh"
Jul 16 20:33:46.064466 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.064417    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a9fdcb17-e49a-4e07-8310-4f4915ef5a51-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7644978f68-hwk4g\" (UID: \"a9fdcb17-e49a-4e07-8310-4f4915ef5a51\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.064466 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.064438    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a9fdcb17-e49a-4e07-8310-4f4915ef5a51-ca\") pod \"cluster-proxy-proxy-agent-7644978f68-hwk4g\" (UID: \"a9fdcb17-e49a-4e07-8310-4f4915ef5a51\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.064466 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.064453    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fdcb17-e49a-4e07-8310-4f4915ef5a51-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7644978f68-hwk4g\" (UID: \"a9fdcb17-e49a-4e07-8310-4f4915ef5a51\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.064613 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.064474    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e40908ed-596b-4ccd-b230-72cf4c41e9a2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8\" (UID: \"e40908ed-596b-4ccd-b230-72cf4c41e9a2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8"
Jul 16 20:33:46.064613 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.064522    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a9fdcb17-e49a-4e07-8310-4f4915ef5a51-hub\") pod \"cluster-proxy-proxy-agent-7644978f68-hwk4g\" (UID: \"a9fdcb17-e49a-4e07-8310-4f4915ef5a51\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.064613 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.064557    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a9fdcb17-e49a-4e07-8310-4f4915ef5a51-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7644978f68-hwk4g\" (UID: \"a9fdcb17-e49a-4e07-8310-4f4915ef5a51\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.153905 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.153868    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85d4cdcc5d-h29lp"]
Jul 16 20:33:46.155786 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.155768    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.165341 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.165321    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d5bde78e-efad-491e-b9b0-59c856a6ac24-klusterlet-config\") pod \"klusterlet-addon-workmgr-67f5949cf8-5mxgh\" (UID: \"d5bde78e-efad-491e-b9b0-59c856a6ac24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh"
Jul 16 20:33:46.165441 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.165350    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a9fdcb17-e49a-4e07-8310-4f4915ef5a51-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7644978f68-hwk4g\" (UID: \"a9fdcb17-e49a-4e07-8310-4f4915ef5a51\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.165441 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.165370    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a9fdcb17-e49a-4e07-8310-4f4915ef5a51-ca\") pod \"cluster-proxy-proxy-agent-7644978f68-hwk4g\" (UID: \"a9fdcb17-e49a-4e07-8310-4f4915ef5a51\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.165441 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.165385    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fdcb17-e49a-4e07-8310-4f4915ef5a51-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7644978f68-hwk4g\" (UID: \"a9fdcb17-e49a-4e07-8310-4f4915ef5a51\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.165441 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.165415    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e40908ed-596b-4ccd-b230-72cf4c41e9a2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8\" (UID: \"e40908ed-596b-4ccd-b230-72cf4c41e9a2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8"
Jul 16 20:33:46.165441 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.165430    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a9fdcb17-e49a-4e07-8310-4f4915ef5a51-hub\") pod \"cluster-proxy-proxy-agent-7644978f68-hwk4g\" (UID: \"a9fdcb17-e49a-4e07-8310-4f4915ef5a51\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.165689 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.165456    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a9fdcb17-e49a-4e07-8310-4f4915ef5a51-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7644978f68-hwk4g\" (UID: \"a9fdcb17-e49a-4e07-8310-4f4915ef5a51\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.165689 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.165518    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5bde78e-efad-491e-b9b0-59c856a6ac24-tmp\") pod \"klusterlet-addon-workmgr-67f5949cf8-5mxgh\" (UID: \"d5bde78e-efad-491e-b9b0-59c856a6ac24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh"
Jul 16 20:33:46.165689 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.165540    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbn9q\" (UniqueName: \"kubernetes.io/projected/a9fdcb17-e49a-4e07-8310-4f4915ef5a51-kube-api-access-vbn9q\") pod \"cluster-proxy-proxy-agent-7644978f68-hwk4g\" (UID: \"a9fdcb17-e49a-4e07-8310-4f4915ef5a51\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.165689 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.165599    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g84gh\" (UniqueName: \"kubernetes.io/projected/e40908ed-596b-4ccd-b230-72cf4c41e9a2-kube-api-access-g84gh\") pod \"managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8\" (UID: \"e40908ed-596b-4ccd-b230-72cf4c41e9a2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8"
Jul 16 20:33:46.165689 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.165654    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj8x7\" (UniqueName: \"kubernetes.io/projected/d5bde78e-efad-491e-b9b0-59c856a6ac24-kube-api-access-dj8x7\") pod \"klusterlet-addon-workmgr-67f5949cf8-5mxgh\" (UID: \"d5bde78e-efad-491e-b9b0-59c856a6ac24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh"
Jul 16 20:33:46.165950 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.165926    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5bde78e-efad-491e-b9b0-59c856a6ac24-tmp\") pod \"klusterlet-addon-workmgr-67f5949cf8-5mxgh\" (UID: \"d5bde78e-efad-491e-b9b0-59c856a6ac24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh"
Jul 16 20:33:46.166399 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.166373    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fdcb17-e49a-4e07-8310-4f4915ef5a51-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7644978f68-hwk4g\" (UID: \"a9fdcb17-e49a-4e07-8310-4f4915ef5a51\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.168566 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.168524    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a9fdcb17-e49a-4e07-8310-4f4915ef5a51-ca\") pod \"cluster-proxy-proxy-agent-7644978f68-hwk4g\" (UID: \"a9fdcb17-e49a-4e07-8310-4f4915ef5a51\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.168770 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.168747    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d5bde78e-efad-491e-b9b0-59c856a6ac24-klusterlet-config\") pod \"klusterlet-addon-workmgr-67f5949cf8-5mxgh\" (UID: \"d5bde78e-efad-491e-b9b0-59c856a6ac24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh"
Jul 16 20:33:46.168906 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.168888    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a9fdcb17-e49a-4e07-8310-4f4915ef5a51-hub\") pod \"cluster-proxy-proxy-agent-7644978f68-hwk4g\" (UID: \"a9fdcb17-e49a-4e07-8310-4f4915ef5a51\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.168979 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.168938    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e40908ed-596b-4ccd-b230-72cf4c41e9a2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8\" (UID: \"e40908ed-596b-4ccd-b230-72cf4c41e9a2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8"
Jul 16 20:33:46.169060 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.169031    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a9fdcb17-e49a-4e07-8310-4f4915ef5a51-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7644978f68-hwk4g\" (UID: \"a9fdcb17-e49a-4e07-8310-4f4915ef5a51\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.169140 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.169058    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a9fdcb17-e49a-4e07-8310-4f4915ef5a51-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7644978f68-hwk4g\" (UID: \"a9fdcb17-e49a-4e07-8310-4f4915ef5a51\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.169807 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.169789    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85d4cdcc5d-h29lp"]
Jul 16 20:33:46.181149 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.181072    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj8x7\" (UniqueName: \"kubernetes.io/projected/d5bde78e-efad-491e-b9b0-59c856a6ac24-kube-api-access-dj8x7\") pod \"klusterlet-addon-workmgr-67f5949cf8-5mxgh\" (UID: \"d5bde78e-efad-491e-b9b0-59c856a6ac24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh"
Jul 16 20:33:46.181250 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.181232    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g84gh\" (UniqueName: \"kubernetes.io/projected/e40908ed-596b-4ccd-b230-72cf4c41e9a2-kube-api-access-g84gh\") pod \"managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8\" (UID: \"e40908ed-596b-4ccd-b230-72cf4c41e9a2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8"
Jul 16 20:33:46.181550 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.181535    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbn9q\" (UniqueName: \"kubernetes.io/projected/a9fdcb17-e49a-4e07-8310-4f4915ef5a51-kube-api-access-vbn9q\") pod \"cluster-proxy-proxy-agent-7644978f68-hwk4g\" (UID: \"a9fdcb17-e49a-4e07-8310-4f4915ef5a51\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.266291 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.266259    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkwzd\" (UniqueName: \"kubernetes.io/projected/fba88453-5e73-4e08-a483-0a764c1a6b6f-kube-api-access-zkwzd\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.266464 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.266310    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fba88453-5e73-4e08-a483-0a764c1a6b6f-ca-trust-extracted\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.266464 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.266337    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fba88453-5e73-4e08-a483-0a764c1a6b6f-installation-pull-secrets\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.266464 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.266366    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fba88453-5e73-4e08-a483-0a764c1a6b6f-bound-sa-token\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.266464 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.266390    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fba88453-5e73-4e08-a483-0a764c1a6b6f-image-registry-private-configuration\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.266464 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.266408    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fba88453-5e73-4e08-a483-0a764c1a6b6f-trusted-ca\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.266464 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.266440    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fba88453-5e73-4e08-a483-0a764c1a6b6f-registry-certificates\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.266648 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.266473    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fba88453-5e73-4e08-a483-0a764c1a6b6f-registry-tls\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.279186 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.279156    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8"
Jul 16 20:33:46.286866 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.286841    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh"
Jul 16 20:33:46.290896 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.290872    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"
Jul 16 20:33:46.367776 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.367742    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fba88453-5e73-4e08-a483-0a764c1a6b6f-ca-trust-extracted\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.367776 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.367781    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fba88453-5e73-4e08-a483-0a764c1a6b6f-installation-pull-secrets\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.368009 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.367798    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fba88453-5e73-4e08-a483-0a764c1a6b6f-bound-sa-token\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.368382 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.368194    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fba88453-5e73-4e08-a483-0a764c1a6b6f-ca-trust-extracted\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.368382 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.368251    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fba88453-5e73-4e08-a483-0a764c1a6b6f-image-registry-private-configuration\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.368382 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.368286    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fba88453-5e73-4e08-a483-0a764c1a6b6f-trusted-ca\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.368382 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.368327    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fba88453-5e73-4e08-a483-0a764c1a6b6f-registry-certificates\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.368382 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.368370    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fba88453-5e73-4e08-a483-0a764c1a6b6f-registry-tls\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.368693 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.368476    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkwzd\" (UniqueName: \"kubernetes.io/projected/fba88453-5e73-4e08-a483-0a764c1a6b6f-kube-api-access-zkwzd\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.369582 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.369558    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fba88453-5e73-4e08-a483-0a764c1a6b6f-trusted-ca\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.370121 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.370065    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fba88453-5e73-4e08-a483-0a764c1a6b6f-registry-certificates\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.371187 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.371068    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fba88453-5e73-4e08-a483-0a764c1a6b6f-installation-pull-secrets\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.371453 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.371261    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fba88453-5e73-4e08-a483-0a764c1a6b6f-image-registry-private-configuration\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.373718 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.372880    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fba88453-5e73-4e08-a483-0a764c1a6b6f-registry-tls\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.379845 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.379797    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fba88453-5e73-4e08-a483-0a764c1a6b6f-bound-sa-token\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.382491 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.382462    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkwzd\" (UniqueName: \"kubernetes.io/projected/fba88453-5e73-4e08-a483-0a764c1a6b6f-kube-api-access-zkwzd\") pod \"image-registry-85d4cdcc5d-h29lp\" (UID: \"fba88453-5e73-4e08-a483-0a764c1a6b6f\") " pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.430815 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.430779    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8"]
Jul 16 20:33:46.434020 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:33:46.433983    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode40908ed_596b_4ccd_b230_72cf4c41e9a2.slice/crio-62c3bd6d56315dca0f69f714dd1ab573620c9d4dd34ad03b4edf7aca58cfd438 WatchSource:0}: Error finding container 62c3bd6d56315dca0f69f714dd1ab573620c9d4dd34ad03b4edf7aca58cfd438: Status 404 returned error can't find the container with id 62c3bd6d56315dca0f69f714dd1ab573620c9d4dd34ad03b4edf7aca58cfd438
Jul 16 20:33:46.448821 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.448789    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh"]
Jul 16 20:33:46.453009 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:33:46.452982    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5bde78e_efad_491e_b9b0_59c856a6ac24.slice/crio-fc805975641336b39983124d8c69e32e78079adc61da041329e3d4a66ab8449d WatchSource:0}: Error finding container fc805975641336b39983124d8c69e32e78079adc61da041329e3d4a66ab8449d: Status 404 returned error can't find the container with id fc805975641336b39983124d8c69e32e78079adc61da041329e3d4a66ab8449d
Jul 16 20:33:46.466356 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.466332    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g"]
Jul 16 20:33:46.468987 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:33:46.468955    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9fdcb17_e49a_4e07_8310_4f4915ef5a51.slice/crio-f7b805ce576accea45afb21c3ef962b1f9aa5d16975f35e37c3886bfb76904bc WatchSource:0}: Error finding container f7b805ce576accea45afb21c3ef962b1f9aa5d16975f35e37c3886bfb76904bc: Status 404 returned error can't find the container with id f7b805ce576accea45afb21c3ef962b1f9aa5d16975f35e37c3886bfb76904bc
Jul 16 20:33:46.491269 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.491240    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-68jbx\""
Jul 16 20:33:46.499993 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.499971    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.623511 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.623482    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85d4cdcc5d-h29lp"]
Jul 16 20:33:46.626335 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:33:46.626312    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba88453_5e73_4e08_a483_0a764c1a6b6f.slice/crio-e28ee87b9b292229412bb328dd629c3544aeb6909656456058496a7c2770a1d6 WatchSource:0}: Error finding container e28ee87b9b292229412bb328dd629c3544aeb6909656456058496a7c2770a1d6: Status 404 returned error can't find the container with id e28ee87b9b292229412bb328dd629c3544aeb6909656456058496a7c2770a1d6
Jul 16 20:33:46.831466 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.831383    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8" event={"ID":"e40908ed-596b-4ccd-b230-72cf4c41e9a2","Type":"ContainerStarted","Data":"62c3bd6d56315dca0f69f714dd1ab573620c9d4dd34ad03b4edf7aca58cfd438"}
Jul 16 20:33:46.832908 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.832876    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp" event={"ID":"fba88453-5e73-4e08-a483-0a764c1a6b6f","Type":"ContainerStarted","Data":"ec113587df668ee3c53e91617ac694462ecdbb9e61856a3d1e70d229bc3c2dcb"}
Jul 16 20:33:46.833046 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.832915    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp" event={"ID":"fba88453-5e73-4e08-a483-0a764c1a6b6f","Type":"ContainerStarted","Data":"e28ee87b9b292229412bb328dd629c3544aeb6909656456058496a7c2770a1d6"}
Jul 16 20:33:46.833046 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.832952    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:33:46.834146 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.834124    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh" event={"ID":"d5bde78e-efad-491e-b9b0-59c856a6ac24","Type":"ContainerStarted","Data":"fc805975641336b39983124d8c69e32e78079adc61da041329e3d4a66ab8449d"}
Jul 16 20:33:46.835310 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.835290    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:46.835421 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.835315    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g" event={"ID":"a9fdcb17-e49a-4e07-8310-4f4915ef5a51","Type":"ContainerStarted","Data":"f7b805ce576accea45afb21c3ef962b1f9aa5d16975f35e37c3886bfb76904bc"}
Jul 16 20:33:46.840580 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.840560    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:46.851710 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.851672    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp" podStartSLOduration=0.851651771 podStartE2EDuration="851.651771ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:46.851054065 +0000 UTC m=+64.872641830" watchObservedRunningTime="2026-07-16 20:33:46.851651771 +0000 UTC m=+64.873239530"
Jul 16 20:33:46.872442 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.872411    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0705ca3b-1ea6-4263-a444-9618d29c2701-ca-trust-extracted\") pod \"0705ca3b-1ea6-4263-a444-9618d29c2701\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") "
Jul 16 20:33:46.872614 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.872449    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kf4k\" (UniqueName: \"kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-kube-api-access-8kf4k\") pod \"0705ca3b-1ea6-4263-a444-9618d29c2701\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") "
Jul 16 20:33:46.872614 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.872492    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0705ca3b-1ea6-4263-a444-9618d29c2701-installation-pull-secrets\") pod \"0705ca3b-1ea6-4263-a444-9618d29c2701\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") "
Jul 16 20:33:46.872614 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.872523    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-certificates\") pod \"0705ca3b-1ea6-4263-a444-9618d29c2701\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") "
Jul 16 20:33:46.872614 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.872575    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0705ca3b-1ea6-4263-a444-9618d29c2701-image-registry-private-configuration\") pod \"0705ca3b-1ea6-4263-a444-9618d29c2701\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") "
Jul 16 20:33:46.872812 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.872623    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0705ca3b-1ea6-4263-a444-9618d29c2701-trusted-ca\") pod \"0705ca3b-1ea6-4263-a444-9618d29c2701\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") "
Jul 16 20:33:46.872812 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.872656    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0705ca3b-1ea6-4263-a444-9618d29c2701-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0705ca3b-1ea6-4263-a444-9618d29c2701" (UID: "0705ca3b-1ea6-4263-a444-9618d29c2701"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 20:33:46.872812 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.872667    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-bound-sa-token\") pod \"0705ca3b-1ea6-4263-a444-9618d29c2701\" (UID: \"0705ca3b-1ea6-4263-a444-9618d29c2701\") "
Jul 16 20:33:46.873051 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.873028    2530 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0705ca3b-1ea6-4263-a444-9618d29c2701-ca-trust-extracted\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 20:33:46.873549 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.873421    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0705ca3b-1ea6-4263-a444-9618d29c2701" (UID: "0705ca3b-1ea6-4263-a444-9618d29c2701"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:33:46.875532 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.875460    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0705ca3b-1ea6-4263-a444-9618d29c2701-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0705ca3b-1ea6-4263-a444-9618d29c2701" (UID: "0705ca3b-1ea6-4263-a444-9618d29c2701"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:33:46.875532 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.875462    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0705ca3b-1ea6-4263-a444-9618d29c2701-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0705ca3b-1ea6-4263-a444-9618d29c2701" (UID: "0705ca3b-1ea6-4263-a444-9618d29c2701"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:33:46.876325 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.876281    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-kube-api-access-8kf4k" (OuterVolumeSpecName: "kube-api-access-8kf4k") pod "0705ca3b-1ea6-4263-a444-9618d29c2701" (UID: "0705ca3b-1ea6-4263-a444-9618d29c2701"). InnerVolumeSpecName "kube-api-access-8kf4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:33:46.878241 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.878215    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0705ca3b-1ea6-4263-a444-9618d29c2701" (UID: "0705ca3b-1ea6-4263-a444-9618d29c2701"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:33:46.879443 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.879422    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0705ca3b-1ea6-4263-a444-9618d29c2701-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0705ca3b-1ea6-4263-a444-9618d29c2701" (UID: "0705ca3b-1ea6-4263-a444-9618d29c2701"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 20:33:46.974178 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.974141    2530 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-bound-sa-token\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 20:33:46.974178 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.974169    2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kf4k\" (UniqueName: \"kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-kube-api-access-8kf4k\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 20:33:46.974178 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.974180    2530 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0705ca3b-1ea6-4263-a444-9618d29c2701-installation-pull-secrets\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 20:33:46.974178 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.974189    2530 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-certificates\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 20:33:46.974630 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.974198    2530 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0705ca3b-1ea6-4263-a444-9618d29c2701-image-registry-private-configuration\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 20:33:46.974630 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:46.974207    2530 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0705ca3b-1ea6-4263-a444-9618d29c2701-trusted-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 20:33:47.806970 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:47.806863    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x7pgr"
Jul 16 20:33:47.840454 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:47.840423    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-869d7cdb67-bzsnl"
Jul 16 20:33:47.884288 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:47.884250    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-869d7cdb67-bzsnl"]
Jul 16 20:33:47.890535 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:47.890498    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-869d7cdb67-bzsnl"]
Jul 16 20:33:47.984523 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:47.984486    2530 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0705ca3b-1ea6-4263-a444-9618d29c2701-registry-tls\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 20:33:48.187237 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:48.187159    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs\") pod \"network-metrics-daemon-4mmnt\" (UID: \"864a17b7-c612-43d2-923e-8843e0f519b0\") " pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:33:48.187237 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:48.187214    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret\") pod \"global-pull-secret-syncer-tjnr2\" (UID: \"7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e\") " pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:33:48.189669 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:48.189644    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 20:33:48.189669 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:48.189655    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 20:33:48.201094 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:48.201058    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864a17b7-c612-43d2-923e-8843e0f519b0-metrics-certs\") pod \"network-metrics-daemon-4mmnt\" (UID: \"864a17b7-c612-43d2-923e-8843e0f519b0\") " pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:33:48.203001 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:48.202921    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e-original-pull-secret\") pod \"global-pull-secret-syncer-tjnr2\" (UID: \"7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e\") " pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:33:48.388272 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:48.388233    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fhrc\" (UniqueName: \"kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc\") pod \"network-check-target-vxmfl\" (UID: \"2aed23ed-5e27-4e01-adc6-d8af18c38baa\") " pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:33:48.390619 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:48.390598    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 20:33:48.401035 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:48.401008    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 20:33:48.412000 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:48.411975    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fhrc\" (UniqueName: \"kubernetes.io/projected/2aed23ed-5e27-4e01-adc6-d8af18c38baa-kube-api-access-9fhrc\") pod \"network-check-target-vxmfl\" (UID: \"2aed23ed-5e27-4e01-adc6-d8af18c38baa\") " pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:33:48.414744 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:48.414719    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tjnr2"
Jul 16 20:33:48.423415 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:48.423390    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pjwwv\""
Jul 16 20:33:48.431620 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:48.431581    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:33:48.431952 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:48.431931    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ql74g\""
Jul 16 20:33:48.439973 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:48.439904    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4mmnt"
Jul 16 20:33:48.506141 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:48.506091    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0705ca3b-1ea6-4263-a444-9618d29c2701" path="/var/lib/kubelet/pods/0705ca3b-1ea6-4263-a444-9618d29c2701/volumes"
Jul 16 20:33:49.521089 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:49.521046    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tjnr2"]
Jul 16 20:33:49.530508 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:49.530431    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4mmnt"]
Jul 16 20:33:49.536566 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:33:49.536527    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864a17b7_c612_43d2_923e_8843e0f519b0.slice/crio-4022ad1f83b2cad6e46aec2a718059aabf854cb160b47ce2a1ddbd57cdee3794 WatchSource:0}: Error finding container 4022ad1f83b2cad6e46aec2a718059aabf854cb160b47ce2a1ddbd57cdee3794: Status 404 returned error can't find the container with id 4022ad1f83b2cad6e46aec2a718059aabf854cb160b47ce2a1ddbd57cdee3794
Jul 16 20:33:49.537901 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:49.537832    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vxmfl"]
Jul 16 20:33:49.542524 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:33:49.542495    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aed23ed_5e27_4e01_adc6_d8af18c38baa.slice/crio-2b05ea71271aecdab3c473345dce2ddbaefd1c8f703f68630ac5b2b7db251951 WatchSource:0}: Error finding container 2b05ea71271aecdab3c473345dce2ddbaefd1c8f703f68630ac5b2b7db251951: Status 404 returned error can't find the container with id 2b05ea71271aecdab3c473345dce2ddbaefd1c8f703f68630ac5b2b7db251951
Jul 16 20:33:49.855683 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:49.855630    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8" event={"ID":"e40908ed-596b-4ccd-b230-72cf4c41e9a2","Type":"ContainerStarted","Data":"baf99b31fe1d6f6cf071848fd695ad501cdb07b2072d61f04f388776c58a7f5e"}
Jul 16 20:33:49.857181 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:49.857156    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4mmnt" event={"ID":"864a17b7-c612-43d2-923e-8843e0f519b0","Type":"ContainerStarted","Data":"4022ad1f83b2cad6e46aec2a718059aabf854cb160b47ce2a1ddbd57cdee3794"}
Jul 16 20:33:49.858429 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:49.858403    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tjnr2" event={"ID":"7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e","Type":"ContainerStarted","Data":"fe081aa710557a77cac01936b7cb369ceddd3e05f781df80d785fbe1f961d297"}
Jul 16 20:33:49.859674 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:49.859651    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vxmfl" event={"ID":"2aed23ed-5e27-4e01-adc6-d8af18c38baa","Type":"ContainerStarted","Data":"2b05ea71271aecdab3c473345dce2ddbaefd1c8f703f68630ac5b2b7db251951"}
Jul 16 20:33:49.872136 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:49.871922    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f8bdbbcbb-kc2v8" podStartSLOduration=1.892531642 podStartE2EDuration="4.871903136s" totalImagesPullingTime="2.979371494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:49.871749553 +0000 UTC m=+67.893337319" watchObservedRunningTime="2026-07-16 20:33:49.871903136 +0000 UTC m=+67.893490903"
Jul 16 20:33:56.665555 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.665519    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:56.668118 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.668078    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71aec882-bb2e-4daa-9975-3eb3b1879cfa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-55rbb\" (UID: \"71aec882-bb2e-4daa-9975-3eb3b1879cfa\") " pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:56.896753 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.896712    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g" event={"ID":"a9fdcb17-e49a-4e07-8310-4f4915ef5a51","Type":"ContainerStarted","Data":"8e98fa5d6d34a707cc5e0f333dfa1b30baa35c23f08aec8367bac75b93d433dc"}
Jul 16 20:33:56.896940 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.896768    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g" event={"ID":"a9fdcb17-e49a-4e07-8310-4f4915ef5a51","Type":"ContainerStarted","Data":"80f6fa79b203fa97b62a92156e3345ebbacf24d45a4e695e3341d200e99ac81c"}
Jul 16 20:33:56.896940 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.896786    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g" event={"ID":"a9fdcb17-e49a-4e07-8310-4f4915ef5a51","Type":"ContainerStarted","Data":"e986e42765beea9c9c2eb8287c8aa483bfd5152afa256622d714d1b6856e3b9f"}
Jul 16 20:33:56.898346 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.898313    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh" event={"ID":"d5bde78e-efad-491e-b9b0-59c856a6ac24","Type":"ContainerStarted","Data":"ba55ca9c9134531b183faee17feb66562c74812b9919a8f7d740c0d42205d7b7"}
Jul 16 20:33:56.898450 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.898396    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh"
Jul 16 20:33:56.900296 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.900268    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4mmnt" event={"ID":"864a17b7-c612-43d2-923e-8843e0f519b0","Type":"ContainerStarted","Data":"da91ab47e4c186e70f8b85e7e4d5ea2b81d46c647ee34859901e155b27838ed9"}
Jul 16 20:33:56.900395 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.900356    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh"
Jul 16 20:33:56.900395 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.900381    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4mmnt" event={"ID":"864a17b7-c612-43d2-923e-8843e0f519b0","Type":"ContainerStarted","Data":"28775d2c4c5eabfa5f48338022b06c243bafbbcd09d4bbbbf7f5b967ab7b8844"}
Jul 16 20:33:56.901764 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.901727    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tjnr2" event={"ID":"7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e","Type":"ContainerStarted","Data":"48fb2ade0dca945ec02e763927ce28187daa0e67e299aecdccc0b70b206da9b2"}
Jul 16 20:33:56.903170 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.903146    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vxmfl" event={"ID":"2aed23ed-5e27-4e01-adc6-d8af18c38baa","Type":"ContainerStarted","Data":"854b7e322af761acfff0ba1edba252a9b37d8ac1e9acb6be7e5027ee295bc322"}
Jul 16 20:33:56.903249 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.903207    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:33:56.918387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.918357    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-khzsm\""
Jul 16 20:33:56.920668 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.920611    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7644978f68-hwk4g" podStartSLOduration=2.215168829 podStartE2EDuration="11.92059553s" totalImagesPullingTime="9.705426701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:56.919731295 +0000 UTC m=+74.941319061" watchObservedRunningTime="2026-07-16 20:33:56.92059553 +0000 UTC m=+74.942183327"
Jul 16 20:33:56.926688 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.926662    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-55rbb"
Jul 16 20:33:56.938046 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.937995    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4mmnt" podStartSLOduration=67.303004867 podStartE2EDuration="1m13.937975293s" totalImagesPullingTime="6.634970426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:56.936077413 +0000 UTC m=+74.957665179" watchObservedRunningTime="2026-07-16 20:33:56.937975293 +0000 UTC m=+74.959563057"
Jul 16 20:33:56.954767 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.954709    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vxmfl" podStartSLOduration=67.323584192 podStartE2EDuration="1m13.954692268s" totalImagesPullingTime="6.631108076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:56.953825246 +0000 UTC m=+74.975413013" watchObservedRunningTime="2026-07-16 20:33:56.954692268 +0000 UTC m=+74.976280067"
Jul 16 20:33:56.973900 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:56.973847    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tjnr2" podStartSLOduration=67.28868224 podStartE2EDuration="1m13.97383087s" totalImagesPullingTime="6.68514863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:56.972924253 +0000 UTC m=+74.994512050" watchObservedRunningTime="2026-07-16 20:33:56.97383087 +0000 UTC m=+74.995418635"
Jul 16 20:33:57.020701 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:57.020644    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67f5949cf8-5mxgh" podStartSLOduration=2.299479586 podStartE2EDuration="12.020624192s" totalImagesPullingTime="9.721144606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:33:57.020080813 +0000 UTC m=+75.041668580" watchObservedRunningTime="2026-07-16 20:33:57.020624192 +0000 UTC m=+75.042211997"
Jul 16 20:33:57.081339 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:57.081263    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-55rbb"]
Jul 16 20:33:57.084057 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:33:57.084023    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71aec882_bb2e_4daa_9975_3eb3b1879cfa.slice/crio-251b4be82e35d201e7d5c0377191f5920199116d9b2c100d0a7d5c6e73a1b605 WatchSource:0}: Error finding container 251b4be82e35d201e7d5c0377191f5920199116d9b2c100d0a7d5c6e73a1b605: Status 404 returned error can't find the container with id 251b4be82e35d201e7d5c0377191f5920199116d9b2c100d0a7d5c6e73a1b605
Jul 16 20:33:57.911710 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:33:57.911673    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-55rbb" event={"ID":"71aec882-bb2e-4daa-9975-3eb3b1879cfa","Type":"ContainerStarted","Data":"251b4be82e35d201e7d5c0377191f5920199116d9b2c100d0a7d5c6e73a1b605"}
Jul 16 20:34:02.934082 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:02.934045    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-55rbb" event={"ID":"71aec882-bb2e-4daa-9975-3eb3b1879cfa","Type":"ContainerStarted","Data":"5c4bc7adeaa05d62a8b62969c1ab5d50248cb71b81fd9bff4df9da5a16e6f7e1"}
Jul 16 20:34:03.850264 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:03.850135    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mlgln"]
Jul 16 20:34:03.857548 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:03.857512    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:03.860386 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:03.860358    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 20:34:03.860638 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:03.860621    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zmp64\""
Jul 16 20:34:03.861089 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:03.861072    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 20:34:03.861177 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:03.861141    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 20:34:03.861314 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:03.861301    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 20:34:03.861369 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:03.861336    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 20:34:03.861642 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:03.861497    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 20:34:03.928462 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:03.928246    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9aedd3d-3901-4d0d-ac17-13f5475deee3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:03.928462 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:03.928311    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9aedd3d-3901-4d0d-ac17-13f5475deee3-sys\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:03.928462 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:03.928349    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a9aedd3d-3901-4d0d-ac17-13f5475deee3-node-exporter-accelerators-collector-config\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:03.928462 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:03.928384    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7jfs\" (UniqueName: \"kubernetes.io/projected/a9aedd3d-3901-4d0d-ac17-13f5475deee3-kube-api-access-s7jfs\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:03.928462 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:03.928411    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9aedd3d-3901-4d0d-ac17-13f5475deee3-metrics-client-ca\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:03.928462 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:03.928429    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a9aedd3d-3901-4d0d-ac17-13f5475deee3-node-exporter-wtmp\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:03.928807 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:03.928501    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a9aedd3d-3901-4d0d-ac17-13f5475deee3-node-exporter-textfile\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:03.928807 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:03.928545    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a9aedd3d-3901-4d0d-ac17-13f5475deee3-node-exporter-tls\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:03.928807 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:03.928578    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a9aedd3d-3901-4d0d-ac17-13f5475deee3-root\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.029394 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.029348    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a9aedd3d-3901-4d0d-ac17-13f5475deee3-node-exporter-accelerators-collector-config\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.029856 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.029427    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jfs\" (UniqueName: \"kubernetes.io/projected/a9aedd3d-3901-4d0d-ac17-13f5475deee3-kube-api-access-s7jfs\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.029856 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.029469    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9aedd3d-3901-4d0d-ac17-13f5475deee3-metrics-client-ca\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.029856 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.029485    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a9aedd3d-3901-4d0d-ac17-13f5475deee3-node-exporter-wtmp\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.029856 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.029516    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a9aedd3d-3901-4d0d-ac17-13f5475deee3-node-exporter-textfile\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.029856 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.029547    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a9aedd3d-3901-4d0d-ac17-13f5475deee3-node-exporter-tls\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.029856 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.029579    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a9aedd3d-3901-4d0d-ac17-13f5475deee3-root\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.029856 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.029619    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9aedd3d-3901-4d0d-ac17-13f5475deee3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.029856 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.029650    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9aedd3d-3901-4d0d-ac17-13f5475deee3-sys\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.029856 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.029721    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9aedd3d-3901-4d0d-ac17-13f5475deee3-sys\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.029856 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.029836    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a9aedd3d-3901-4d0d-ac17-13f5475deee3-node-exporter-wtmp\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.030321 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.029880    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a9aedd3d-3901-4d0d-ac17-13f5475deee3-root\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.030321 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.029903    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a9aedd3d-3901-4d0d-ac17-13f5475deee3-node-exporter-accelerators-collector-config\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.030321 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.030132    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a9aedd3d-3901-4d0d-ac17-13f5475deee3-node-exporter-textfile\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.030321 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.030132    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9aedd3d-3901-4d0d-ac17-13f5475deee3-metrics-client-ca\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.032467 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.032447    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9aedd3d-3901-4d0d-ac17-13f5475deee3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.032587 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.032565    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a9aedd3d-3901-4d0d-ac17-13f5475deee3-node-exporter-tls\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.039453 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.039404    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7jfs\" (UniqueName: \"kubernetes.io/projected/a9aedd3d-3901-4d0d-ac17-13f5475deee3-kube-api-access-s7jfs\") pod \"node-exporter-mlgln\" (UID: \"a9aedd3d-3901-4d0d-ac17-13f5475deee3\") " pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:04.171334 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:04.171240    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mlgln"
Jul 16 20:34:05.484686 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:34:05.483884    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9aedd3d_3901_4d0d_ac17_13f5475deee3.slice/crio-f1993b77108fcff6a9aac4e4b0a626c766597d732ff14db96586142c3899d4db WatchSource:0}: Error finding container f1993b77108fcff6a9aac4e4b0a626c766597d732ff14db96586142c3899d4db: Status 404 returned error can't find the container with id f1993b77108fcff6a9aac4e4b0a626c766597d732ff14db96586142c3899d4db
Jul 16 20:34:05.954225 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:05.954173    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-55rbb" event={"ID":"71aec882-bb2e-4daa-9975-3eb3b1879cfa","Type":"ContainerStarted","Data":"beb9391fee4a8bcce2a2204b837620bfac4de77836aa0635adc3b3d8403ab21c"}
Jul 16 20:34:05.955413 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:05.955380    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mlgln" event={"ID":"a9aedd3d-3901-4d0d-ac17-13f5475deee3","Type":"ContainerStarted","Data":"f1993b77108fcff6a9aac4e4b0a626c766597d732ff14db96586142c3899d4db"}
Jul 16 20:34:06.961522 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:06.961481    2530 generic.go:356] "Generic (PLEG): container finished" podID="a9aedd3d-3901-4d0d-ac17-13f5475deee3" containerID="6ccf046c249fdb9d644c7f8e52a1cfeba142a0018a1b1046cfc1da21eb48afc7" exitCode=0
Jul 16 20:34:06.961999 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:06.961601    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mlgln" event={"ID":"a9aedd3d-3901-4d0d-ac17-13f5475deee3","Type":"ContainerDied","Data":"6ccf046c249fdb9d644c7f8e52a1cfeba142a0018a1b1046cfc1da21eb48afc7"}
Jul 16 20:34:07.847375 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:07.847345    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-85d4cdcc5d-h29lp"
Jul 16 20:34:07.968614 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:07.968571    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mlgln" event={"ID":"a9aedd3d-3901-4d0d-ac17-13f5475deee3","Type":"ContainerStarted","Data":"5d1f0405f76df01327d90a498859d45c3bbdb0e7f55dece1a0bcc6023321223e"}
Jul 16 20:34:07.969025 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:07.968653    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mlgln" event={"ID":"a9aedd3d-3901-4d0d-ac17-13f5475deee3","Type":"ContainerStarted","Data":"c9d89ca3768b04c5b3082fe811b926b183401f7f927c30f180b249c7c13358d3"}
Jul 16 20:34:07.996877 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:07.996822    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mlgln" podStartSLOduration=3.918851983 podStartE2EDuration="4.996805091s" totalImagesPullingTime="994.658169ms" totalInitContainerRuntime="83.294939ms" isStatefulPod=true podCreationTimestamp="2026-07-16 20:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:07.995536106 +0000 UTC m=+86.017123871" watchObservedRunningTime="2026-07-16 20:34:07.996805091 +0000 UTC m=+86.018392856"
Jul 16 20:34:09.978012 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:09.977972    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-55rbb" event={"ID":"71aec882-bb2e-4daa-9975-3eb3b1879cfa","Type":"ContainerStarted","Data":"0dbee381ddf28ee7e8ba5edbbd5e9b1544b57938ea98021e26e5685ec23e8611"}
Jul 16 20:34:09.997069 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:09.997016    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-55rbb" podStartSLOduration=33.984319046 podStartE2EDuration="45.997000683s" totalImagesPullingTime="12.012681637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 20:33:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:34:09.9965701 +0000 UTC m=+88.018157867" watchObservedRunningTime="2026-07-16 20:34:09.997000683 +0000 UTC m=+88.018588448"
Jul 16 20:34:27.914690 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:27.914660    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vxmfl"
Jul 16 20:34:55.333752 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:34:55.333716    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hmjxf_a949c15b-79d3-42e2-ba6f-aaa345dd0bcf/serve-healthcheck-canary/0.log"
Jul 16 20:37:42.378291 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:37:42.378257    2530 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 20:38:30.222440 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.222403    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"]
Jul 16 20:38:30.224644 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.224628    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:38:30.227478 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.227455    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 16 20:38:30.227603 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.227479    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 16 20:38:30.227603 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.227455    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 16 20:38:30.227603 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.227484    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 16 20:38:30.227798 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.227784    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qrv96\""
Jul 16 20:38:30.243637 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.243614    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"]
Jul 16 20:38:30.310845 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.310812    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34fa3be8-1154-4bdd-bdd5-690f98005299-tmp\") pod \"opendatahub-operator-controller-manager-58fdbbf655-zgnxb\" (UID: \"34fa3be8-1154-4bdd-bdd5-690f98005299\") " pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:38:30.310845 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.310845    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34fa3be8-1154-4bdd-bdd5-690f98005299-webhook-cert\") pod \"opendatahub-operator-controller-manager-58fdbbf655-zgnxb\" (UID: \"34fa3be8-1154-4bdd-bdd5-690f98005299\") " pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:38:30.311077 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.310866    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpbdz\" (UniqueName: \"kubernetes.io/projected/34fa3be8-1154-4bdd-bdd5-690f98005299-kube-api-access-qpbdz\") pod \"opendatahub-operator-controller-manager-58fdbbf655-zgnxb\" (UID: \"34fa3be8-1154-4bdd-bdd5-690f98005299\") " pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:38:30.311077 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.310900    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/34fa3be8-1154-4bdd-bdd5-690f98005299-manifests\") pod \"opendatahub-operator-controller-manager-58fdbbf655-zgnxb\" (UID: \"34fa3be8-1154-4bdd-bdd5-690f98005299\") " pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:38:30.311077 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.311006    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/34fa3be8-1154-4bdd-bdd5-690f98005299-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58fdbbf655-zgnxb\" (UID: \"34fa3be8-1154-4bdd-bdd5-690f98005299\") " pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:38:30.411995 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.411954    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/34fa3be8-1154-4bdd-bdd5-690f98005299-manifests\") pod \"opendatahub-operator-controller-manager-58fdbbf655-zgnxb\" (UID: \"34fa3be8-1154-4bdd-bdd5-690f98005299\") " pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:38:30.412226 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.412013    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/34fa3be8-1154-4bdd-bdd5-690f98005299-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58fdbbf655-zgnxb\" (UID: \"34fa3be8-1154-4bdd-bdd5-690f98005299\") " pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:38:30.412311 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.412285    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34fa3be8-1154-4bdd-bdd5-690f98005299-tmp\") pod \"opendatahub-operator-controller-manager-58fdbbf655-zgnxb\" (UID: \"34fa3be8-1154-4bdd-bdd5-690f98005299\") " pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:38:30.412356 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.412337    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34fa3be8-1154-4bdd-bdd5-690f98005299-webhook-cert\") pod \"opendatahub-operator-controller-manager-58fdbbf655-zgnxb\" (UID: \"34fa3be8-1154-4bdd-bdd5-690f98005299\") " pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:38:30.412393 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.412367    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbdz\" (UniqueName: \"kubernetes.io/projected/34fa3be8-1154-4bdd-bdd5-690f98005299-kube-api-access-qpbdz\") pod \"opendatahub-operator-controller-manager-58fdbbf655-zgnxb\" (UID: \"34fa3be8-1154-4bdd-bdd5-690f98005299\") " pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:38:30.412510 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.412442    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/34fa3be8-1154-4bdd-bdd5-690f98005299-manifests\") pod \"opendatahub-operator-controller-manager-58fdbbf655-zgnxb\" (UID: \"34fa3be8-1154-4bdd-bdd5-690f98005299\") " pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:38:30.414586 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.414556    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34fa3be8-1154-4bdd-bdd5-690f98005299-tmp\") pod \"opendatahub-operator-controller-manager-58fdbbf655-zgnxb\" (UID: \"34fa3be8-1154-4bdd-bdd5-690f98005299\") " pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:38:30.414712 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.414656    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/34fa3be8-1154-4bdd-bdd5-690f98005299-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58fdbbf655-zgnxb\" (UID: \"34fa3be8-1154-4bdd-bdd5-690f98005299\") " pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:38:30.414835 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.414816    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34fa3be8-1154-4bdd-bdd5-690f98005299-webhook-cert\") pod \"opendatahub-operator-controller-manager-58fdbbf655-zgnxb\" (UID: \"34fa3be8-1154-4bdd-bdd5-690f98005299\") " pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:38:30.423652 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.423627    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpbdz\" (UniqueName: \"kubernetes.io/projected/34fa3be8-1154-4bdd-bdd5-690f98005299-kube-api-access-qpbdz\") pod \"opendatahub-operator-controller-manager-58fdbbf655-zgnxb\" (UID: \"34fa3be8-1154-4bdd-bdd5-690f98005299\") " pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:38:30.534276 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.534199    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:38:30.664775 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.664745    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"]
Jul 16 20:38:30.668701 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:38:30.668678    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34fa3be8_1154_4bdd_bdd5_690f98005299.slice/crio-0840db529a9dd96235f344b9e9fe3335bd342449d818d750e34a442ee1ba0ecc WatchSource:0}: Error finding container 0840db529a9dd96235f344b9e9fe3335bd342449d818d750e34a442ee1ba0ecc: Status 404 returned error can't find the container with id 0840db529a9dd96235f344b9e9fe3335bd342449d818d750e34a442ee1ba0ecc
Jul 16 20:38:30.670442 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.670426    2530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 20:38:30.833689 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:30.833619    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb" event={"ID":"34fa3be8-1154-4bdd-bdd5-690f98005299","Type":"ContainerStarted","Data":"0840db529a9dd96235f344b9e9fe3335bd342449d818d750e34a442ee1ba0ecc"}
Jul 16 20:38:33.846418 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:33.846378    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb" event={"ID":"34fa3be8-1154-4bdd-bdd5-690f98005299","Type":"ContainerStarted","Data":"78172931be452481ddd635757fce54839089172569ca8a0f3a0ec90492b3f10f"}
Jul 16 20:38:34.851630 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:34.851596    2530 generic.go:356] "Generic (PLEG): container finished" podID="34fa3be8-1154-4bdd-bdd5-690f98005299" containerID="78172931be452481ddd635757fce54839089172569ca8a0f3a0ec90492b3f10f" exitCode=0
Jul 16 20:38:34.851995 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:34.851652    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb" event={"ID":"34fa3be8-1154-4bdd-bdd5-690f98005299","Type":"ContainerDied","Data":"78172931be452481ddd635757fce54839089172569ca8a0f3a0ec90492b3f10f"}
Jul 16 20:38:35.863749 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:35.863707    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb" event={"ID":"34fa3be8-1154-4bdd-bdd5-690f98005299","Type":"ContainerStarted","Data":"a791e6f0c2334f32fb70e6f2c9c706c60f6b55efb0418c633ef90254ec27f5e6"}
Jul 16 20:38:35.864338 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:35.863922    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:38:35.890064 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:35.890012    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb" podStartSLOduration=1.5921074160000002 podStartE2EDuration="5.889995849s" totalImagesPullingTime="3.246082236s" totalInitContainerRuntime="1.051806197s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:38:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:38:35.888288652 +0000 UTC m=+353.909876419" watchObservedRunningTime="2026-07-16 20:38:35.889995849 +0000 UTC m=+353.911583616"
Jul 16 20:38:46.869332 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:38:46.869303    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-58fdbbf655-zgnxb"
Jul 16 20:39:00.264168 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.264134    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-tkrwz"]
Jul 16 20:39:00.276007 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.275974    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-sdzsg"]
Jul 16 20:39:00.276176 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.276123    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrwz"
Jul 16 20:39:00.277938 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.277915    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sdzsg"
Jul 16 20:39:00.281971 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.281951    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 16 20:39:00.282138 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.281982    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-dtvt5\""
Jul 16 20:39:00.282138 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.281950    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-nvvjs\""
Jul 16 20:39:00.282856 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.282840    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 16 20:39:00.284637 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.284389    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-tkrwz"]
Jul 16 20:39:00.297650 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.297629    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-sdzsg"]
Jul 16 20:39:00.439459 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.439433    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4hlg\" (UniqueName: \"kubernetes.io/projected/88af9955-7488-44ce-a8a4-2589b1b302ae-kube-api-access-c4hlg\") pod \"kserve-controller-manager-d6f4b65b8-tkrwz\" (UID: \"88af9955-7488-44ce-a8a4-2589b1b302ae\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrwz"
Jul 16 20:39:00.439623 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.439477    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzmfb\" (UniqueName: \"kubernetes.io/projected/a8f11bc3-e9ed-4832-b76f-94e5e31522dd-kube-api-access-rzmfb\") pod \"llmisvc-controller-manager-777c6bd7d5-sdzsg\" (UID: \"a8f11bc3-e9ed-4832-b76f-94e5e31522dd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sdzsg"
Jul 16 20:39:00.439623 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.439587    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8f11bc3-e9ed-4832-b76f-94e5e31522dd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-sdzsg\" (UID: \"a8f11bc3-e9ed-4832-b76f-94e5e31522dd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sdzsg"
Jul 16 20:39:00.439732 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.439656    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88af9955-7488-44ce-a8a4-2589b1b302ae-cert\") pod \"kserve-controller-manager-d6f4b65b8-tkrwz\" (UID: \"88af9955-7488-44ce-a8a4-2589b1b302ae\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrwz"
Jul 16 20:39:00.540894 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.540820    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8f11bc3-e9ed-4832-b76f-94e5e31522dd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-sdzsg\" (UID: \"a8f11bc3-e9ed-4832-b76f-94e5e31522dd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sdzsg"
Jul 16 20:39:00.540894 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.540871    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88af9955-7488-44ce-a8a4-2589b1b302ae-cert\") pod \"kserve-controller-manager-d6f4b65b8-tkrwz\" (UID: \"88af9955-7488-44ce-a8a4-2589b1b302ae\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrwz"
Jul 16 20:39:00.541133 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.540902    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4hlg\" (UniqueName: \"kubernetes.io/projected/88af9955-7488-44ce-a8a4-2589b1b302ae-kube-api-access-c4hlg\") pod \"kserve-controller-manager-d6f4b65b8-tkrwz\" (UID: \"88af9955-7488-44ce-a8a4-2589b1b302ae\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrwz"
Jul 16 20:39:00.541133 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.540937    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzmfb\" (UniqueName: \"kubernetes.io/projected/a8f11bc3-e9ed-4832-b76f-94e5e31522dd-kube-api-access-rzmfb\") pod \"llmisvc-controller-manager-777c6bd7d5-sdzsg\" (UID: \"a8f11bc3-e9ed-4832-b76f-94e5e31522dd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sdzsg"
Jul 16 20:39:00.541133 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:39:00.541019    2530 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 16 20:39:00.541133 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:39:00.541091    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88af9955-7488-44ce-a8a4-2589b1b302ae-cert podName:88af9955-7488-44ce-a8a4-2589b1b302ae nodeName:}" failed. No retries permitted until 2026-07-16 20:39:01.041071236 +0000 UTC m=+379.062658985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/88af9955-7488-44ce-a8a4-2589b1b302ae-cert") pod "kserve-controller-manager-d6f4b65b8-tkrwz" (UID: "88af9955-7488-44ce-a8a4-2589b1b302ae") : secret "kserve-webhook-server-cert" not found
Jul 16 20:39:00.543473 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.543450    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8f11bc3-e9ed-4832-b76f-94e5e31522dd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-sdzsg\" (UID: \"a8f11bc3-e9ed-4832-b76f-94e5e31522dd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sdzsg"
Jul 16 20:39:00.555977 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.555950    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4hlg\" (UniqueName: \"kubernetes.io/projected/88af9955-7488-44ce-a8a4-2589b1b302ae-kube-api-access-c4hlg\") pod \"kserve-controller-manager-d6f4b65b8-tkrwz\" (UID: \"88af9955-7488-44ce-a8a4-2589b1b302ae\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrwz"
Jul 16 20:39:00.571212 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.571192    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzmfb\" (UniqueName: \"kubernetes.io/projected/a8f11bc3-e9ed-4832-b76f-94e5e31522dd-kube-api-access-rzmfb\") pod \"llmisvc-controller-manager-777c6bd7d5-sdzsg\" (UID: \"a8f11bc3-e9ed-4832-b76f-94e5e31522dd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sdzsg"
Jul 16 20:39:00.597955 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.597919    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sdzsg"
Jul 16 20:39:00.614393 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.614373    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-74779b8bd6-b48mq"]
Jul 16 20:39:00.616596 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.616578    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-74779b8bd6-b48mq"
Jul 16 20:39:00.620219 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.620191    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 16 20:39:00.620354 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.620339    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 16 20:39:00.620744 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.620726    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 16 20:39:00.622826 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.622808    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 16 20:39:00.634385 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.634360    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-74779b8bd6-b48mq"]
Jul 16 20:39:00.734931 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.734899    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-sdzsg"]
Jul 16 20:39:00.739643 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:39:00.739618    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda8f11bc3_e9ed_4832_b76f_94e5e31522dd.slice/crio-8dbb8513b70e9a2595eb26f620a9e4f5e41de0b482894180031e1db9f8723e28 WatchSource:0}: Error finding container 8dbb8513b70e9a2595eb26f620a9e4f5e41de0b482894180031e1db9f8723e28: Status 404 returned error can't find the container with id 8dbb8513b70e9a2595eb26f620a9e4f5e41de0b482894180031e1db9f8723e28
Jul 16 20:39:00.742675 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.742657    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/487f8c35-e7db-47aa-aa92-b9187479bae0-tmp\") pod \"kube-auth-proxy-74779b8bd6-b48mq\" (UID: \"487f8c35-e7db-47aa-aa92-b9187479bae0\") " pod="openshift-ingress/kube-auth-proxy-74779b8bd6-b48mq"
Jul 16 20:39:00.742751 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.742690    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc86f\" (UniqueName: \"kubernetes.io/projected/487f8c35-e7db-47aa-aa92-b9187479bae0-kube-api-access-zc86f\") pod \"kube-auth-proxy-74779b8bd6-b48mq\" (UID: \"487f8c35-e7db-47aa-aa92-b9187479bae0\") " pod="openshift-ingress/kube-auth-proxy-74779b8bd6-b48mq"
Jul 16 20:39:00.742796 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.742785    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/487f8c35-e7db-47aa-aa92-b9187479bae0-tls-certs\") pod \"kube-auth-proxy-74779b8bd6-b48mq\" (UID: \"487f8c35-e7db-47aa-aa92-b9187479bae0\") " pod="openshift-ingress/kube-auth-proxy-74779b8bd6-b48mq"
Jul 16 20:39:00.843399 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.843335    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/487f8c35-e7db-47aa-aa92-b9187479bae0-tls-certs\") pod \"kube-auth-proxy-74779b8bd6-b48mq\" (UID: \"487f8c35-e7db-47aa-aa92-b9187479bae0\") " pod="openshift-ingress/kube-auth-proxy-74779b8bd6-b48mq"
Jul 16 20:39:00.843517 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.843404    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/487f8c35-e7db-47aa-aa92-b9187479bae0-tmp\") pod \"kube-auth-proxy-74779b8bd6-b48mq\" (UID: \"487f8c35-e7db-47aa-aa92-b9187479bae0\") " pod="openshift-ingress/kube-auth-proxy-74779b8bd6-b48mq"
Jul 16 20:39:00.843517 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.843440    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc86f\" (UniqueName: \"kubernetes.io/projected/487f8c35-e7db-47aa-aa92-b9187479bae0-kube-api-access-zc86f\") pod \"kube-auth-proxy-74779b8bd6-b48mq\" (UID: \"487f8c35-e7db-47aa-aa92-b9187479bae0\") " pod="openshift-ingress/kube-auth-proxy-74779b8bd6-b48mq"
Jul 16 20:39:00.845775 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.845749    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/487f8c35-e7db-47aa-aa92-b9187479bae0-tmp\") pod \"kube-auth-proxy-74779b8bd6-b48mq\" (UID: \"487f8c35-e7db-47aa-aa92-b9187479bae0\") " pod="openshift-ingress/kube-auth-proxy-74779b8bd6-b48mq"
Jul 16 20:39:00.845883 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.845867    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/487f8c35-e7db-47aa-aa92-b9187479bae0-tls-certs\") pod \"kube-auth-proxy-74779b8bd6-b48mq\" (UID: \"487f8c35-e7db-47aa-aa92-b9187479bae0\") " pod="openshift-ingress/kube-auth-proxy-74779b8bd6-b48mq"
Jul 16 20:39:00.853436 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.853419    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc86f\" (UniqueName: \"kubernetes.io/projected/487f8c35-e7db-47aa-aa92-b9187479bae0-kube-api-access-zc86f\") pod \"kube-auth-proxy-74779b8bd6-b48mq\" (UID: \"487f8c35-e7db-47aa-aa92-b9187479bae0\") " pod="openshift-ingress/kube-auth-proxy-74779b8bd6-b48mq"
Jul 16 20:39:00.926499 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.926479    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-74779b8bd6-b48mq"
Jul 16 20:39:00.948036 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:00.948008    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sdzsg" event={"ID":"a8f11bc3-e9ed-4832-b76f-94e5e31522dd","Type":"ContainerStarted","Data":"8dbb8513b70e9a2595eb26f620a9e4f5e41de0b482894180031e1db9f8723e28"}
Jul 16 20:39:01.044574 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:01.044546    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88af9955-7488-44ce-a8a4-2589b1b302ae-cert\") pod \"kserve-controller-manager-d6f4b65b8-tkrwz\" (UID: \"88af9955-7488-44ce-a8a4-2589b1b302ae\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrwz"
Jul 16 20:39:01.046881 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:01.046858    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/88af9955-7488-44ce-a8a4-2589b1b302ae-cert\") pod \"kserve-controller-manager-d6f4b65b8-tkrwz\" (UID: \"88af9955-7488-44ce-a8a4-2589b1b302ae\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrwz"
Jul 16 20:39:01.067601 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:01.067579    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-74779b8bd6-b48mq"]
Jul 16 20:39:01.068073 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:39:01.068046    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487f8c35_e7db_47aa_aa92_b9187479bae0.slice/crio-50d64c0e929bf7ef1427401af495b0b809837f9708109d5040b813c4997c8778 WatchSource:0}: Error finding container 50d64c0e929bf7ef1427401af495b0b809837f9708109d5040b813c4997c8778: Status 404 returned error can't find the container with id 50d64c0e929bf7ef1427401af495b0b809837f9708109d5040b813c4997c8778
Jul 16 20:39:01.189762 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:01.189690    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrwz"
Jul 16 20:39:01.315943 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:01.315920    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-tkrwz"]
Jul 16 20:39:01.318194 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:39:01.318157    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88af9955_7488_44ce_a8a4_2589b1b302ae.slice/crio-29701d596f7346b4238fec91100696c294dec06af4bf2037ce567bd105cdd7f7 WatchSource:0}: Error finding container 29701d596f7346b4238fec91100696c294dec06af4bf2037ce567bd105cdd7f7: Status 404 returned error can't find the container with id 29701d596f7346b4238fec91100696c294dec06af4bf2037ce567bd105cdd7f7
Jul 16 20:39:01.954754 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:01.954681    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrwz" event={"ID":"88af9955-7488-44ce-a8a4-2589b1b302ae","Type":"ContainerStarted","Data":"29701d596f7346b4238fec91100696c294dec06af4bf2037ce567bd105cdd7f7"}
Jul 16 20:39:01.956827 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:01.956792    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-74779b8bd6-b48mq" event={"ID":"487f8c35-e7db-47aa-aa92-b9187479bae0","Type":"ContainerStarted","Data":"50d64c0e929bf7ef1427401af495b0b809837f9708109d5040b813c4997c8778"}
Jul 16 20:39:03.036135 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.036087    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"]
Jul 16 20:39:03.039498 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.039477    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.043445 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.043419    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 16 20:39:03.043445 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.043422    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 16 20:39:03.043633 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.043452    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 16 20:39:03.043633 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.043422    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 16 20:39:03.043633 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.043423    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bcrsz\""
Jul 16 20:39:03.065982 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.065959    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"]
Jul 16 20:39:03.164968 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.164921    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47t8b\" (UniqueName: \"kubernetes.io/projected/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-kube-api-access-47t8b\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.165169 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.164983    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.165169 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.165064    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.165169 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.165119    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.165169 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.165158    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.165357 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.165179    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.165357 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.165195    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.266439 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.266393    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.266613 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.266472    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.266613 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.266506    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.266613 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.266549    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.266613 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.266578    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.266613 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.266600    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.266876 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.266654    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47t8b\" (UniqueName: \"kubernetes.io/projected/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-kube-api-access-47t8b\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.267340 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.267289    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.269913 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.269874    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.270029 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.269917    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.270165 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.270141    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.270413 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.270395    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.284289 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.284266    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.285810 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.285777    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47t8b\" (UniqueName: \"kubernetes.io/projected/1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96-kube-api-access-47t8b\") pod \"istiod-openshift-gateway-94bb8fbfd-czqvm\" (UID: \"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:03.350659 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:03.350579    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:04.825467 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:04.825441    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"]
Jul 16 20:39:04.825867 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:39:04.825843    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc66d7b_0db0_4b3a_88a1_b14e9f6eaf96.slice/crio-d886b66f5a361f7bd4883c9ef3d2d7134092874d51ef077496bd57cf3f6898a9 WatchSource:0}: Error finding container d886b66f5a361f7bd4883c9ef3d2d7134092874d51ef077496bd57cf3f6898a9: Status 404 returned error can't find the container with id d886b66f5a361f7bd4883c9ef3d2d7134092874d51ef077496bd57cf3f6898a9
Jul 16 20:39:04.972428 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:04.972384    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm" event={"ID":"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96","Type":"ContainerStarted","Data":"d886b66f5a361f7bd4883c9ef3d2d7134092874d51ef077496bd57cf3f6898a9"}
Jul 16 20:39:04.974044 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:04.974009    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sdzsg" event={"ID":"a8f11bc3-e9ed-4832-b76f-94e5e31522dd","Type":"ContainerStarted","Data":"8f5faf6127f01fc51ed289127441aaa818d2ac3e1a1fe5156d55f3350406bb9b"}
Jul 16 20:39:04.974213 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:04.974151    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sdzsg"
Jul 16 20:39:04.975629 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:04.975601    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrwz" event={"ID":"88af9955-7488-44ce-a8a4-2589b1b302ae","Type":"ContainerStarted","Data":"38bef8ebb3c7a80a9bb6efdd0ba1d43bae35ad03b2adad0e4816adb77c9a377e"}
Jul 16 20:39:04.975772 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:04.975654    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrwz"
Jul 16 20:39:04.977086 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:04.977064    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-74779b8bd6-b48mq" event={"ID":"487f8c35-e7db-47aa-aa92-b9187479bae0","Type":"ContainerStarted","Data":"0ab226be6530477ec2245186fd91e9d3f08fbf2cea19f3780b5eba76a06d2569"}
Jul 16 20:39:05.010243 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:05.010203    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sdzsg" podStartSLOduration=1.080439157 podStartE2EDuration="5.010188318s" totalImagesPullingTime="3.929749161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:39:05.009690182 +0000 UTC m=+383.031277948" watchObservedRunningTime="2026-07-16 20:39:05.010188318 +0000 UTC m=+383.031776062"
Jul 16 20:39:05.037073 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:05.037023    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrwz" podStartSLOduration=1.680984356 podStartE2EDuration="5.037004658s" totalImagesPullingTime="3.356020302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:39:05.03547504 +0000 UTC m=+383.057062807" watchObservedRunningTime="2026-07-16 20:39:05.037004658 +0000 UTC m=+383.058592425"
Jul 16 20:39:05.063225 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:05.063180    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-74779b8bd6-b48mq" podStartSLOduration=1.463118853 podStartE2EDuration="5.063165147s" totalImagesPullingTime="3.600046294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:39:05.062203885 +0000 UTC m=+383.083791653" watchObservedRunningTime="2026-07-16 20:39:05.063165147 +0000 UTC m=+383.084752913"
Jul 16 20:39:07.274943 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:07.274906    2530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:39:07.275307 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:07.274973    2530 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 20:39:07.995968 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:07.995931    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm" event={"ID":"1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96","Type":"ContainerStarted","Data":"d834ff97105ff443faa939f443719ec32be47d092a63a9a18a04167db1af0459"}
Jul 16 20:39:07.996228 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:07.996209    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:08.035953 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:08.035909    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm" podStartSLOduration=2.589541042 podStartE2EDuration="5.035894594s" totalImagesPullingTime="2.446353552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:39:08.03123006 +0000 UTC m=+386.052817851" watchObservedRunningTime="2026-07-16 20:39:08.035894594 +0000 UTC m=+386.057482360"
Jul 16 20:39:09.002353 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:09.002320    2530 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-czqvm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 20:39:09.002709 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:09.002395    2530 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm" podUID="1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 20:39:12.001774 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:12.001743    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-czqvm"
Jul 16 20:39:35.983237 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:35.983204    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sdzsg"
Jul 16 20:39:35.986203 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:39:35.986184    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-tkrwz"
Jul 16 20:40:34.163457 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:34.163420    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"]
Jul 16 20:40:34.165694 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:34.165678    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"
Jul 16 20:40:34.168680 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:34.168662    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 20:40:34.169383 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:34.169359    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-pbbv6\""
Jul 16 20:40:34.169481 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:34.169425    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 20:40:34.184619 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:34.184590    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"]
Jul 16 20:40:34.332392 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:34.332364    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8cv8\" (UniqueName: \"kubernetes.io/projected/5fe19fc5-a082-42cf-9be1-3ee7297e864a-kube-api-access-p8cv8\") pod \"limitador-operator-controller-manager-859d7b78bd-f4cwv\" (UID: \"5fe19fc5-a082-42cf-9be1-3ee7297e864a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"
Jul 16 20:40:34.433563 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:34.433489    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8cv8\" (UniqueName: \"kubernetes.io/projected/5fe19fc5-a082-42cf-9be1-3ee7297e864a-kube-api-access-p8cv8\") pod \"limitador-operator-controller-manager-859d7b78bd-f4cwv\" (UID: \"5fe19fc5-a082-42cf-9be1-3ee7297e864a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"
Jul 16 20:40:34.443741 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:34.443718    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8cv8\" (UniqueName: \"kubernetes.io/projected/5fe19fc5-a082-42cf-9be1-3ee7297e864a-kube-api-access-p8cv8\") pod \"limitador-operator-controller-manager-859d7b78bd-f4cwv\" (UID: \"5fe19fc5-a082-42cf-9be1-3ee7297e864a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"
Jul 16 20:40:34.475708 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:34.475678    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"
Jul 16 20:40:34.620166 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:34.620087    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"]
Jul 16 20:40:34.622748 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:40:34.622721    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe19fc5_a082_42cf_9be1_3ee7297e864a.slice/crio-482c6d8c6ad0ac0a0b6d92f63cccc968ebea5ad6019ecdc71192f133b222176a WatchSource:0}: Error finding container 482c6d8c6ad0ac0a0b6d92f63cccc968ebea5ad6019ecdc71192f133b222176a: Status 404 returned error can't find the container with id 482c6d8c6ad0ac0a0b6d92f63cccc968ebea5ad6019ecdc71192f133b222176a
Jul 16 20:40:35.337870 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:35.337840    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv" event={"ID":"5fe19fc5-a082-42cf-9be1-3ee7297e864a","Type":"ContainerStarted","Data":"482c6d8c6ad0ac0a0b6d92f63cccc968ebea5ad6019ecdc71192f133b222176a"}
Jul 16 20:40:37.347801 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:37.347761    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv" event={"ID":"5fe19fc5-a082-42cf-9be1-3ee7297e864a","Type":"ContainerStarted","Data":"abd306fa87320cd338b2c40fc8817eb10595d108dd887f31c6c0726243c448a7"}
Jul 16 20:40:37.348262 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:37.347952    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"
Jul 16 20:40:37.374524 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:37.374475    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv" podStartSLOduration=1.490386866 podStartE2EDuration="3.3744627s" totalImagesPullingTime="1.884075834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:40:37.372634946 +0000 UTC m=+475.394222714" watchObservedRunningTime="2026-07-16 20:40:37.3744627 +0000 UTC m=+475.396050466"
Jul 16 20:40:48.354324 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:48.354282    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"
Jul 16 20:40:49.117548 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.117518    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"]
Jul 16 20:40:49.117730 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.117709    2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv" podUID="5fe19fc5-a082-42cf-9be1-3ee7297e864a" containerName="manager" containerID="cri-o://abd306fa87320cd338b2c40fc8817eb10595d108dd887f31c6c0726243c448a7" gracePeriod=2
Jul 16 20:40:49.142428 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.142396    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"]
Jul 16 20:40:49.158680 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.158645    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bxpjf"]
Jul 16 20:40:49.159000 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.158976    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe19fc5-a082-42cf-9be1-3ee7297e864a" containerName="manager"
Jul 16 20:40:49.159000 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.158993    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe19fc5-a082-42cf-9be1-3ee7297e864a" containerName="manager"
Jul 16 20:40:49.159211 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.159046    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe19fc5-a082-42cf-9be1-3ee7297e864a" containerName="manager"
Jul 16 20:40:49.160856 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.160838    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bxpjf"
Jul 16 20:40:49.178053 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.178026    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bxpjf"]
Jul 16 20:40:49.194806 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:40:49.194767    2530 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-f4cwv\" is forbidden: User \"system:node:ip-10-0-136-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-249.ec2.internal' and this object" podUID="5fe19fc5-a082-42cf-9be1-3ee7297e864a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"
Jul 16 20:40:49.242023 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.242000    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x8lp\" (UniqueName: \"kubernetes.io/projected/76444973-2719-4125-aa2d-663c2758b13e-kube-api-access-8x8lp\") pod \"limitador-operator-controller-manager-859d7b78bd-bxpjf\" (UID: \"76444973-2719-4125-aa2d-663c2758b13e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bxpjf"
Jul 16 20:40:49.343133 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.343086    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x8lp\" (UniqueName: \"kubernetes.io/projected/76444973-2719-4125-aa2d-663c2758b13e-kube-api-access-8x8lp\") pod \"limitador-operator-controller-manager-859d7b78bd-bxpjf\" (UID: \"76444973-2719-4125-aa2d-663c2758b13e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bxpjf"
Jul 16 20:40:49.364660 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.364635    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x8lp\" (UniqueName: \"kubernetes.io/projected/76444973-2719-4125-aa2d-663c2758b13e-kube-api-access-8x8lp\") pod \"limitador-operator-controller-manager-859d7b78bd-bxpjf\" (UID: \"76444973-2719-4125-aa2d-663c2758b13e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bxpjf"
Jul 16 20:40:49.365286 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.365266    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"
Jul 16 20:40:49.367371 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:40:49.367343    2530 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-f4cwv\" is forbidden: User \"system:node:ip-10-0-136-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-249.ec2.internal' and this object" podUID="5fe19fc5-a082-42cf-9be1-3ee7297e864a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"
Jul 16 20:40:49.394991 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.394921    2530 generic.go:356] "Generic (PLEG): container finished" podID="5fe19fc5-a082-42cf-9be1-3ee7297e864a" containerID="abd306fa87320cd338b2c40fc8817eb10595d108dd887f31c6c0726243c448a7" exitCode=0
Jul 16 20:40:49.394991 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.394976    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"
Jul 16 20:40:49.395184 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.394980    2530 scope.go:117] "RemoveContainer" containerID="abd306fa87320cd338b2c40fc8817eb10595d108dd887f31c6c0726243c448a7"
Jul 16 20:40:49.397648 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:40:49.397623    2530 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-f4cwv\" is forbidden: User \"system:node:ip-10-0-136-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-249.ec2.internal' and this object" podUID="5fe19fc5-a082-42cf-9be1-3ee7297e864a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"
Jul 16 20:40:49.402986 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.402969    2530 scope.go:117] "RemoveContainer" containerID="abd306fa87320cd338b2c40fc8817eb10595d108dd887f31c6c0726243c448a7"
Jul 16 20:40:49.403332 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:40:49.403312    2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abd306fa87320cd338b2c40fc8817eb10595d108dd887f31c6c0726243c448a7\": container with ID starting with abd306fa87320cd338b2c40fc8817eb10595d108dd887f31c6c0726243c448a7 not found: ID does not exist" containerID="abd306fa87320cd338b2c40fc8817eb10595d108dd887f31c6c0726243c448a7"
Jul 16 20:40:49.403401 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.403345    2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abd306fa87320cd338b2c40fc8817eb10595d108dd887f31c6c0726243c448a7"} err="failed to get container status \"abd306fa87320cd338b2c40fc8817eb10595d108dd887f31c6c0726243c448a7\": rpc error: code = NotFound desc = could not find container \"abd306fa87320cd338b2c40fc8817eb10595d108dd887f31c6c0726243c448a7\": container with ID starting with abd306fa87320cd338b2c40fc8817eb10595d108dd887f31c6c0726243c448a7 not found: ID does not exist"
Jul 16 20:40:49.443654 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.443634    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8cv8\" (UniqueName: \"kubernetes.io/projected/5fe19fc5-a082-42cf-9be1-3ee7297e864a-kube-api-access-p8cv8\") pod \"5fe19fc5-a082-42cf-9be1-3ee7297e864a\" (UID: \"5fe19fc5-a082-42cf-9be1-3ee7297e864a\") "
Jul 16 20:40:49.445735 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.445715    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe19fc5-a082-42cf-9be1-3ee7297e864a-kube-api-access-p8cv8" (OuterVolumeSpecName: "kube-api-access-p8cv8") pod "5fe19fc5-a082-42cf-9be1-3ee7297e864a" (UID: "5fe19fc5-a082-42cf-9be1-3ee7297e864a"). InnerVolumeSpecName "kube-api-access-p8cv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:40:49.499685 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.499653    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bxpjf"
Jul 16 20:40:49.544666 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.544633    2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8cv8\" (UniqueName: \"kubernetes.io/projected/5fe19fc5-a082-42cf-9be1-3ee7297e864a-kube-api-access-p8cv8\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 20:40:49.640639 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:49.640608    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bxpjf"]
Jul 16 20:40:49.643455 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:40:49.643420    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76444973_2719_4125_aa2d_663c2758b13e.slice/crio-b4390546ab0ce9f299a892ebde023dd3357023b359f73ebad4953602a3ce7a6d WatchSource:0}: Error finding container b4390546ab0ce9f299a892ebde023dd3357023b359f73ebad4953602a3ce7a6d: Status 404 returned error can't find the container with id b4390546ab0ce9f299a892ebde023dd3357023b359f73ebad4953602a3ce7a6d
Jul 16 20:40:49.708073 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:40:49.708045    2530 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-f4cwv\" is forbidden: User \"system:node:ip-10-0-136-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-249.ec2.internal' and this object" podUID="5fe19fc5-a082-42cf-9be1-3ee7297e864a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"
Jul 16 20:40:50.401118 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:50.401068    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bxpjf" event={"ID":"76444973-2719-4125-aa2d-663c2758b13e","Type":"ContainerStarted","Data":"5d76ebf703afb94dae50951be420b6d1f84edb4ccb89ef2422ea809fe0467eab"}
Jul 16 20:40:50.401500 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:50.401167    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bxpjf" event={"ID":"76444973-2719-4125-aa2d-663c2758b13e","Type":"ContainerStarted","Data":"b4390546ab0ce9f299a892ebde023dd3357023b359f73ebad4953602a3ce7a6d"}
Jul 16 20:40:50.401500 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:50.401227    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bxpjf"
Jul 16 20:40:50.442400 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:50.442341    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bxpjf" podStartSLOduration=1.442322639 podStartE2EDuration="1.442322639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:40:50.441971849 +0000 UTC m=+488.463559628" watchObservedRunningTime="2026-07-16 20:40:50.442322639 +0000 UTC m=+488.463910406"
Jul 16 20:40:50.444352 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:40:50.444326    2530 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-f4cwv\" is forbidden: User \"system:node:ip-10-0-136-249.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-249.ec2.internal' and this object" podUID="5fe19fc5-a082-42cf-9be1-3ee7297e864a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-f4cwv"
Jul 16 20:40:50.506623 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:40:50.506589    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fe19fc5-a082-42cf-9be1-3ee7297e864a" path="/var/lib/kubelet/pods/5fe19fc5-a082-42cf-9be1-3ee7297e864a/volumes"
Jul 16 20:41:01.407363 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:01.407332    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bxpjf"
Jul 16 20:41:21.138703 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:21.138669    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"]
Jul 16 20:41:21.144928 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:21.144908    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"
Jul 16 20:41:21.147253 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:21.147225    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 16 20:41:21.147366 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:21.147226    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-27m6w\""
Jul 16 20:41:21.151087 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:21.151066    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"]
Jul 16 20:41:21.232464 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:21.232437    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"]
Jul 16 20:41:21.260585 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:21.260554    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5987w\" (UniqueName: \"kubernetes.io/projected/68ac170d-edc0-46bc-a619-ef9f5ad88cd7-kube-api-access-5987w\") pod \"limitador-limitador-64c94fd64f-f2lfv\" (UID: \"68ac170d-edc0-46bc-a619-ef9f5ad88cd7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"
Jul 16 20:41:21.260585 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:21.260586    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/68ac170d-edc0-46bc-a619-ef9f5ad88cd7-config-file\") pod \"limitador-limitador-64c94fd64f-f2lfv\" (UID: \"68ac170d-edc0-46bc-a619-ef9f5ad88cd7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"
Jul 16 20:41:21.361244 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:21.361214    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5987w\" (UniqueName: \"kubernetes.io/projected/68ac170d-edc0-46bc-a619-ef9f5ad88cd7-kube-api-access-5987w\") pod \"limitador-limitador-64c94fd64f-f2lfv\" (UID: \"68ac170d-edc0-46bc-a619-ef9f5ad88cd7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"
Jul 16 20:41:21.361244 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:21.361247    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/68ac170d-edc0-46bc-a619-ef9f5ad88cd7-config-file\") pod \"limitador-limitador-64c94fd64f-f2lfv\" (UID: \"68ac170d-edc0-46bc-a619-ef9f5ad88cd7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"
Jul 16 20:41:21.361811 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:21.361793    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/68ac170d-edc0-46bc-a619-ef9f5ad88cd7-config-file\") pod \"limitador-limitador-64c94fd64f-f2lfv\" (UID: \"68ac170d-edc0-46bc-a619-ef9f5ad88cd7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"
Jul 16 20:41:21.375419 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:21.375391    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5987w\" (UniqueName: \"kubernetes.io/projected/68ac170d-edc0-46bc-a619-ef9f5ad88cd7-kube-api-access-5987w\") pod \"limitador-limitador-64c94fd64f-f2lfv\" (UID: \"68ac170d-edc0-46bc-a619-ef9f5ad88cd7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"
Jul 16 20:41:21.456721 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:21.456693    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"
Jul 16 20:41:21.585084 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:21.585057    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"]
Jul 16 20:41:21.588138 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:41:21.588085    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ac170d_edc0_46bc_a619_ef9f5ad88cd7.slice/crio-2d1a07689c49a753acb1109a6d5d16bf167b06435547c5bfa3eb4e15cf460756 WatchSource:0}: Error finding container 2d1a07689c49a753acb1109a6d5d16bf167b06435547c5bfa3eb4e15cf460756: Status 404 returned error can't find the container with id 2d1a07689c49a753acb1109a6d5d16bf167b06435547c5bfa3eb4e15cf460756
Jul 16 20:41:22.533627 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:22.533584    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f2lfv" event={"ID":"68ac170d-edc0-46bc-a619-ef9f5ad88cd7","Type":"ContainerStarted","Data":"2d1a07689c49a753acb1109a6d5d16bf167b06435547c5bfa3eb4e15cf460756"}
Jul 16 20:41:24.542321 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:24.542281    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f2lfv" event={"ID":"68ac170d-edc0-46bc-a619-ef9f5ad88cd7","Type":"ContainerStarted","Data":"864dd46e9c402462ecb2bbb3bf5db53a260b6231593257e24c66612ce2f552ad"}
Jul 16 20:41:24.542720 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:24.542410    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"
Jul 16 20:41:35.547818 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:35.547742    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"
Jul 16 20:41:35.564943 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:35.564897    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-f2lfv" podStartSLOduration=12.090365401 podStartE2EDuration="14.564882478s" totalImagesPullingTime="2.474517077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:41:24.570710645 +0000 UTC m=+522.592298411" watchObservedRunningTime="2026-07-16 20:41:35.564882478 +0000 UTC m=+533.586470241"
Jul 16 20:41:36.194379 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:36.194345    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"]
Jul 16 20:41:36.194598 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:36.194549    2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-f2lfv" podUID="68ac170d-edc0-46bc-a619-ef9f5ad88cd7" containerName="limitador" containerID="cri-o://864dd46e9c402462ecb2bbb3bf5db53a260b6231593257e24c66612ce2f552ad" gracePeriod=30
Jul 16 20:41:36.594439 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:36.594403    2530 generic.go:356] "Generic (PLEG): container finished" podID="68ac170d-edc0-46bc-a619-ef9f5ad88cd7" containerID="864dd46e9c402462ecb2bbb3bf5db53a260b6231593257e24c66612ce2f552ad" exitCode=0
Jul 16 20:41:36.594789 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:36.594458    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f2lfv" event={"ID":"68ac170d-edc0-46bc-a619-ef9f5ad88cd7","Type":"ContainerDied","Data":"864dd46e9c402462ecb2bbb3bf5db53a260b6231593257e24c66612ce2f552ad"}
Jul 16 20:41:37.136545 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.136523    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"
Jul 16 20:41:37.184037 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.184011    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5987w\" (UniqueName: \"kubernetes.io/projected/68ac170d-edc0-46bc-a619-ef9f5ad88cd7-kube-api-access-5987w\") pod \"68ac170d-edc0-46bc-a619-ef9f5ad88cd7\" (UID: \"68ac170d-edc0-46bc-a619-ef9f5ad88cd7\") "
Jul 16 20:41:37.184186 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.184050    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/68ac170d-edc0-46bc-a619-ef9f5ad88cd7-config-file\") pod \"68ac170d-edc0-46bc-a619-ef9f5ad88cd7\" (UID: \"68ac170d-edc0-46bc-a619-ef9f5ad88cd7\") "
Jul 16 20:41:37.184417 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.184394    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ac170d-edc0-46bc-a619-ef9f5ad88cd7-config-file" (OuterVolumeSpecName: "config-file") pod "68ac170d-edc0-46bc-a619-ef9f5ad88cd7" (UID: "68ac170d-edc0-46bc-a619-ef9f5ad88cd7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 20:41:37.186240 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.186221    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ac170d-edc0-46bc-a619-ef9f5ad88cd7-kube-api-access-5987w" (OuterVolumeSpecName: "kube-api-access-5987w") pod "68ac170d-edc0-46bc-a619-ef9f5ad88cd7" (UID: "68ac170d-edc0-46bc-a619-ef9f5ad88cd7"). InnerVolumeSpecName "kube-api-access-5987w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 20:41:37.285550 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.285482    2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5987w\" (UniqueName: \"kubernetes.io/projected/68ac170d-edc0-46bc-a619-ef9f5ad88cd7-kube-api-access-5987w\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 20:41:37.285550 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.285510    2530 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/68ac170d-edc0-46bc-a619-ef9f5ad88cd7-config-file\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 20:41:37.599191 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.599093    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"
Jul 16 20:41:37.599191 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.599136    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f2lfv" event={"ID":"68ac170d-edc0-46bc-a619-ef9f5ad88cd7","Type":"ContainerDied","Data":"2d1a07689c49a753acb1109a6d5d16bf167b06435547c5bfa3eb4e15cf460756"}
Jul 16 20:41:37.599666 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.599203    2530 scope.go:117] "RemoveContainer" containerID="864dd46e9c402462ecb2bbb3bf5db53a260b6231593257e24c66612ce2f552ad"
Jul 16 20:41:37.622226 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.622197    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"]
Jul 16 20:41:37.625954 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.625931    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f2lfv"]
Jul 16 20:41:37.765953 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.765924    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hvclx"]
Jul 16 20:41:37.766237 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.766224    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68ac170d-edc0-46bc-a619-ef9f5ad88cd7" containerName="limitador"
Jul 16 20:41:37.766280 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.766238    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="68ac170d-edc0-46bc-a619-ef9f5ad88cd7" containerName="limitador"
Jul 16 20:41:37.766317 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.766291    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="68ac170d-edc0-46bc-a619-ef9f5ad88cd7" containerName="limitador"
Jul 16 20:41:37.770312 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.770293    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvclx"
Jul 16 20:41:37.772686 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.772663    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 16 20:41:37.772802 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.772665    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 20:41:37.772802 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.772761    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 16 20:41:37.772802 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.772792    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-27mq6\""
Jul 16 20:41:37.778653 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.778633    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hvclx"]
Jul 16 20:41:37.890071 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.889995    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f2f1cf1d-ba83-4ede-8fac-f1734033dee6-data\") pod \"postgres-69fb4594cf-hvclx\" (UID: \"f2f1cf1d-ba83-4ede-8fac-f1734033dee6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvclx"
Jul 16 20:41:37.890071 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.890027    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-294cp\" (UniqueName: \"kubernetes.io/projected/f2f1cf1d-ba83-4ede-8fac-f1734033dee6-kube-api-access-294cp\") pod \"postgres-69fb4594cf-hvclx\" (UID: \"f2f1cf1d-ba83-4ede-8fac-f1734033dee6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvclx"
Jul 16 20:41:37.991254 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.991226    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-294cp\" (UniqueName: \"kubernetes.io/projected/f2f1cf1d-ba83-4ede-8fac-f1734033dee6-kube-api-access-294cp\") pod \"postgres-69fb4594cf-hvclx\" (UID: \"f2f1cf1d-ba83-4ede-8fac-f1734033dee6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvclx"
Jul 16 20:41:37.991381 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.991305    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f2f1cf1d-ba83-4ede-8fac-f1734033dee6-data\") pod \"postgres-69fb4594cf-hvclx\" (UID: \"f2f1cf1d-ba83-4ede-8fac-f1734033dee6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvclx"
Jul 16 20:41:37.991592 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.991577    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f2f1cf1d-ba83-4ede-8fac-f1734033dee6-data\") pod \"postgres-69fb4594cf-hvclx\" (UID: \"f2f1cf1d-ba83-4ede-8fac-f1734033dee6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvclx"
Jul 16 20:41:37.999623 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:37.999593    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-294cp\" (UniqueName: \"kubernetes.io/projected/f2f1cf1d-ba83-4ede-8fac-f1734033dee6-kube-api-access-294cp\") pod \"postgres-69fb4594cf-hvclx\" (UID: \"f2f1cf1d-ba83-4ede-8fac-f1734033dee6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvclx"
Jul 16 20:41:38.082179 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:38.082144    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvclx"
Jul 16 20:41:38.212620 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:38.212589    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hvclx"]
Jul 16 20:41:38.215522 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:41:38.215490    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f1cf1d_ba83_4ede_8fac_f1734033dee6.slice/crio-ad94f7f5d81de8ea5030d265e5c576a036dcd1313a918cf556415e5be0ed3a23 WatchSource:0}: Error finding container ad94f7f5d81de8ea5030d265e5c576a036dcd1313a918cf556415e5be0ed3a23: Status 404 returned error can't find the container with id ad94f7f5d81de8ea5030d265e5c576a036dcd1313a918cf556415e5be0ed3a23
Jul 16 20:41:38.511048 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:38.511006    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68ac170d-edc0-46bc-a619-ef9f5ad88cd7" path="/var/lib/kubelet/pods/68ac170d-edc0-46bc-a619-ef9f5ad88cd7/volumes"
Jul 16 20:41:38.604251 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:38.604216    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvclx" event={"ID":"f2f1cf1d-ba83-4ede-8fac-f1734033dee6","Type":"ContainerStarted","Data":"ad94f7f5d81de8ea5030d265e5c576a036dcd1313a918cf556415e5be0ed3a23"}
Jul 16 20:41:42.905670 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:42.905649    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 16 20:41:43.630973 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:43.630931    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvclx" event={"ID":"f2f1cf1d-ba83-4ede-8fac-f1734033dee6","Type":"ContainerStarted","Data":"e20f3d039aa4ca104e7b1b56fcd988a3e8259fc11bbcefb8ce215138c32431b7"}
Jul 16 20:41:43.631131 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:43.631067    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvclx"
Jul 16 20:41:43.645299 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:43.645256    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvclx" podStartSLOduration=1.9588144330000001 podStartE2EDuration="6.645241842s" totalImagesPullingTime="4.686427409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:41:43.644269301 +0000 UTC m=+541.665857068" watchObservedRunningTime="2026-07-16 20:41:43.645241842 +0000 UTC m=+541.666829609"
Jul 16 20:41:49.663760 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:41:49.663733    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hvclx"
Jul 16 20:42:34.306312 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.306276    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt"]
Jul 16 20:42:34.313348 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.313321    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt"
Jul 16 20:42:34.316345 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.316320    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 16 20:42:34.316508 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.316328    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-js5vf\""
Jul 16 20:42:34.317603 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.317580    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt"]
Jul 16 20:42:34.429165 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.429095    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a71d5c09-6f12-49df-8e5c-dfc6e75c9a85-maas-api-tls\") pod \"maas-api-6d7676bd74-hc9zt\" (UID: \"a71d5c09-6f12-49df-8e5c-dfc6e75c9a85\") " pod="odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt"
Jul 16 20:42:34.429165 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.429166    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a71d5c09-6f12-49df-8e5c-dfc6e75c9a85-openshift-service-ca\") pod \"maas-api-6d7676bd74-hc9zt\" (UID: \"a71d5c09-6f12-49df-8e5c-dfc6e75c9a85\") " pod="odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt"
Jul 16 20:42:34.429400 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.429236    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bng9k\" (UniqueName: \"kubernetes.io/projected/a71d5c09-6f12-49df-8e5c-dfc6e75c9a85-kube-api-access-bng9k\") pod \"maas-api-6d7676bd74-hc9zt\" (UID: \"a71d5c09-6f12-49df-8e5c-dfc6e75c9a85\") " pod="odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt"
Jul 16 20:42:34.530300 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.530267    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bng9k\" (UniqueName: \"kubernetes.io/projected/a71d5c09-6f12-49df-8e5c-dfc6e75c9a85-kube-api-access-bng9k\") pod \"maas-api-6d7676bd74-hc9zt\" (UID: \"a71d5c09-6f12-49df-8e5c-dfc6e75c9a85\") " pod="odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt"
Jul 16 20:42:34.530491 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.530310    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a71d5c09-6f12-49df-8e5c-dfc6e75c9a85-maas-api-tls\") pod \"maas-api-6d7676bd74-hc9zt\" (UID: \"a71d5c09-6f12-49df-8e5c-dfc6e75c9a85\") " pod="odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt"
Jul 16 20:42:34.530491 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.530332    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a71d5c09-6f12-49df-8e5c-dfc6e75c9a85-openshift-service-ca\") pod \"maas-api-6d7676bd74-hc9zt\" (UID: \"a71d5c09-6f12-49df-8e5c-dfc6e75c9a85\") " pod="odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt"
Jul 16 20:42:34.531022 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.531003    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a71d5c09-6f12-49df-8e5c-dfc6e75c9a85-openshift-service-ca\") pod \"maas-api-6d7676bd74-hc9zt\" (UID: \"a71d5c09-6f12-49df-8e5c-dfc6e75c9a85\") " pod="odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt"
Jul 16 20:42:34.533012 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.532985    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a71d5c09-6f12-49df-8e5c-dfc6e75c9a85-maas-api-tls\") pod \"maas-api-6d7676bd74-hc9zt\" (UID: \"a71d5c09-6f12-49df-8e5c-dfc6e75c9a85\") " pod="odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt"
Jul 16 20:42:34.540914 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.540894    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bng9k\" (UniqueName: \"kubernetes.io/projected/a71d5c09-6f12-49df-8e5c-dfc6e75c9a85-kube-api-access-bng9k\") pod \"maas-api-6d7676bd74-hc9zt\" (UID: \"a71d5c09-6f12-49df-8e5c-dfc6e75c9a85\") " pod="odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt"
Jul 16 20:42:34.626029 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.625950    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt"
Jul 16 20:42:34.769899 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.769875    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt"]
Jul 16 20:42:34.772590 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:42:34.772567    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71d5c09_6f12_49df_8e5c_dfc6e75c9a85.slice/crio-a70f6a35eb2e1f3a9cf0633902e16d25fecb96ab23f216e1e4992188cfdbf5c2 WatchSource:0}: Error finding container a70f6a35eb2e1f3a9cf0633902e16d25fecb96ab23f216e1e4992188cfdbf5c2: Status 404 returned error can't find the container with id a70f6a35eb2e1f3a9cf0633902e16d25fecb96ab23f216e1e4992188cfdbf5c2
Jul 16 20:42:34.832566 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.832531    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt" event={"ID":"a71d5c09-6f12-49df-8e5c-dfc6e75c9a85","Type":"ContainerStarted","Data":"a70f6a35eb2e1f3a9cf0633902e16d25fecb96ab23f216e1e4992188cfdbf5c2"}
Jul 16 20:42:34.862044 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.862014    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tc2bv"]
Jul 16 20:42:34.866313 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.866295    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-tc2bv"
Jul 16 20:42:34.869296 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.869273    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 16 20:42:34.869416 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.869273    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-7pxps\""
Jul 16 20:42:34.875966 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.875942    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tc2bv"]
Jul 16 20:42:34.934385 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.934296    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b7ccc1-812d-45b7-a525-a6f017dd936c-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tc2bv\" (UID: \"e3b7ccc1-812d-45b7-a525-a6f017dd936c\") " pod="openshift-ingress/payload-processing-5bf47cd988-tc2bv"
Jul 16 20:42:34.934385 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:34.934360    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-966f9\" (UniqueName: \"kubernetes.io/projected/e3b7ccc1-812d-45b7-a525-a6f017dd936c-kube-api-access-966f9\") pod \"payload-processing-5bf47cd988-tc2bv\" (UID: \"e3b7ccc1-812d-45b7-a525-a6f017dd936c\") " pod="openshift-ingress/payload-processing-5bf47cd988-tc2bv"
Jul 16 20:42:35.035597 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:35.035563    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b7ccc1-812d-45b7-a525-a6f017dd936c-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tc2bv\" (UID: \"e3b7ccc1-812d-45b7-a525-a6f017dd936c\") " pod="openshift-ingress/payload-processing-5bf47cd988-tc2bv"
Jul 16 20:42:35.035772 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:35.035611    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-966f9\" (UniqueName: \"kubernetes.io/projected/e3b7ccc1-812d-45b7-a525-a6f017dd936c-kube-api-access-966f9\") pod \"payload-processing-5bf47cd988-tc2bv\" (UID: \"e3b7ccc1-812d-45b7-a525-a6f017dd936c\") " pod="openshift-ingress/payload-processing-5bf47cd988-tc2bv"
Jul 16 20:42:35.036469 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:35.036448    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b7ccc1-812d-45b7-a525-a6f017dd936c-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tc2bv\" (UID: \"e3b7ccc1-812d-45b7-a525-a6f017dd936c\") " pod="openshift-ingress/payload-processing-5bf47cd988-tc2bv"
Jul 16 20:42:35.044119 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:35.044068    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-966f9\" (UniqueName: \"kubernetes.io/projected/e3b7ccc1-812d-45b7-a525-a6f017dd936c-kube-api-access-966f9\") pod \"payload-processing-5bf47cd988-tc2bv\" (UID: \"e3b7ccc1-812d-45b7-a525-a6f017dd936c\") " pod="openshift-ingress/payload-processing-5bf47cd988-tc2bv"
Jul 16 20:42:35.178387 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:35.178354    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-tc2bv"
Jul 16 20:42:35.311859 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:35.311836    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tc2bv"]
Jul 16 20:42:35.314432 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:42:35.314402    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b7ccc1_812d_45b7_a525_a6f017dd936c.slice/crio-e8bcec63286a6fd4f19d2c0fd362706298d3cb8f9dfa8a7896e1b1feae3547ed WatchSource:0}: Error finding container e8bcec63286a6fd4f19d2c0fd362706298d3cb8f9dfa8a7896e1b1feae3547ed: Status 404 returned error can't find the container with id e8bcec63286a6fd4f19d2c0fd362706298d3cb8f9dfa8a7896e1b1feae3547ed
Jul 16 20:42:35.842492 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:35.842454    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-tc2bv" event={"ID":"e3b7ccc1-812d-45b7-a525-a6f017dd936c","Type":"ContainerStarted","Data":"e8bcec63286a6fd4f19d2c0fd362706298d3cb8f9dfa8a7896e1b1feae3547ed"}
Jul 16 20:42:38.855988 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:38.855950    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt" event={"ID":"a71d5c09-6f12-49df-8e5c-dfc6e75c9a85","Type":"ContainerStarted","Data":"3b43ca99bffd5c50052ebb048eb8fd6180130120d7adf33b86ec266930138eac"}
Jul 16 20:42:38.856431 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:38.856124    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt"
Jul 16 20:42:38.857534 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:38.857511    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-tc2bv" event={"ID":"e3b7ccc1-812d-45b7-a525-a6f017dd936c","Type":"ContainerStarted","Data":"e3de780909810fb98fe160aa9ef5d046cc00d6acc76723c70efcf1d9aa2bd93c"}
Jul 16 20:42:38.857663 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:38.857646    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-tc2bv"
Jul 16 20:42:38.872471 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:38.872432    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt" podStartSLOduration=1.412629203 podStartE2EDuration="4.872418255s" totalImagesPullingTime="3.459789052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:42:38.871058266 +0000 UTC m=+596.892646054" watchObservedRunningTime="2026-07-16 20:42:38.872418255 +0000 UTC m=+596.894006024"
Jul 16 20:42:38.885567 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:38.885522    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-tc2bv" podStartSLOduration=1.9259159430000001 podStartE2EDuration="4.885505322s" totalImagesPullingTime="2.959589379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:42:38.885356645 +0000 UTC m=+596.906944427" watchObservedRunningTime="2026-07-16 20:42:38.885505322 +0000 UTC m=+596.907093089"
Jul 16 20:42:44.867707 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:44.867676    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d7676bd74-hc9zt"
Jul 16 20:42:49.864263 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:42:49.864234    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-tc2bv"
Jul 16 20:43:00.491299 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.491264    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"]
Jul 16 20:43:00.495917 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.495897    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.499213 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.499189    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 16 20:43:00.500222 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.500021    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2ztqv\""
Jul 16 20:43:00.500222 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.500066    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 16 20:43:00.500438 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.500420    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 16 20:43:00.510955 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.510934    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"]
Jul 16 20:43:00.652880 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.652849    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0719-44d3-454f-b45f-d6d3dd0726e9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8\" (UID: \"3fdd0719-44d3-454f-b45f-d6d3dd0726e9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.652880 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.652881    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0719-44d3-454f-b45f-d6d3dd0726e9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8\" (UID: \"3fdd0719-44d3-454f-b45f-d6d3dd0726e9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.653126 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.652906    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fdd0719-44d3-454f-b45f-d6d3dd0726e9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8\" (UID: \"3fdd0719-44d3-454f-b45f-d6d3dd0726e9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.653126 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.652925    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvfrw\" (UniqueName: \"kubernetes.io/projected/3fdd0719-44d3-454f-b45f-d6d3dd0726e9-kube-api-access-qvfrw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8\" (UID: \"3fdd0719-44d3-454f-b45f-d6d3dd0726e9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.653126 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.652945    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0719-44d3-454f-b45f-d6d3dd0726e9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8\" (UID: \"3fdd0719-44d3-454f-b45f-d6d3dd0726e9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.653126 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.653028    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0719-44d3-454f-b45f-d6d3dd0726e9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8\" (UID: \"3fdd0719-44d3-454f-b45f-d6d3dd0726e9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.754002 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.753913    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0719-44d3-454f-b45f-d6d3dd0726e9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8\" (UID: \"3fdd0719-44d3-454f-b45f-d6d3dd0726e9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.754002 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.753950    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0719-44d3-454f-b45f-d6d3dd0726e9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8\" (UID: \"3fdd0719-44d3-454f-b45f-d6d3dd0726e9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.754002 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.753975    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fdd0719-44d3-454f-b45f-d6d3dd0726e9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8\" (UID: \"3fdd0719-44d3-454f-b45f-d6d3dd0726e9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.754002 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.753994    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvfrw\" (UniqueName: \"kubernetes.io/projected/3fdd0719-44d3-454f-b45f-d6d3dd0726e9-kube-api-access-qvfrw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8\" (UID: \"3fdd0719-44d3-454f-b45f-d6d3dd0726e9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.754346 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.754012    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0719-44d3-454f-b45f-d6d3dd0726e9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8\" (UID: \"3fdd0719-44d3-454f-b45f-d6d3dd0726e9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.754346 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.754047    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0719-44d3-454f-b45f-d6d3dd0726e9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8\" (UID: \"3fdd0719-44d3-454f-b45f-d6d3dd0726e9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.754446 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.754398    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0719-44d3-454f-b45f-d6d3dd0726e9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8\" (UID: \"3fdd0719-44d3-454f-b45f-d6d3dd0726e9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.754446 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.754432    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0719-44d3-454f-b45f-d6d3dd0726e9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8\" (UID: \"3fdd0719-44d3-454f-b45f-d6d3dd0726e9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.754521 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.754488    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0719-44d3-454f-b45f-d6d3dd0726e9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8\" (UID: \"3fdd0719-44d3-454f-b45f-d6d3dd0726e9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.756440 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.756408    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0719-44d3-454f-b45f-d6d3dd0726e9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8\" (UID: \"3fdd0719-44d3-454f-b45f-d6d3dd0726e9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.756574 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.756559    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fdd0719-44d3-454f-b45f-d6d3dd0726e9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8\" (UID: \"3fdd0719-44d3-454f-b45f-d6d3dd0726e9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.762152 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.762123    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvfrw\" (UniqueName: \"kubernetes.io/projected/3fdd0719-44d3-454f-b45f-d6d3dd0726e9-kube-api-access-qvfrw\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8\" (UID: \"3fdd0719-44d3-454f-b45f-d6d3dd0726e9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.806464 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.806436    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:00.945931 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.945907    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"]
Jul 16 20:43:00.948820 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:43:00.948786    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fdd0719_44d3_454f_b45f_d6d3dd0726e9.slice/crio-2b3cbcb4e92f0480fc4572195cd953fc2247774fef3da0847ce63f6ed802c4f9 WatchSource:0}: Error finding container 2b3cbcb4e92f0480fc4572195cd953fc2247774fef3da0847ce63f6ed802c4f9: Status 404 returned error can't find the container with id 2b3cbcb4e92f0480fc4572195cd953fc2247774fef3da0847ce63f6ed802c4f9
Jul 16 20:43:00.957696 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:00.957666    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8" event={"ID":"3fdd0719-44d3-454f-b45f-d6d3dd0726e9","Type":"ContainerStarted","Data":"2b3cbcb4e92f0480fc4572195cd953fc2247774fef3da0847ce63f6ed802c4f9"}
Jul 16 20:43:02.966830 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:02.966790    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8" event={"ID":"3fdd0719-44d3-454f-b45f-d6d3dd0726e9","Type":"ContainerStarted","Data":"85616bfcecd461f73fbab709396f5cee128fac66f70f053c816513a4e1489671"}
Jul 16 20:43:02.988460 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:02.988410    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8" podStartSLOduration=1.967758052 podStartE2EDuration="2.988393858s" totalImagesPullingTime="1.020635806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:43:02.986696201 +0000 UTC m=+621.008283969" watchObservedRunningTime="2026-07-16 20:43:02.988393858 +0000 UTC m=+621.009981624"
Jul 16 20:43:10.806962 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:10.806921    2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:10.807577 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:10.807001    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:10.819997 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:10.819972    2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:11.015944 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:11.015916    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8"
Jul 16 20:43:49.412774 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.412737    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"]
Jul 16 20:43:49.415539 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.415522    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.419889 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.419864    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 16 20:43:49.435021 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.434993    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"]
Jul 16 20:43:49.528545 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.528506    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/756ba2bd-efa3-40a2-b89a-a29285e54a84-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz\" (UID: \"756ba2bd-efa3-40a2-b89a-a29285e54a84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.528718 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.528576    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/756ba2bd-efa3-40a2-b89a-a29285e54a84-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz\" (UID: \"756ba2bd-efa3-40a2-b89a-a29285e54a84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.528718 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.528620    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/756ba2bd-efa3-40a2-b89a-a29285e54a84-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz\" (UID: \"756ba2bd-efa3-40a2-b89a-a29285e54a84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.528817 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.528711    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jph4\" (UniqueName: \"kubernetes.io/projected/756ba2bd-efa3-40a2-b89a-a29285e54a84-kube-api-access-4jph4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz\" (UID: \"756ba2bd-efa3-40a2-b89a-a29285e54a84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.528817 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.528773    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/756ba2bd-efa3-40a2-b89a-a29285e54a84-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz\" (UID: \"756ba2bd-efa3-40a2-b89a-a29285e54a84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.528817 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.528792    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/756ba2bd-efa3-40a2-b89a-a29285e54a84-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz\" (UID: \"756ba2bd-efa3-40a2-b89a-a29285e54a84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.629438 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.629398    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/756ba2bd-efa3-40a2-b89a-a29285e54a84-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz\" (UID: \"756ba2bd-efa3-40a2-b89a-a29285e54a84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.629601 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.629445    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/756ba2bd-efa3-40a2-b89a-a29285e54a84-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz\" (UID: \"756ba2bd-efa3-40a2-b89a-a29285e54a84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.629601 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.629493    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jph4\" (UniqueName: \"kubernetes.io/projected/756ba2bd-efa3-40a2-b89a-a29285e54a84-kube-api-access-4jph4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz\" (UID: \"756ba2bd-efa3-40a2-b89a-a29285e54a84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.629601 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.629542    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/756ba2bd-efa3-40a2-b89a-a29285e54a84-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz\" (UID: \"756ba2bd-efa3-40a2-b89a-a29285e54a84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.629601 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.629572    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/756ba2bd-efa3-40a2-b89a-a29285e54a84-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz\" (UID: \"756ba2bd-efa3-40a2-b89a-a29285e54a84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.629761 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.629614    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/756ba2bd-efa3-40a2-b89a-a29285e54a84-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz\" (UID: \"756ba2bd-efa3-40a2-b89a-a29285e54a84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.629913 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.629889    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/756ba2bd-efa3-40a2-b89a-a29285e54a84-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz\" (UID: \"756ba2bd-efa3-40a2-b89a-a29285e54a84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.629981 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.629932    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/756ba2bd-efa3-40a2-b89a-a29285e54a84-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz\" (UID: \"756ba2bd-efa3-40a2-b89a-a29285e54a84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.630031 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.629978    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/756ba2bd-efa3-40a2-b89a-a29285e54a84-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz\" (UID: \"756ba2bd-efa3-40a2-b89a-a29285e54a84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.631874 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.631850    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/756ba2bd-efa3-40a2-b89a-a29285e54a84-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz\" (UID: \"756ba2bd-efa3-40a2-b89a-a29285e54a84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.632170 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.632153    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/756ba2bd-efa3-40a2-b89a-a29285e54a84-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz\" (UID: \"756ba2bd-efa3-40a2-b89a-a29285e54a84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.641833 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.641803    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jph4\" (UniqueName: \"kubernetes.io/projected/756ba2bd-efa3-40a2-b89a-a29285e54a84-kube-api-access-4jph4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz\" (UID: \"756ba2bd-efa3-40a2-b89a-a29285e54a84\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.725483 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.725440    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:49.882130 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.882082    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"]
Jul 16 20:43:49.887066 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:43:49.887040    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756ba2bd_efa3_40a2_b89a_a29285e54a84.slice/crio-47e65f27dd70ef1e813971350bf84cf79b28018abac94f4432695ce32d0715b2 WatchSource:0}: Error finding container 47e65f27dd70ef1e813971350bf84cf79b28018abac94f4432695ce32d0715b2: Status 404 returned error can't find the container with id 47e65f27dd70ef1e813971350bf84cf79b28018abac94f4432695ce32d0715b2
Jul 16 20:43:49.888347 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:49.888330    2530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 20:43:50.163398 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:50.163352    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz" event={"ID":"756ba2bd-efa3-40a2-b89a-a29285e54a84","Type":"ContainerStarted","Data":"693d98d0cb42922ba9084e4a801ee44c7a2968fecff672e3f12a531673128066"}
Jul 16 20:43:50.163398 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:50.163404    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz" event={"ID":"756ba2bd-efa3-40a2-b89a-a29285e54a84","Type":"ContainerStarted","Data":"47e65f27dd70ef1e813971350bf84cf79b28018abac94f4432695ce32d0715b2"}
Jul 16 20:43:50.185681 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:50.185628    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz" podStartSLOduration=1.013168926 podStartE2EDuration="1.185600917s" totalImagesPullingTime="172.431991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 20:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 20:43:50.184078454 +0000 UTC m=+668.205666231" watchObservedRunningTime="2026-07-16 20:43:50.185600917 +0000 UTC m=+668.207188683"
Jul 16 20:43:59.726512 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:59.726468    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:59.726996 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:59.726544    2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:43:59.738987 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:43:59.738962    2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:44:00.215997 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:44:00.215970    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz"
Jul 16 20:59:18.197095 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:18.197059    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"]
Jul 16 20:59:18.200374 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:18.200351    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 20:59:18.202985 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:18.202951    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-23a5a19c-serving-cert\""
Jul 16 20:59:18.211563 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:18.211532    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"]
Jul 16 20:59:18.294265 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:18.294228    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lmvj\" (UniqueName: \"kubernetes.io/projected/6bf69704-cc7f-4737-8375-0618f45d11ec-kube-api-access-2lmvj\") pod \"maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg\" (UID: \"6bf69704-cc7f-4737-8375-0618f45d11ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 20:59:18.294265 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:18.294270    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bf69704-cc7f-4737-8375-0618f45d11ec-maas-api-tls\") pod \"maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg\" (UID: \"6bf69704-cc7f-4737-8375-0618f45d11ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 20:59:18.294478 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:18.294345    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf69704-cc7f-4737-8375-0618f45d11ec-openshift-service-ca\") pod \"maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg\" (UID: \"6bf69704-cc7f-4737-8375-0618f45d11ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 20:59:18.394894 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:18.394856    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bf69704-cc7f-4737-8375-0618f45d11ec-maas-api-tls\") pod \"maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg\" (UID: \"6bf69704-cc7f-4737-8375-0618f45d11ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 20:59:18.395131 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:18.394930    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf69704-cc7f-4737-8375-0618f45d11ec-openshift-service-ca\") pod \"maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg\" (UID: \"6bf69704-cc7f-4737-8375-0618f45d11ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 20:59:18.395131 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:18.394974    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lmvj\" (UniqueName: \"kubernetes.io/projected/6bf69704-cc7f-4737-8375-0618f45d11ec-kube-api-access-2lmvj\") pod \"maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg\" (UID: \"6bf69704-cc7f-4737-8375-0618f45d11ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 20:59:18.395688 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:18.395663    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf69704-cc7f-4737-8375-0618f45d11ec-openshift-service-ca\") pod \"maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg\" (UID: \"6bf69704-cc7f-4737-8375-0618f45d11ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 20:59:18.397470 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:18.397452    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bf69704-cc7f-4737-8375-0618f45d11ec-maas-api-tls\") pod \"maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg\" (UID: \"6bf69704-cc7f-4737-8375-0618f45d11ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 20:59:18.406092 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:18.406065    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lmvj\" (UniqueName: \"kubernetes.io/projected/6bf69704-cc7f-4737-8375-0618f45d11ec-kube-api-access-2lmvj\") pod \"maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg\" (UID: \"6bf69704-cc7f-4737-8375-0618f45d11ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 20:59:18.514868 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:18.514837    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 20:59:18.659495 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:18.659471    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"]
Jul 16 20:59:18.662404 ip-10-0-136-249 kubenswrapper[2530]: W0716 20:59:18.662374    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf69704_cc7f_4737_8375_0618f45d11ec.slice/crio-105276b9925dbf7f0b723468be21c093785d5bec58661f5e36dae48c90862dfd WatchSource:0}: Error finding container 105276b9925dbf7f0b723468be21c093785d5bec58661f5e36dae48c90862dfd: Status 404 returned error can't find the container with id 105276b9925dbf7f0b723468be21c093785d5bec58661f5e36dae48c90862dfd
Jul 16 20:59:18.663709 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:18.663693    2530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 20:59:19.032936 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:19.032898    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" event={"ID":"6bf69704-cc7f-4737-8375-0618f45d11ec","Type":"ContainerStarted","Data":"105276b9925dbf7f0b723468be21c093785d5bec58661f5e36dae48c90862dfd"}
Jul 16 20:59:20.038271 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:20.038233    2530 generic.go:356] "Generic (PLEG): container finished" podID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerID="d972d269989c2148f2d37e4641639f143e595e68f3c615e96832fcf6e6f53218" exitCode=1
Jul 16 20:59:20.038746 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:20.038326    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" event={"ID":"6bf69704-cc7f-4737-8375-0618f45d11ec","Type":"ContainerDied","Data":"d972d269989c2148f2d37e4641639f143e595e68f3c615e96832fcf6e6f53218"}
Jul 16 20:59:20.038746 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:20.038558    2530 scope.go:117] "RemoveContainer" containerID="d972d269989c2148f2d37e4641639f143e595e68f3c615e96832fcf6e6f53218"
Jul 16 20:59:21.044386 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:21.044353    2530 generic.go:356] "Generic (PLEG): container finished" podID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerID="f82c76d6c6dcaba2086cd2b1f698cfe1c30d31272831887593a118d226ea2659" exitCode=1
Jul 16 20:59:21.044849 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:21.044434    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" event={"ID":"6bf69704-cc7f-4737-8375-0618f45d11ec","Type":"ContainerDied","Data":"f82c76d6c6dcaba2086cd2b1f698cfe1c30d31272831887593a118d226ea2659"}
Jul 16 20:59:21.044849 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:21.044480    2530 scope.go:117] "RemoveContainer" containerID="d972d269989c2148f2d37e4641639f143e595e68f3c615e96832fcf6e6f53218"
Jul 16 20:59:21.044849 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:21.044701    2530 scope.go:117] "RemoveContainer" containerID="f82c76d6c6dcaba2086cd2b1f698cfe1c30d31272831887593a118d226ea2659"
Jul 16 20:59:21.045001 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:59:21.044938    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 20:59:22.050332 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:22.050301    2530 scope.go:117] "RemoveContainer" containerID="f82c76d6c6dcaba2086cd2b1f698cfe1c30d31272831887593a118d226ea2659"
Jul 16 20:59:22.050778 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:59:22.050528    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 20:59:23.515851 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:23.515815    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 20:59:23.516352 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:23.516341    2530 scope.go:117] "RemoveContainer" containerID="f82c76d6c6dcaba2086cd2b1f698cfe1c30d31272831887593a118d226ea2659"
Jul 16 20:59:23.516553 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:59:23.516530    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 20:59:28.514985 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:28.514953    2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 20:59:28.515392 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:28.515322    2530 scope.go:117] "RemoveContainer" containerID="f82c76d6c6dcaba2086cd2b1f698cfe1c30d31272831887593a118d226ea2659"
Jul 16 20:59:28.515545 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:59:28.515496    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 20:59:43.502745 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:43.502706    2530 scope.go:117] "RemoveContainer" containerID="f82c76d6c6dcaba2086cd2b1f698cfe1c30d31272831887593a118d226ea2659"
Jul 16 20:59:45.149125 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:45.149069    2530 generic.go:356] "Generic (PLEG): container finished" podID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerID="23b3f421982364b9ba58d5cb6d11927610b1948ae904f9d2d29a1fa8cf1c39ec" exitCode=1
Jul 16 20:59:45.149613 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:45.149152    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" event={"ID":"6bf69704-cc7f-4737-8375-0618f45d11ec","Type":"ContainerDied","Data":"23b3f421982364b9ba58d5cb6d11927610b1948ae904f9d2d29a1fa8cf1c39ec"}
Jul 16 20:59:45.149613 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:45.149226    2530 scope.go:117] "RemoveContainer" containerID="f82c76d6c6dcaba2086cd2b1f698cfe1c30d31272831887593a118d226ea2659"
Jul 16 20:59:45.149613 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:45.149587    2530 scope.go:117] "RemoveContainer" containerID="23b3f421982364b9ba58d5cb6d11927610b1948ae904f9d2d29a1fa8cf1c39ec"
Jul 16 20:59:45.149821 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:59:45.149798    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 20:59:48.515430 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:48.515403    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 20:59:48.515892 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:48.515453    2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 20:59:48.515892 ip-10-0-136-249 kubenswrapper[2530]: I0716 20:59:48.515768    2530 scope.go:117] "RemoveContainer" containerID="23b3f421982364b9ba58d5cb6d11927610b1948ae904f9d2d29a1fa8cf1c39ec"
Jul 16 20:59:48.516033 ip-10-0-136-249 kubenswrapper[2530]: E0716 20:59:48.515934    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 21:00:00.174015 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:00.173981    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw"]
Jul 16 21:00:00.178934 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:00.178912    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw"
Jul 16 21:00:00.181799 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:00.181776    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:00:00.187743 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:00.187726    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw"
Jul 16 21:00:00.193160 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:00.193138    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw"]
Jul 16 21:00:00.337543 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:00.337516    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw"]
Jul 16 21:00:00.339172 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:00:00.339141    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d232199_b225_446b_922c_ed28e011d3ca.slice/crio-99ac848f3d87e96f96896f624879558b7d1e611603b501de51e3d84f60d7d875 WatchSource:0}: Error finding container 99ac848f3d87e96f96896f624879558b7d1e611603b501de51e3d84f60d7d875: Status 404 returned error can't find the container with id 99ac848f3d87e96f96896f624879558b7d1e611603b501de51e3d84f60d7d875
Jul 16 21:00:01.221662 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:01.221617    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw" event={"ID":"5d232199-b225-446b-922c-ed28e011d3ca","Type":"ContainerStarted","Data":"99ac848f3d87e96f96896f624879558b7d1e611603b501de51e3d84f60d7d875"}
Jul 16 21:00:02.504079 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:02.504037    2530 scope.go:117] "RemoveContainer" containerID="23b3f421982364b9ba58d5cb6d11927610b1948ae904f9d2d29a1fa8cf1c39ec"
Jul 16 21:00:02.504479 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:00:02.504299    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 21:00:04.237634 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:04.237595    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw" event={"ID":"5d232199-b225-446b-922c-ed28e011d3ca","Type":"ContainerStarted","Data":"272519919b6508e39a750cc3d3fd43e4142b368daa3ae0f2a4faecc84ba615b0"}
Jul 16 21:00:05.244498 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:05.244463    2530 generic.go:356] "Generic (PLEG): container finished" podID="5d232199-b225-446b-922c-ed28e011d3ca" containerID="272519919b6508e39a750cc3d3fd43e4142b368daa3ae0f2a4faecc84ba615b0" exitCode=7
Jul 16 21:00:05.244877 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:05.244547    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw" event={"ID":"5d232199-b225-446b-922c-ed28e011d3ca","Type":"ContainerDied","Data":"272519919b6508e39a750cc3d3fd43e4142b368daa3ae0f2a4faecc84ba615b0"}
Jul 16 21:00:05.244877 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:05.244686    2530 scope.go:117] "RemoveContainer" containerID="272519919b6508e39a750cc3d3fd43e4142b368daa3ae0f2a4faecc84ba615b0"
Jul 16 21:00:06.251109 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:06.251059    2530 generic.go:356] "Generic (PLEG): container finished" podID="5d232199-b225-446b-922c-ed28e011d3ca" containerID="88b82e9604ff430c778dd2f0caba0880bd29f95952b0e69f7eebe5e2f440af0f" exitCode=7
Jul 16 21:00:06.251509 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:06.251142    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw" event={"ID":"5d232199-b225-446b-922c-ed28e011d3ca","Type":"ContainerDied","Data":"88b82e9604ff430c778dd2f0caba0880bd29f95952b0e69f7eebe5e2f440af0f"}
Jul 16 21:00:06.251509 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:06.251192    2530 scope.go:117] "RemoveContainer" containerID="272519919b6508e39a750cc3d3fd43e4142b368daa3ae0f2a4faecc84ba615b0"
Jul 16 21:00:06.251509 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:06.251330    2530 scope.go:117] "RemoveContainer" containerID="88b82e9604ff430c778dd2f0caba0880bd29f95952b0e69f7eebe5e2f440af0f"
Jul 16 21:00:06.251631 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:00:06.251572    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw_odh-ai-gateway-infra(5d232199-b225-446b-922c-ed28e011d3ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw" podUID="5d232199-b225-446b-922c-ed28e011d3ca"
Jul 16 21:00:07.257111 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:07.257074    2530 scope.go:117] "RemoveContainer" containerID="88b82e9604ff430c778dd2f0caba0880bd29f95952b0e69f7eebe5e2f440af0f"
Jul 16 21:00:07.257531 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:00:07.257303    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw_odh-ai-gateway-infra(5d232199-b225-446b-922c-ed28e011d3ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw" podUID="5d232199-b225-446b-922c-ed28e011d3ca"
Jul 16 21:00:15.502681 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:15.502647    2530 scope.go:117] "RemoveContainer" containerID="23b3f421982364b9ba58d5cb6d11927610b1948ae904f9d2d29a1fa8cf1c39ec"
Jul 16 21:00:16.295375 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:16.295343    2530 generic.go:356] "Generic (PLEG): container finished" podID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerID="ae2d55abe11901a0dd6f95b217e265b76ac3559d864d327ee8c09e83ce3cf489" exitCode=1
Jul 16 21:00:16.295517 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:16.295412    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" event={"ID":"6bf69704-cc7f-4737-8375-0618f45d11ec","Type":"ContainerDied","Data":"ae2d55abe11901a0dd6f95b217e265b76ac3559d864d327ee8c09e83ce3cf489"}
Jul 16 21:00:16.295517 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:16.295476    2530 scope.go:117] "RemoveContainer" containerID="23b3f421982364b9ba58d5cb6d11927610b1948ae904f9d2d29a1fa8cf1c39ec"
Jul 16 21:00:16.295887 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:16.295869    2530 scope.go:117] "RemoveContainer" containerID="ae2d55abe11901a0dd6f95b217e265b76ac3559d864d327ee8c09e83ce3cf489"
Jul 16 21:00:16.296074 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:00:16.296057    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 21:00:18.515078 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:18.515050    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 21:00:18.515587 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:18.515133    2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 21:00:18.515587 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:18.515546    2530 scope.go:117] "RemoveContainer" containerID="ae2d55abe11901a0dd6f95b217e265b76ac3559d864d327ee8c09e83ce3cf489"
Jul 16 21:00:18.515778 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:00:18.515754    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 21:00:22.504248 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:22.504221    2530 scope.go:117] "RemoveContainer" containerID="88b82e9604ff430c778dd2f0caba0880bd29f95952b0e69f7eebe5e2f440af0f"
Jul 16 21:00:23.116751 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:23.116725    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd"]
Jul 16 21:00:23.119984 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:23.119963    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd"
Jul 16 21:00:23.128402 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:23.128371    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd"]
Jul 16 21:00:23.160444 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:23.160403    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89c6aba-4579-422c-b2bf-8fdb811c04cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd\" (UID: \"e89c6aba-4579-422c-b2bf-8fdb811c04cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd"
Jul 16 21:00:23.261739 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:23.261698    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89c6aba-4579-422c-b2bf-8fdb811c04cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd\" (UID: \"e89c6aba-4579-422c-b2bf-8fdb811c04cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd"
Jul 16 21:00:23.262395 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:23.262374    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89c6aba-4579-422c-b2bf-8fdb811c04cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd\" (UID: \"e89c6aba-4579-422c-b2bf-8fdb811c04cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd"
Jul 16 21:00:23.329214 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:23.329180    2530 generic.go:356] "Generic (PLEG): container finished" podID="5d232199-b225-446b-922c-ed28e011d3ca" containerID="a8d21b20e78ebf262ea12e1b6f02d9113b964879361ee24939b4a314963f84ef" exitCode=7
Jul 16 21:00:23.329403 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:23.329258    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw" event={"ID":"5d232199-b225-446b-922c-ed28e011d3ca","Type":"ContainerDied","Data":"a8d21b20e78ebf262ea12e1b6f02d9113b964879361ee24939b4a314963f84ef"}
Jul 16 21:00:23.329403 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:23.329319    2530 scope.go:117] "RemoveContainer" containerID="88b82e9604ff430c778dd2f0caba0880bd29f95952b0e69f7eebe5e2f440af0f"
Jul 16 21:00:23.329616 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:23.329599    2530 scope.go:117] "RemoveContainer" containerID="a8d21b20e78ebf262ea12e1b6f02d9113b964879361ee24939b4a314963f84ef"
Jul 16 21:00:23.329857 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:00:23.329828    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw_odh-ai-gateway-infra(5d232199-b225-446b-922c-ed28e011d3ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw" podUID="5d232199-b225-446b-922c-ed28e011d3ca"
Jul 16 21:00:23.430782 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:23.430694    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd"
Jul 16 21:00:23.567409 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:23.567085    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd"]
Jul 16 21:00:23.570066 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:00:23.570029    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89c6aba_4579_422c_b2bf_8fdb811c04cf.slice/crio-1be56f4b4c425b2545862fa1ef2fb77454015f517c4a7c85f3593300d3e6ff48 WatchSource:0}: Error finding container 1be56f4b4c425b2545862fa1ef2fb77454015f517c4a7c85f3593300d3e6ff48: Status 404 returned error can't find the container with id 1be56f4b4c425b2545862fa1ef2fb77454015f517c4a7c85f3593300d3e6ff48
Jul 16 21:00:24.336513 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:24.336482    2530 generic.go:356] "Generic (PLEG): container finished" podID="e89c6aba-4579-422c-b2bf-8fdb811c04cf" containerID="b55c2f89f75fe0c6d418b40c01c16ea5a3ebe5b5a76cabc50ab460498b28c71c" exitCode=7
Jul 16 21:00:24.336685 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:24.336572    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd" event={"ID":"e89c6aba-4579-422c-b2bf-8fdb811c04cf","Type":"ContainerDied","Data":"b55c2f89f75fe0c6d418b40c01c16ea5a3ebe5b5a76cabc50ab460498b28c71c"}
Jul 16 21:00:24.336685 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:24.336624    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd" event={"ID":"e89c6aba-4579-422c-b2bf-8fdb811c04cf","Type":"ContainerStarted","Data":"1be56f4b4c425b2545862fa1ef2fb77454015f517c4a7c85f3593300d3e6ff48"}
Jul 16 21:00:24.336760 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:24.336714    2530 scope.go:117] "RemoveContainer" containerID="b55c2f89f75fe0c6d418b40c01c16ea5a3ebe5b5a76cabc50ab460498b28c71c"
Jul 16 21:00:24.352574 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:24.352550    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw"]
Jul 16 21:00:24.489312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:24.489288    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw"
Jul 16 21:00:25.341995 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:25.341908    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw"
Jul 16 21:00:25.341995 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:25.341936    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw" event={"ID":"5d232199-b225-446b-922c-ed28e011d3ca","Type":"ContainerDied","Data":"99ac848f3d87e96f96896f624879558b7d1e611603b501de51e3d84f60d7d875"}
Jul 16 21:00:25.342519 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:25.342005    2530 scope.go:117] "RemoveContainer" containerID="a8d21b20e78ebf262ea12e1b6f02d9113b964879361ee24939b4a314963f84ef"
Jul 16 21:00:25.344241 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:25.344214    2530 generic.go:356] "Generic (PLEG): container finished" podID="e89c6aba-4579-422c-b2bf-8fdb811c04cf" containerID="bc904b262368fb10cd83c8fdf455cdf2f85e051476e0c871e6165c3c527dc02e" exitCode=7
Jul 16 21:00:25.344367 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:25.344254    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd" event={"ID":"e89c6aba-4579-422c-b2bf-8fdb811c04cf","Type":"ContainerDied","Data":"bc904b262368fb10cd83c8fdf455cdf2f85e051476e0c871e6165c3c527dc02e"}
Jul 16 21:00:25.344541 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:25.344524    2530 scope.go:117] "RemoveContainer" containerID="bc904b262368fb10cd83c8fdf455cdf2f85e051476e0c871e6165c3c527dc02e"
Jul 16 21:00:25.344743 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:00:25.344725    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd_odh-ai-gateway-infra(e89c6aba-4579-422c-b2bf-8fdb811c04cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd" podUID="e89c6aba-4579-422c-b2bf-8fdb811c04cf"
Jul 16 21:00:25.350360 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:25.350342    2530 scope.go:117] "RemoveContainer" containerID="b55c2f89f75fe0c6d418b40c01c16ea5a3ebe5b5a76cabc50ab460498b28c71c"
Jul 16 21:00:25.360443 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:25.360419    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw"]
Jul 16 21:00:25.368837 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:25.368808    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-23a5a19c-29737260-62bsw"]
Jul 16 21:00:26.350643 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:26.350617    2530 scope.go:117] "RemoveContainer" containerID="bc904b262368fb10cd83c8fdf455cdf2f85e051476e0c871e6165c3c527dc02e"
Jul 16 21:00:26.351043 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:00:26.350800    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd_odh-ai-gateway-infra(e89c6aba-4579-422c-b2bf-8fdb811c04cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd" podUID="e89c6aba-4579-422c-b2bf-8fdb811c04cf"
Jul 16 21:00:26.507449 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:26.507415    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d232199-b225-446b-922c-ed28e011d3ca" path="/var/lib/kubelet/pods/5d232199-b225-446b-922c-ed28e011d3ca/volumes"
Jul 16 21:00:33.502392 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:33.502354    2530 scope.go:117] "RemoveContainer" containerID="ae2d55abe11901a0dd6f95b217e265b76ac3559d864d327ee8c09e83ce3cf489"
Jul 16 21:00:33.502883 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:00:33.502574    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 21:00:40.502265 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:40.502233    2530 scope.go:117] "RemoveContainer" containerID="bc904b262368fb10cd83c8fdf455cdf2f85e051476e0c871e6165c3c527dc02e"
Jul 16 21:00:41.417528 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:41.417443    2530 generic.go:356] "Generic (PLEG): container finished" podID="e89c6aba-4579-422c-b2bf-8fdb811c04cf" containerID="641e39460e64e0cb88b6a08b7c048dd2dcd6dd24e468ce5d4b9fb9b439941704" exitCode=7
Jul 16 21:00:41.417669 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:41.417526    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd" event={"ID":"e89c6aba-4579-422c-b2bf-8fdb811c04cf","Type":"ContainerDied","Data":"641e39460e64e0cb88b6a08b7c048dd2dcd6dd24e468ce5d4b9fb9b439941704"}
Jul 16 21:00:41.417669 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:41.417585    2530 scope.go:117] "RemoveContainer" containerID="bc904b262368fb10cd83c8fdf455cdf2f85e051476e0c871e6165c3c527dc02e"
Jul 16 21:00:41.417918 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:41.417899    2530 scope.go:117] "RemoveContainer" containerID="641e39460e64e0cb88b6a08b7c048dd2dcd6dd24e468ce5d4b9fb9b439941704"
Jul 16 21:00:41.418163 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:00:41.418133    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd_odh-ai-gateway-infra(e89c6aba-4579-422c-b2bf-8fdb811c04cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd" podUID="e89c6aba-4579-422c-b2bf-8fdb811c04cf"
Jul 16 21:00:42.443896 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:42.443863    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd"]
Jul 16 21:00:42.571655 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:42.571630    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd"
Jul 16 21:00:42.733330 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:42.733298    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89c6aba-4579-422c-b2bf-8fdb811c04cf-maas-api-service-ca\") pod \"e89c6aba-4579-422c-b2bf-8fdb811c04cf\" (UID: \"e89c6aba-4579-422c-b2bf-8fdb811c04cf\") "
Jul 16 21:00:42.733641 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:42.733620    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89c6aba-4579-422c-b2bf-8fdb811c04cf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e89c6aba-4579-422c-b2bf-8fdb811c04cf" (UID: "e89c6aba-4579-422c-b2bf-8fdb811c04cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:00:42.834422 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:42.834394    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89c6aba-4579-422c-b2bf-8fdb811c04cf-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:00:43.428819 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:43.428793    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd"
Jul 16 21:00:43.428967 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:43.428789    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd" event={"ID":"e89c6aba-4579-422c-b2bf-8fdb811c04cf","Type":"ContainerDied","Data":"1be56f4b4c425b2545862fa1ef2fb77454015f517c4a7c85f3593300d3e6ff48"}
Jul 16 21:00:43.428967 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:43.428952    2530 scope.go:117] "RemoveContainer" containerID="641e39460e64e0cb88b6a08b7c048dd2dcd6dd24e468ce5d4b9fb9b439941704"
Jul 16 21:00:43.449815 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:43.449791    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd"]
Jul 16 21:00:43.458269 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:43.458248    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-gp8dd"]
Jul 16 21:00:44.502429 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:44.502399    2530 scope.go:117] "RemoveContainer" containerID="ae2d55abe11901a0dd6f95b217e265b76ac3559d864d327ee8c09e83ce3cf489"
Jul 16 21:00:44.502796 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:00:44.502554    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 21:00:44.506929 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:44.506908    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e89c6aba-4579-422c-b2bf-8fdb811c04cf" path="/var/lib/kubelet/pods/e89c6aba-4579-422c-b2bf-8fdb811c04cf/volumes"
Jul 16 21:00:56.502069 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:56.502039    2530 scope.go:117] "RemoveContainer" containerID="ae2d55abe11901a0dd6f95b217e265b76ac3559d864d327ee8c09e83ce3cf489"
Jul 16 21:00:59.501472 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:59.501434    2530 generic.go:356] "Generic (PLEG): container finished" podID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerID="e5a048bd06cbe49999bca2f3430c5b77ef79baebc275235c0555675fb8214df8" exitCode=1
Jul 16 21:00:59.501887 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:59.501512    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" event={"ID":"6bf69704-cc7f-4737-8375-0618f45d11ec","Type":"ContainerDied","Data":"e5a048bd06cbe49999bca2f3430c5b77ef79baebc275235c0555675fb8214df8"}
Jul 16 21:00:59.501887 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:59.501580    2530 scope.go:117] "RemoveContainer" containerID="ae2d55abe11901a0dd6f95b217e265b76ac3559d864d327ee8c09e83ce3cf489"
Jul 16 21:00:59.501986 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:00:59.501899    2530 scope.go:117] "RemoveContainer" containerID="e5a048bd06cbe49999bca2f3430c5b77ef79baebc275235c0555675fb8214df8"
Jul 16 21:00:59.502093 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:00:59.502072    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 21:01:03.515720 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:03.515644    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 21:01:03.516154 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:03.516041    2530 scope.go:117] "RemoveContainer" containerID="e5a048bd06cbe49999bca2f3430c5b77ef79baebc275235c0555675fb8214df8"
Jul 16 21:01:03.516272 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:01:03.516251    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 21:01:08.515516 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:08.515486    2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 21:01:08.515873 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:08.515769    2530 scope.go:117] "RemoveContainer" containerID="e5a048bd06cbe49999bca2f3430c5b77ef79baebc275235c0555675fb8214df8"
Jul 16 21:01:08.515935 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:01:08.515917    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 21:01:23.162919 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.162880    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz"]
Jul 16 21:01:23.163283 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.163212    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89c6aba-4579-422c-b2bf-8fdb811c04cf" containerName="revoke-keys"
Jul 16 21:01:23.163283 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.163227    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89c6aba-4579-422c-b2bf-8fdb811c04cf" containerName="revoke-keys"
Jul 16 21:01:23.163283 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.163241    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d232199-b225-446b-922c-ed28e011d3ca" containerName="cleanup"
Jul 16 21:01:23.163283 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.163249    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d232199-b225-446b-922c-ed28e011d3ca" containerName="cleanup"
Jul 16 21:01:23.163283 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.163259    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d232199-b225-446b-922c-ed28e011d3ca" containerName="cleanup"
Jul 16 21:01:23.163283 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.163267    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d232199-b225-446b-922c-ed28e011d3ca" containerName="cleanup"
Jul 16 21:01:23.163283 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.163275    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89c6aba-4579-422c-b2bf-8fdb811c04cf" containerName="revoke-keys"
Jul 16 21:01:23.163283 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.163283    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89c6aba-4579-422c-b2bf-8fdb811c04cf" containerName="revoke-keys"
Jul 16 21:01:23.163555 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.163298    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d232199-b225-446b-922c-ed28e011d3ca" containerName="cleanup"
Jul 16 21:01:23.163555 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.163305    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d232199-b225-446b-922c-ed28e011d3ca" containerName="cleanup"
Jul 16 21:01:23.163555 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.163419    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="e89c6aba-4579-422c-b2bf-8fdb811c04cf" containerName="revoke-keys"
Jul 16 21:01:23.163555 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.163429    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d232199-b225-446b-922c-ed28e011d3ca" containerName="cleanup"
Jul 16 21:01:23.163555 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.163438    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="e89c6aba-4579-422c-b2bf-8fdb811c04cf" containerName="revoke-keys"
Jul 16 21:01:23.163555 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.163447    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="e89c6aba-4579-422c-b2bf-8fdb811c04cf" containerName="revoke-keys"
Jul 16 21:01:23.163555 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.163456    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d232199-b225-446b-922c-ed28e011d3ca" containerName="cleanup"
Jul 16 21:01:23.166704 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.166684    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz"
Jul 16 21:01:23.168938 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.168915    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:01:23.172663 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.172639    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz"]
Jul 16 21:01:23.271024 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.270991    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3664a6-9b0b-4d5e-a697-d2d243b75417-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz\" (UID: \"ef3664a6-9b0b-4d5e-a697-d2d243b75417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz"
Jul 16 21:01:23.372284 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.372252    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3664a6-9b0b-4d5e-a697-d2d243b75417-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz\" (UID: \"ef3664a6-9b0b-4d5e-a697-d2d243b75417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz"
Jul 16 21:01:23.372838 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.372820    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3664a6-9b0b-4d5e-a697-d2d243b75417-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz\" (UID: \"ef3664a6-9b0b-4d5e-a697-d2d243b75417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz"
Jul 16 21:01:23.478804 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.478772    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz"
Jul 16 21:01:23.502223 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.502204    2530 scope.go:117] "RemoveContainer" containerID="e5a048bd06cbe49999bca2f3430c5b77ef79baebc275235c0555675fb8214df8"
Jul 16 21:01:23.502434 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:01:23.502411    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 21:01:23.604249 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:23.604226    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz"]
Jul 16 21:01:23.606643 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:01:23.606615    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3664a6_9b0b_4d5e_a697_d2d243b75417.slice/crio-2d4b73b973b57fd844e8db466a5ad3d85a974cb414ec14366c4439c315a559d2 WatchSource:0}: Error finding container 2d4b73b973b57fd844e8db466a5ad3d85a974cb414ec14366c4439c315a559d2: Status 404 returned error can't find the container with id 2d4b73b973b57fd844e8db466a5ad3d85a974cb414ec14366c4439c315a559d2
Jul 16 21:01:24.608081 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:24.608002    2530 generic.go:356] "Generic (PLEG): container finished" podID="ef3664a6-9b0b-4d5e-a697-d2d243b75417" containerID="a82fa4e86308628a947fa471ee7288f6f142be5a96bde611725e4ca571cb648e" exitCode=7
Jul 16 21:01:24.608433 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:24.608091    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz" event={"ID":"ef3664a6-9b0b-4d5e-a697-d2d243b75417","Type":"ContainerDied","Data":"a82fa4e86308628a947fa471ee7288f6f142be5a96bde611725e4ca571cb648e"}
Jul 16 21:01:24.608433 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:24.608163    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz" event={"ID":"ef3664a6-9b0b-4d5e-a697-d2d243b75417","Type":"ContainerStarted","Data":"2d4b73b973b57fd844e8db466a5ad3d85a974cb414ec14366c4439c315a559d2"}
Jul 16 21:01:24.608433 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:24.608245    2530 scope.go:117] "RemoveContainer" containerID="a82fa4e86308628a947fa471ee7288f6f142be5a96bde611725e4ca571cb648e"
Jul 16 21:01:25.614707 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:25.614636    2530 generic.go:356] "Generic (PLEG): container finished" podID="ef3664a6-9b0b-4d5e-a697-d2d243b75417" containerID="937becdf691deb40ada9b007f60ddb2979d6d7eec9716a5aa251099391f3e7ff" exitCode=7
Jul 16 21:01:25.615066 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:25.614722    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz" event={"ID":"ef3664a6-9b0b-4d5e-a697-d2d243b75417","Type":"ContainerDied","Data":"937becdf691deb40ada9b007f60ddb2979d6d7eec9716a5aa251099391f3e7ff"}
Jul 16 21:01:25.615066 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:25.614784    2530 scope.go:117] "RemoveContainer" containerID="a82fa4e86308628a947fa471ee7288f6f142be5a96bde611725e4ca571cb648e"
Jul 16 21:01:25.615066 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:25.614841    2530 scope.go:117] "RemoveContainer" containerID="937becdf691deb40ada9b007f60ddb2979d6d7eec9716a5aa251099391f3e7ff"
Jul 16 21:01:25.615066 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:01:25.615003    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz_odh-ai-gateway-infra(ef3664a6-9b0b-4d5e-a697-d2d243b75417)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz" podUID="ef3664a6-9b0b-4d5e-a697-d2d243b75417"
Jul 16 21:01:26.620221 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:26.620196    2530 scope.go:117] "RemoveContainer" containerID="937becdf691deb40ada9b007f60ddb2979d6d7eec9716a5aa251099391f3e7ff"
Jul 16 21:01:26.620607 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:01:26.620355    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz_odh-ai-gateway-infra(ef3664a6-9b0b-4d5e-a697-d2d243b75417)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz" podUID="ef3664a6-9b0b-4d5e-a697-d2d243b75417"
Jul 16 21:01:34.502878 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:34.502845    2530 scope.go:117] "RemoveContainer" containerID="e5a048bd06cbe49999bca2f3430c5b77ef79baebc275235c0555675fb8214df8"
Jul 16 21:01:34.503472 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:01:34.503017    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 21:01:41.502861 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:41.502831    2530 scope.go:117] "RemoveContainer" containerID="937becdf691deb40ada9b007f60ddb2979d6d7eec9716a5aa251099391f3e7ff"
Jul 16 21:01:42.694068 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:42.694033    2530 generic.go:356] "Generic (PLEG): container finished" podID="ef3664a6-9b0b-4d5e-a697-d2d243b75417" containerID="2ebfe5700c58376a80c26f812ec16c83f560800a579d23089988fa0577c80cc3" exitCode=7
Jul 16 21:01:42.694068 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:42.694072    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz" event={"ID":"ef3664a6-9b0b-4d5e-a697-d2d243b75417","Type":"ContainerDied","Data":"2ebfe5700c58376a80c26f812ec16c83f560800a579d23089988fa0577c80cc3"}
Jul 16 21:01:42.694570 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:42.694142    2530 scope.go:117] "RemoveContainer" containerID="937becdf691deb40ada9b007f60ddb2979d6d7eec9716a5aa251099391f3e7ff"
Jul 16 21:01:42.694570 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:42.694458    2530 scope.go:117] "RemoveContainer" containerID="2ebfe5700c58376a80c26f812ec16c83f560800a579d23089988fa0577c80cc3"
Jul 16 21:01:42.694674 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:01:42.694654    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz_odh-ai-gateway-infra(ef3664a6-9b0b-4d5e-a697-d2d243b75417)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz" podUID="ef3664a6-9b0b-4d5e-a697-d2d243b75417"
Jul 16 21:01:43.720997 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:43.720960    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz"]
Jul 16 21:01:43.852311 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:43.852289    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz"
Jul 16 21:01:43.946884 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:43.946853    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3664a6-9b0b-4d5e-a697-d2d243b75417-maas-api-service-ca\") pod \"ef3664a6-9b0b-4d5e-a697-d2d243b75417\" (UID: \"ef3664a6-9b0b-4d5e-a697-d2d243b75417\") "
Jul 16 21:01:43.947203 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:43.947182    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3664a6-9b0b-4d5e-a697-d2d243b75417-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef3664a6-9b0b-4d5e-a697-d2d243b75417" (UID: "ef3664a6-9b0b-4d5e-a697-d2d243b75417"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:01:44.047759 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:44.047694    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef3664a6-9b0b-4d5e-a697-d2d243b75417-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:01:44.704428 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:44.704398    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz"
Jul 16 21:01:44.704598 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:44.704437    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz" event={"ID":"ef3664a6-9b0b-4d5e-a697-d2d243b75417","Type":"ContainerDied","Data":"2d4b73b973b57fd844e8db466a5ad3d85a974cb414ec14366c4439c315a559d2"}
Jul 16 21:01:44.704598 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:44.704501    2530 scope.go:117] "RemoveContainer" containerID="2ebfe5700c58376a80c26f812ec16c83f560800a579d23089988fa0577c80cc3"
Jul 16 21:01:44.720393 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:44.720359    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz"]
Jul 16 21:01:44.721888 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:44.721869    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-xlzsz"]
Jul 16 21:01:45.502829 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:45.502795    2530 scope.go:117] "RemoveContainer" containerID="e5a048bd06cbe49999bca2f3430c5b77ef79baebc275235c0555675fb8214df8"
Jul 16 21:01:45.503012 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:01:45.502972    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 21:01:46.507236 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:46.507205    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef3664a6-9b0b-4d5e-a697-d2d243b75417" path="/var/lib/kubelet/pods/ef3664a6-9b0b-4d5e-a697-d2d243b75417/volumes"
Jul 16 21:01:56.502498 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:01:56.502460    2530 scope.go:117] "RemoveContainer" containerID="e5a048bd06cbe49999bca2f3430c5b77ef79baebc275235c0555675fb8214df8"
Jul 16 21:01:56.502964 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:01:56.502653    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 21:02:09.501795 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:09.501763    2530 scope.go:117] "RemoveContainer" containerID="e5a048bd06cbe49999bca2f3430c5b77ef79baebc275235c0555675fb8214df8"
Jul 16 21:02:09.502294 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:02:09.501922    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 21:02:22.504175 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:22.504143    2530 scope.go:117] "RemoveContainer" containerID="e5a048bd06cbe49999bca2f3430c5b77ef79baebc275235c0555675fb8214df8"
Jul 16 21:02:23.220476 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.220442    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j"]
Jul 16 21:02:23.220789 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.220776    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89c6aba-4579-422c-b2bf-8fdb811c04cf" containerName="revoke-keys"
Jul 16 21:02:23.220854 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.220792    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89c6aba-4579-422c-b2bf-8fdb811c04cf" containerName="revoke-keys"
Jul 16 21:02:23.220854 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.220805    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3664a6-9b0b-4d5e-a697-d2d243b75417" containerName="revoke-keys"
Jul 16 21:02:23.220854 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.220811    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3664a6-9b0b-4d5e-a697-d2d243b75417" containerName="revoke-keys"
Jul 16 21:02:23.220854 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.220818    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3664a6-9b0b-4d5e-a697-d2d243b75417" containerName="revoke-keys"
Jul 16 21:02:23.220854 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.220823    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3664a6-9b0b-4d5e-a697-d2d243b75417" containerName="revoke-keys"
Jul 16 21:02:23.220854 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.220836    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef3664a6-9b0b-4d5e-a697-d2d243b75417" containerName="revoke-keys"
Jul 16 21:02:23.220854 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.220843    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef3664a6-9b0b-4d5e-a697-d2d243b75417" containerName="revoke-keys"
Jul 16 21:02:23.221115 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.220922    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef3664a6-9b0b-4d5e-a697-d2d243b75417" containerName="revoke-keys"
Jul 16 21:02:23.221115 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.220933    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef3664a6-9b0b-4d5e-a697-d2d243b75417" containerName="revoke-keys"
Jul 16 21:02:23.221115 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.220947    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d232199-b225-446b-922c-ed28e011d3ca" containerName="cleanup"
Jul 16 21:02:23.223997 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.223980    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j"
Jul 16 21:02:23.226055 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.226033    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:02:23.230005 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.229983    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j"]
Jul 16 21:02:23.381701 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.381627    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f3df62-653b-4a8c-a29a-42bae9c214d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j\" (UID: \"f5f3df62-653b-4a8c-a29a-42bae9c214d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j"
Jul 16 21:02:23.482773 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.482746    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f3df62-653b-4a8c-a29a-42bae9c214d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j\" (UID: \"f5f3df62-653b-4a8c-a29a-42bae9c214d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j"
Jul 16 21:02:23.483333 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.483317    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f3df62-653b-4a8c-a29a-42bae9c214d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j\" (UID: \"f5f3df62-653b-4a8c-a29a-42bae9c214d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j"
Jul 16 21:02:23.536930 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.536907    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j"
Jul 16 21:02:23.662583 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.662559    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j"]
Jul 16 21:02:23.664519 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:02:23.664492    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f3df62_653b_4a8c_a29a_42bae9c214d2.slice/crio-c861789bd6ed0ce6ac0cf3df8b482e940cc6fd262305289cdeb3568ab8914393 WatchSource:0}: Error finding container c861789bd6ed0ce6ac0cf3df8b482e940cc6fd262305289cdeb3568ab8914393: Status 404 returned error can't find the container with id c861789bd6ed0ce6ac0cf3df8b482e940cc6fd262305289cdeb3568ab8914393
Jul 16 21:02:23.863078 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.863054    2530 generic.go:356] "Generic (PLEG): container finished" podID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerID="68e5445964e3d01d64f4bc53776fd575a3f0390dec0f45160dec91e0de601510" exitCode=1
Jul 16 21:02:23.863212 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.863082    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" event={"ID":"6bf69704-cc7f-4737-8375-0618f45d11ec","Type":"ContainerDied","Data":"68e5445964e3d01d64f4bc53776fd575a3f0390dec0f45160dec91e0de601510"}
Jul 16 21:02:23.863212 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.863169    2530 scope.go:117] "RemoveContainer" containerID="e5a048bd06cbe49999bca2f3430c5b77ef79baebc275235c0555675fb8214df8"
Jul 16 21:02:23.863460 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.863442    2530 scope.go:117] "RemoveContainer" containerID="68e5445964e3d01d64f4bc53776fd575a3f0390dec0f45160dec91e0de601510"
Jul 16 21:02:23.863651 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:02:23.863631    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 21:02:23.864726 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.864696    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j" event={"ID":"f5f3df62-653b-4a8c-a29a-42bae9c214d2","Type":"ContainerStarted","Data":"dead18a097fb6d3cfa1a499c19328db85f1d91fae9e639adfccc2ed64c57ed58"}
Jul 16 21:02:23.864842 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.864754    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j" event={"ID":"f5f3df62-653b-4a8c-a29a-42bae9c214d2","Type":"ContainerStarted","Data":"c861789bd6ed0ce6ac0cf3df8b482e940cc6fd262305289cdeb3568ab8914393"}
Jul 16 21:02:23.895067 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:23.895026    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j" podStartSLOduration=0.895014799 podStartE2EDuration="895.014799ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:02:23.892908873 +0000 UTC m=+1781.914496637" watchObservedRunningTime="2026-07-16 21:02:23.895014799 +0000 UTC m=+1781.916602564"
Jul 16 21:02:24.871730 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:24.871698    2530 generic.go:356] "Generic (PLEG): container finished" podID="f5f3df62-653b-4a8c-a29a-42bae9c214d2" containerID="dead18a097fb6d3cfa1a499c19328db85f1d91fae9e639adfccc2ed64c57ed58" exitCode=7
Jul 16 21:02:24.872079 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:24.871759    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j" event={"ID":"f5f3df62-653b-4a8c-a29a-42bae9c214d2","Type":"ContainerDied","Data":"dead18a097fb6d3cfa1a499c19328db85f1d91fae9e639adfccc2ed64c57ed58"}
Jul 16 21:02:24.872079 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:24.871920    2530 scope.go:117] "RemoveContainer" containerID="dead18a097fb6d3cfa1a499c19328db85f1d91fae9e639adfccc2ed64c57ed58"
Jul 16 21:02:25.883077 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:25.883043    2530 generic.go:356] "Generic (PLEG): container finished" podID="f5f3df62-653b-4a8c-a29a-42bae9c214d2" containerID="dbcbd131c50adea3cef87447a6bc0018e6907fd575ebc4a46c0ea21117dcec20" exitCode=7
Jul 16 21:02:25.883512 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:25.883128    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j" event={"ID":"f5f3df62-653b-4a8c-a29a-42bae9c214d2","Type":"ContainerDied","Data":"dbcbd131c50adea3cef87447a6bc0018e6907fd575ebc4a46c0ea21117dcec20"}
Jul 16 21:02:25.883512 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:25.883213    2530 scope.go:117] "RemoveContainer" containerID="dead18a097fb6d3cfa1a499c19328db85f1d91fae9e639adfccc2ed64c57ed58"
Jul 16 21:02:25.883512 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:25.883373    2530 scope.go:117] "RemoveContainer" containerID="dbcbd131c50adea3cef87447a6bc0018e6907fd575ebc4a46c0ea21117dcec20"
Jul 16 21:02:25.883644 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:02:25.883551    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j_odh-ai-gateway-infra(f5f3df62-653b-4a8c-a29a-42bae9c214d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j" podUID="f5f3df62-653b-4a8c-a29a-42bae9c214d2"
Jul 16 21:02:26.888910 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:26.888881    2530 scope.go:117] "RemoveContainer" containerID="dbcbd131c50adea3cef87447a6bc0018e6907fd575ebc4a46c0ea21117dcec20"
Jul 16 21:02:26.889322 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:02:26.889063    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j_odh-ai-gateway-infra(f5f3df62-653b-4a8c-a29a-42bae9c214d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j" podUID="f5f3df62-653b-4a8c-a29a-42bae9c214d2"
Jul 16 21:02:28.515211 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:28.515184    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 21:02:28.515574 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:28.515233    2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 21:02:28.515574 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:28.515551    2530 scope.go:117] "RemoveContainer" containerID="68e5445964e3d01d64f4bc53776fd575a3f0390dec0f45160dec91e0de601510"
Jul 16 21:02:28.515768 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:02:28.515749    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg_odh-ai-gateway-infra(6bf69704-cc7f-4737-8375-0618f45d11ec)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec"
Jul 16 21:02:28.940275 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:28.940146    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"]
Jul 16 21:02:29.075475 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:29.075450    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 21:02:29.232910 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:29.232885    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bf69704-cc7f-4737-8375-0618f45d11ec-maas-api-tls\") pod \"6bf69704-cc7f-4737-8375-0618f45d11ec\" (UID: \"6bf69704-cc7f-4737-8375-0618f45d11ec\") "
Jul 16 21:02:29.233059 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:29.232957    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf69704-cc7f-4737-8375-0618f45d11ec-openshift-service-ca\") pod \"6bf69704-cc7f-4737-8375-0618f45d11ec\" (UID: \"6bf69704-cc7f-4737-8375-0618f45d11ec\") "
Jul 16 21:02:29.233059 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:29.232983    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lmvj\" (UniqueName: \"kubernetes.io/projected/6bf69704-cc7f-4737-8375-0618f45d11ec-kube-api-access-2lmvj\") pod \"6bf69704-cc7f-4737-8375-0618f45d11ec\" (UID: \"6bf69704-cc7f-4737-8375-0618f45d11ec\") "
Jul 16 21:02:29.233388 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:29.233365    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf69704-cc7f-4737-8375-0618f45d11ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6bf69704-cc7f-4737-8375-0618f45d11ec" (UID: "6bf69704-cc7f-4737-8375-0618f45d11ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:02:29.235396 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:29.235369    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bf69704-cc7f-4737-8375-0618f45d11ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6bf69704-cc7f-4737-8375-0618f45d11ec" (UID: "6bf69704-cc7f-4737-8375-0618f45d11ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:02:29.235507 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:29.235392    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf69704-cc7f-4737-8375-0618f45d11ec-kube-api-access-2lmvj" (OuterVolumeSpecName: "kube-api-access-2lmvj") pod "6bf69704-cc7f-4737-8375-0618f45d11ec" (UID: "6bf69704-cc7f-4737-8375-0618f45d11ec"). InnerVolumeSpecName "kube-api-access-2lmvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:02:29.334220 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:29.334194    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bf69704-cc7f-4737-8375-0618f45d11ec-maas-api-tls\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:02:29.334220 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:29.334217    2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf69704-cc7f-4737-8375-0618f45d11ec-openshift-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:02:29.334359 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:29.334227    2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lmvj\" (UniqueName: \"kubernetes.io/projected/6bf69704-cc7f-4737-8375-0618f45d11ec-kube-api-access-2lmvj\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:02:29.902245 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:29.902214    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg" event={"ID":"6bf69704-cc7f-4737-8375-0618f45d11ec","Type":"ContainerDied","Data":"105276b9925dbf7f0b723468be21c093785d5bec58661f5e36dae48c90862dfd"}
Jul 16 21:02:29.902640 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:29.902276    2530 scope.go:117] "RemoveContainer" containerID="68e5445964e3d01d64f4bc53776fd575a3f0390dec0f45160dec91e0de601510"
Jul 16 21:02:29.902640 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:29.902310    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"
Jul 16 21:02:29.923643 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:29.923612    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"]
Jul 16 21:02:29.927269 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:29.927246    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23a5a19c-858cfff698-xd7rg"]
Jul 16 21:02:30.507295 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:30.507264    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" path="/var/lib/kubelet/pods/6bf69704-cc7f-4737-8375-0618f45d11ec/volumes"
Jul 16 21:02:36.961700 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:36.961671    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"]
Jul 16 21:02:36.962173 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:36.962150    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerName="maas-api"
Jul 16 21:02:36.962256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:36.962174    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerName="maas-api"
Jul 16 21:02:36.962256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:36.962188    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerName="maas-api"
Jul 16 21:02:36.962256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:36.962196    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerName="maas-api"
Jul 16 21:02:36.962256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:36.962206    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerName="maas-api"
Jul 16 21:02:36.962256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:36.962214    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerName="maas-api"
Jul 16 21:02:36.962256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:36.962235    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerName="maas-api"
Jul 16 21:02:36.962256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:36.962243    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerName="maas-api"
Jul 16 21:02:36.962562 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:36.962341    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerName="maas-api"
Jul 16 21:02:36.962562 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:36.962352    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerName="maas-api"
Jul 16 21:02:36.962562 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:36.962365    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef3664a6-9b0b-4d5e-a697-d2d243b75417" containerName="revoke-keys"
Jul 16 21:02:36.962562 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:36.962376    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerName="maas-api"
Jul 16 21:02:36.962562 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:36.962386    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerName="maas-api"
Jul 16 21:02:36.966885 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:36.966863    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:02:36.969463 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:36.969443    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ad09ff70-serving-cert\""
Jul 16 21:02:36.975894 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:36.975558    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"]
Jul 16 21:02:37.097126 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:37.097070    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bw4f\" (UniqueName: \"kubernetes.io/projected/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4-kube-api-access-8bw4f\") pod \"maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7\" (UID: \"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:02:37.097272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:37.097151    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4-openshift-service-ca\") pod \"maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7\" (UID: \"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:02:37.097272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:37.097194    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4-maas-api-tls\") pod \"maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7\" (UID: \"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:02:37.198284 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:37.198253    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4-maas-api-tls\") pod \"maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7\" (UID: \"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:02:37.198423 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:37.198326    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bw4f\" (UniqueName: \"kubernetes.io/projected/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4-kube-api-access-8bw4f\") pod \"maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7\" (UID: \"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:02:37.198423 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:37.198354    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4-openshift-service-ca\") pod \"maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7\" (UID: \"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:02:37.199002 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:37.198981    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4-openshift-service-ca\") pod \"maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7\" (UID: \"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:02:37.200752 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:37.200728    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4-maas-api-tls\") pod \"maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7\" (UID: \"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:02:37.207913 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:37.207895    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bw4f\" (UniqueName: \"kubernetes.io/projected/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4-kube-api-access-8bw4f\") pod \"maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7\" (UID: \"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:02:37.280214 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:37.280167    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:02:37.411813 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:37.411791    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"]
Jul 16 21:02:37.413768 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:02:37.413742    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e04ef8c_3f1b_4c0d_9b2a_234c5a76c1b4.slice/crio-ec7a78f9cda55d41e5448a790fd3065005c670e50c7c85c0b7603d1e872c96d4 WatchSource:0}: Error finding container ec7a78f9cda55d41e5448a790fd3065005c670e50c7c85c0b7603d1e872c96d4: Status 404 returned error can't find the container with id ec7a78f9cda55d41e5448a790fd3065005c670e50c7c85c0b7603d1e872c96d4
Jul 16 21:02:37.937092 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:37.937064    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" event={"ID":"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4","Type":"ContainerStarted","Data":"c96a3d7956def48d5d81d416372abf142d678f5b7f9001f32244327d69ef5d8f"}
Jul 16 21:02:37.937263 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:37.937128    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" event={"ID":"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4","Type":"ContainerStarted","Data":"ec7a78f9cda55d41e5448a790fd3065005c670e50c7c85c0b7603d1e872c96d4"}
Jul 16 21:02:37.937263 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:37.937159    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:02:37.987812 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:37.987751    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" podStartSLOduration=1.7083024020000002 podStartE2EDuration="1.987733681s" totalImagesPullingTime="279.431279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:02:37.984789318 +0000 UTC m=+1796.006377086" watchObservedRunningTime="2026-07-16 21:02:37.987733681 +0000 UTC m=+1796.009321450"
Jul 16 21:02:38.943772 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:38.943729    2530 generic.go:356] "Generic (PLEG): container finished" podID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" containerID="c96a3d7956def48d5d81d416372abf142d678f5b7f9001f32244327d69ef5d8f" exitCode=1
Jul 16 21:02:38.943772 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:38.943777    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" event={"ID":"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4","Type":"ContainerDied","Data":"c96a3d7956def48d5d81d416372abf142d678f5b7f9001f32244327d69ef5d8f"}
Jul 16 21:02:38.944066 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:38.944048    2530 scope.go:117] "RemoveContainer" containerID="c96a3d7956def48d5d81d416372abf142d678f5b7f9001f32244327d69ef5d8f"
Jul 16 21:02:39.952204 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:39.952127    2530 generic.go:356] "Generic (PLEG): container finished" podID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" containerID="7567da0ff5d80f9e0bf9acce39699ad92519321ece4fee6dc73072d973197e2d" exitCode=1
Jul 16 21:02:39.952614 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:39.952215    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" event={"ID":"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4","Type":"ContainerDied","Data":"7567da0ff5d80f9e0bf9acce39699ad92519321ece4fee6dc73072d973197e2d"}
Jul 16 21:02:39.952614 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:39.952292    2530 scope.go:117] "RemoveContainer" containerID="c96a3d7956def48d5d81d416372abf142d678f5b7f9001f32244327d69ef5d8f"
Jul 16 21:02:39.952614 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:39.952517    2530 scope.go:117] "RemoveContainer" containerID="7567da0ff5d80f9e0bf9acce39699ad92519321ece4fee6dc73072d973197e2d"
Jul 16 21:02:39.952756 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:02:39.952736    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7_odh-ai-gateway-infra(0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4"
Jul 16 21:02:40.958350 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:40.958330    2530 scope.go:117] "RemoveContainer" containerID="7567da0ff5d80f9e0bf9acce39699ad92519321ece4fee6dc73072d973197e2d"
Jul 16 21:02:40.958813 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:02:40.958549    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7_odh-ai-gateway-infra(0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4"
Jul 16 21:02:41.323240 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:41.323163    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f"]
Jul 16 21:02:41.323580 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:41.323562    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerName="maas-api"
Jul 16 21:02:41.323661 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:41.323583    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerName="maas-api"
Jul 16 21:02:41.323661 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:41.323606    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerName="maas-api"
Jul 16 21:02:41.323661 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:41.323615    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerName="maas-api"
Jul 16 21:02:41.323800 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:41.323708    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerName="maas-api"
Jul 16 21:02:41.323800 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:41.323721    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bf69704-cc7f-4737-8375-0618f45d11ec" containerName="maas-api"
Jul 16 21:02:41.326705 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:41.326685    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f"
Jul 16 21:02:41.335356 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:41.335328    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f7b58dd-3fc1-485d-9f18-70e1a45664c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f\" (UID: \"7f7b58dd-3fc1-485d-9f18-70e1a45664c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f"
Jul 16 21:02:41.335584 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:41.335566    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f"]
Jul 16 21:02:41.436531 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:41.436507    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f7b58dd-3fc1-485d-9f18-70e1a45664c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f\" (UID: \"7f7b58dd-3fc1-485d-9f18-70e1a45664c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f"
Jul 16 21:02:41.437045 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:41.437028    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f7b58dd-3fc1-485d-9f18-70e1a45664c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f\" (UID: \"7f7b58dd-3fc1-485d-9f18-70e1a45664c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f"
Jul 16 21:02:41.502065 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:41.502045    2530 scope.go:117] "RemoveContainer" containerID="dbcbd131c50adea3cef87447a6bc0018e6907fd575ebc4a46c0ea21117dcec20"
Jul 16 21:02:41.638416 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:41.638369    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f"
Jul 16 21:02:41.768968 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:41.768944    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f"]
Jul 16 21:02:41.818059 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:02:41.818009    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7b58dd_3fc1_485d_9f18_70e1a45664c4.slice/crio-be77f0c82c95192497a10be1eac1b57b2e6db2f7e2ca78668cc2ec6e7f99546d WatchSource:0}: Error finding container be77f0c82c95192497a10be1eac1b57b2e6db2f7e2ca78668cc2ec6e7f99546d: Status 404 returned error can't find the container with id be77f0c82c95192497a10be1eac1b57b2e6db2f7e2ca78668cc2ec6e7f99546d
Jul 16 21:02:41.964759 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:41.964732    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j" event={"ID":"f5f3df62-653b-4a8c-a29a-42bae9c214d2","Type":"ContainerStarted","Data":"4d0f550be8ca2e1a693035c7f77b159a007f4941126f0564573ed422fb24795a"}
Jul 16 21:02:41.965912 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:41.965891    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f" event={"ID":"7f7b58dd-3fc1-485d-9f18-70e1a45664c4","Type":"ContainerStarted","Data":"be77f0c82c95192497a10be1eac1b57b2e6db2f7e2ca78668cc2ec6e7f99546d"}
Jul 16 21:02:42.938188 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:42.938163    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:02:42.938507 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:42.938494    2530 scope.go:117] "RemoveContainer" containerID="7567da0ff5d80f9e0bf9acce39699ad92519321ece4fee6dc73072d973197e2d"
Jul 16 21:02:42.938664 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:02:42.938644    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7_odh-ai-gateway-infra(0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4"
Jul 16 21:02:42.971683 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:42.971659    2530 generic.go:356] "Generic (PLEG): container finished" podID="f5f3df62-653b-4a8c-a29a-42bae9c214d2" containerID="4d0f550be8ca2e1a693035c7f77b159a007f4941126f0564573ed422fb24795a" exitCode=6
Jul 16 21:02:42.972060 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:42.971739    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j" event={"ID":"f5f3df62-653b-4a8c-a29a-42bae9c214d2","Type":"ContainerDied","Data":"4d0f550be8ca2e1a693035c7f77b159a007f4941126f0564573ed422fb24795a"}
Jul 16 21:02:42.972060 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:42.971802    2530 scope.go:117] "RemoveContainer" containerID="dbcbd131c50adea3cef87447a6bc0018e6907fd575ebc4a46c0ea21117dcec20"
Jul 16 21:02:42.972060 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:42.972031    2530 scope.go:117] "RemoveContainer" containerID="4d0f550be8ca2e1a693035c7f77b159a007f4941126f0564573ed422fb24795a"
Jul 16 21:02:42.972247 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:02:42.972225    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j_odh-ai-gateway-infra(f5f3df62-653b-4a8c-a29a-42bae9c214d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j" podUID="f5f3df62-653b-4a8c-a29a-42bae9c214d2"
Jul 16 21:02:42.973491 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:42.973470    2530 generic.go:356] "Generic (PLEG): container finished" podID="7f7b58dd-3fc1-485d-9f18-70e1a45664c4" containerID="15aea6df7edf3942bac9da6fa0087efc22a8ae0bc1a1b28cb0d262710a8b1305" exitCode=7
Jul 16 21:02:42.973619 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:42.973504    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f" event={"ID":"7f7b58dd-3fc1-485d-9f18-70e1a45664c4","Type":"ContainerDied","Data":"15aea6df7edf3942bac9da6fa0087efc22a8ae0bc1a1b28cb0d262710a8b1305"}
Jul 16 21:02:42.973696 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:42.973657    2530 scope.go:117] "RemoveContainer" containerID="15aea6df7edf3942bac9da6fa0087efc22a8ae0bc1a1b28cb0d262710a8b1305"
Jul 16 21:02:42.996900 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:42.996876    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j"]
Jul 16 21:02:43.981473 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:43.981441    2530 generic.go:356] "Generic (PLEG): container finished" podID="7f7b58dd-3fc1-485d-9f18-70e1a45664c4" containerID="d3beace24576ced0145fe4a417b040c5da1973a3d64fa13ee23310f556e81c77" exitCode=7
Jul 16 21:02:43.981775 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:43.981485    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f" event={"ID":"7f7b58dd-3fc1-485d-9f18-70e1a45664c4","Type":"ContainerDied","Data":"d3beace24576ced0145fe4a417b040c5da1973a3d64fa13ee23310f556e81c77"}
Jul 16 21:02:43.981775 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:43.981539    2530 scope.go:117] "RemoveContainer" containerID="15aea6df7edf3942bac9da6fa0087efc22a8ae0bc1a1b28cb0d262710a8b1305"
Jul 16 21:02:43.981775 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:43.981690    2530 scope.go:117] "RemoveContainer" containerID="d3beace24576ced0145fe4a417b040c5da1973a3d64fa13ee23310f556e81c77"
Jul 16 21:02:43.981907 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:02:43.981888    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f_odh-ai-gateway-infra(7f7b58dd-3fc1-485d-9f18-70e1a45664c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f" podUID="7f7b58dd-3fc1-485d-9f18-70e1a45664c4"
Jul 16 21:02:44.111271 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:44.111242    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j"
Jul 16 21:02:44.153952 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:44.153928    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f3df62-653b-4a8c-a29a-42bae9c214d2-maas-api-service-ca\") pod \"f5f3df62-653b-4a8c-a29a-42bae9c214d2\" (UID: \"f5f3df62-653b-4a8c-a29a-42bae9c214d2\") "
Jul 16 21:02:44.154306 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:44.154285    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f3df62-653b-4a8c-a29a-42bae9c214d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5f3df62-653b-4a8c-a29a-42bae9c214d2" (UID: "f5f3df62-653b-4a8c-a29a-42bae9c214d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:02:44.255169 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:44.255088    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f3df62-653b-4a8c-a29a-42bae9c214d2-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:02:44.987071 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:44.987045    2530 scope.go:117] "RemoveContainer" containerID="d3beace24576ced0145fe4a417b040c5da1973a3d64fa13ee23310f556e81c77"
Jul 16 21:02:44.987481 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:02:44.987269    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f_odh-ai-gateway-infra(7f7b58dd-3fc1-485d-9f18-70e1a45664c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f" podUID="7f7b58dd-3fc1-485d-9f18-70e1a45664c4"
Jul 16 21:02:44.988068 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:44.988037    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j" event={"ID":"f5f3df62-653b-4a8c-a29a-42bae9c214d2","Type":"ContainerDied","Data":"c861789bd6ed0ce6ac0cf3df8b482e940cc6fd262305289cdeb3568ab8914393"}
Jul 16 21:02:44.988167 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:44.988076    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j"
Jul 16 21:02:44.988167 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:44.988121    2530 scope.go:117] "RemoveContainer" containerID="4d0f550be8ca2e1a693035c7f77b159a007f4941126f0564573ed422fb24795a"
Jul 16 21:02:45.016270 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:45.016247    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j"]
Jul 16 21:02:45.021422 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:45.021398    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-7r52j"]
Jul 16 21:02:46.507030 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:46.506997    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5f3df62-653b-4a8c-a29a-42bae9c214d2" path="/var/lib/kubelet/pods/f5f3df62-653b-4a8c-a29a-42bae9c214d2/volumes"
Jul 16 21:02:47.280667 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:47.280626    2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:02:47.281010 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:47.280996    2530 scope.go:117] "RemoveContainer" containerID="7567da0ff5d80f9e0bf9acce39699ad92519321ece4fee6dc73072d973197e2d"
Jul 16 21:02:47.281202 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:02:47.281181    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7_odh-ai-gateway-infra(0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4"
Jul 16 21:02:55.502807 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:55.502773    2530 scope.go:117] "RemoveContainer" containerID="d3beace24576ced0145fe4a417b040c5da1973a3d64fa13ee23310f556e81c77"
Jul 16 21:02:56.038486 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:56.038455    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f" event={"ID":"7f7b58dd-3fc1-485d-9f18-70e1a45664c4","Type":"ContainerStarted","Data":"c3c2039b5e444d2ae45a37e5417c5374354c8e1c452d72a4dede67d258970467"}
Jul 16 21:02:56.053791 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:56.053750    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f" podStartSLOduration=15.053735022 podStartE2EDuration="15.053735022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:02:56.052355484 +0000 UTC m=+1814.073943251" watchObservedRunningTime="2026-07-16 21:02:56.053735022 +0000 UTC m=+1814.075322821"
Jul 16 21:02:57.044929 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:57.044897    2530 generic.go:356] "Generic (PLEG): container finished" podID="7f7b58dd-3fc1-485d-9f18-70e1a45664c4" containerID="c3c2039b5e444d2ae45a37e5417c5374354c8e1c452d72a4dede67d258970467" exitCode=7
Jul 16 21:02:57.045405 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:57.044967    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f" event={"ID":"7f7b58dd-3fc1-485d-9f18-70e1a45664c4","Type":"ContainerDied","Data":"c3c2039b5e444d2ae45a37e5417c5374354c8e1c452d72a4dede67d258970467"}
Jul 16 21:02:57.045405 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:57.045024    2530 scope.go:117] "RemoveContainer" containerID="d3beace24576ced0145fe4a417b040c5da1973a3d64fa13ee23310f556e81c77"
Jul 16 21:02:57.045405 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:57.045312    2530 scope.go:117] "RemoveContainer" containerID="c3c2039b5e444d2ae45a37e5417c5374354c8e1c452d72a4dede67d258970467"
Jul 16 21:02:57.045565 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:02:57.045526    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f_odh-ai-gateway-infra(7f7b58dd-3fc1-485d-9f18-70e1a45664c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f" podUID="7f7b58dd-3fc1-485d-9f18-70e1a45664c4"
Jul 16 21:02:57.061634 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:57.061611    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f"]
Jul 16 21:02:58.192558 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:58.192534    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f"
Jul 16 21:02:58.260547 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:58.260518    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f7b58dd-3fc1-485d-9f18-70e1a45664c4-maas-api-service-ca\") pod \"7f7b58dd-3fc1-485d-9f18-70e1a45664c4\" (UID: \"7f7b58dd-3fc1-485d-9f18-70e1a45664c4\") "
Jul 16 21:02:58.260846 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:58.260824    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7b58dd-3fc1-485d-9f18-70e1a45664c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7f7b58dd-3fc1-485d-9f18-70e1a45664c4" (UID: "7f7b58dd-3fc1-485d-9f18-70e1a45664c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:02:58.361803 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:58.361746    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f7b58dd-3fc1-485d-9f18-70e1a45664c4-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:02:59.065942 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:59.065901    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f" event={"ID":"7f7b58dd-3fc1-485d-9f18-70e1a45664c4","Type":"ContainerDied","Data":"be77f0c82c95192497a10be1eac1b57b2e6db2f7e2ca78668cc2ec6e7f99546d"}
Jul 16 21:02:59.065942 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:59.065926    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f"
Jul 16 21:02:59.066234 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:59.065985    2530 scope.go:117] "RemoveContainer" containerID="c3c2039b5e444d2ae45a37e5417c5374354c8e1c452d72a4dede67d258970467"
Jul 16 21:02:59.084646 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:59.084621    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f"]
Jul 16 21:02:59.089892 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:59.089864    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-t2m7f"]
Jul 16 21:02:59.502627 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:02:59.502602    2530 scope.go:117] "RemoveContainer" containerID="7567da0ff5d80f9e0bf9acce39699ad92519321ece4fee6dc73072d973197e2d"
Jul 16 21:03:00.506851 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:00.506819    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f7b58dd-3fc1-485d-9f18-70e1a45664c4" path="/var/lib/kubelet/pods/7f7b58dd-3fc1-485d-9f18-70e1a45664c4/volumes"
Jul 16 21:03:03.085994 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:03.085956    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" event={"ID":"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4","Type":"ContainerStarted","Data":"0bcb89352f529190a05d7eaefd92d0de10e21e7055a76abbaa0738010c931510"}
Jul 16 21:03:03.086413 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:03.086210    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:03:04.092881 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:04.092846    2530 generic.go:356] "Generic (PLEG): container finished" podID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" containerID="0bcb89352f529190a05d7eaefd92d0de10e21e7055a76abbaa0738010c931510" exitCode=1
Jul 16 21:03:04.093314 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:04.092931    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" event={"ID":"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4","Type":"ContainerDied","Data":"0bcb89352f529190a05d7eaefd92d0de10e21e7055a76abbaa0738010c931510"}
Jul 16 21:03:04.093314 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:04.092989    2530 scope.go:117] "RemoveContainer" containerID="7567da0ff5d80f9e0bf9acce39699ad92519321ece4fee6dc73072d973197e2d"
Jul 16 21:03:04.093314 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:04.093291    2530 scope.go:117] "RemoveContainer" containerID="0bcb89352f529190a05d7eaefd92d0de10e21e7055a76abbaa0738010c931510"
Jul 16 21:03:04.093527 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:03:04.093507    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7_odh-ai-gateway-infra(0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4"
Jul 16 21:03:05.099364 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:05.099336    2530 scope.go:117] "RemoveContainer" containerID="0bcb89352f529190a05d7eaefd92d0de10e21e7055a76abbaa0738010c931510"
Jul 16 21:03:05.099860 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:03:05.099581    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7_odh-ai-gateway-infra(0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4"
Jul 16 21:03:07.281041 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:07.280979    2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:03:07.281503 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:07.281485    2530 scope.go:117] "RemoveContainer" containerID="0bcb89352f529190a05d7eaefd92d0de10e21e7055a76abbaa0738010c931510"
Jul 16 21:03:07.281674 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:03:07.281653    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7_odh-ai-gateway-infra(0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4"
Jul 16 21:03:21.502223 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:21.502195    2530 scope.go:117] "RemoveContainer" containerID="0bcb89352f529190a05d7eaefd92d0de10e21e7055a76abbaa0738010c931510"
Jul 16 21:03:21.502708 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:03:21.502374    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7_odh-ai-gateway-infra(0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4"
Jul 16 21:03:31.365959 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.365924    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c"]
Jul 16 21:03:31.366355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.366269    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f3df62-653b-4a8c-a29a-42bae9c214d2" containerName="revoke-keys"
Jul 16 21:03:31.366355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.366284    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f3df62-653b-4a8c-a29a-42bae9c214d2" containerName="revoke-keys"
Jul 16 21:03:31.366355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.366304    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7b58dd-3fc1-485d-9f18-70e1a45664c4" containerName="revoke-keys"
Jul 16 21:03:31.366355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.366309    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7b58dd-3fc1-485d-9f18-70e1a45664c4" containerName="revoke-keys"
Jul 16 21:03:31.366355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.366316    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f3df62-653b-4a8c-a29a-42bae9c214d2" containerName="revoke-keys"
Jul 16 21:03:31.366355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.366321    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f3df62-653b-4a8c-a29a-42bae9c214d2" containerName="revoke-keys"
Jul 16 21:03:31.366355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.366330    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7b58dd-3fc1-485d-9f18-70e1a45664c4" containerName="revoke-keys"
Jul 16 21:03:31.366355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.366335    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7b58dd-3fc1-485d-9f18-70e1a45664c4" containerName="revoke-keys"
Jul 16 21:03:31.366355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.366343    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f3df62-653b-4a8c-a29a-42bae9c214d2" containerName="revoke-keys"
Jul 16 21:03:31.366355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.366349    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f3df62-653b-4a8c-a29a-42bae9c214d2" containerName="revoke-keys"
Jul 16 21:03:31.366647 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.366360    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7b58dd-3fc1-485d-9f18-70e1a45664c4" containerName="revoke-keys"
Jul 16 21:03:31.366647 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.366368    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7b58dd-3fc1-485d-9f18-70e1a45664c4" containerName="revoke-keys"
Jul 16 21:03:31.366647 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.366430    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5f3df62-653b-4a8c-a29a-42bae9c214d2" containerName="revoke-keys"
Jul 16 21:03:31.366647 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.366440    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f7b58dd-3fc1-485d-9f18-70e1a45664c4" containerName="revoke-keys"
Jul 16 21:03:31.366647 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.366449    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f7b58dd-3fc1-485d-9f18-70e1a45664c4" containerName="revoke-keys"
Jul 16 21:03:31.366647 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.366454    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5f3df62-653b-4a8c-a29a-42bae9c214d2" containerName="revoke-keys"
Jul 16 21:03:31.366647 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.366461    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f7b58dd-3fc1-485d-9f18-70e1a45664c4" containerName="revoke-keys"
Jul 16 21:03:31.369302 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.369286    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c"
Jul 16 21:03:31.371736 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.371713    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:03:31.377166 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.377138    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c"]
Jul 16 21:03:31.552417 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.552365    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06539da2-47aa-4c8c-bb2a-1c665adcfa75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c\" (UID: \"06539da2-47aa-4c8c-bb2a-1c665adcfa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c"
Jul 16 21:03:31.653754 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.653657    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06539da2-47aa-4c8c-bb2a-1c665adcfa75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c\" (UID: \"06539da2-47aa-4c8c-bb2a-1c665adcfa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c"
Jul 16 21:03:31.654375 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.654350    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06539da2-47aa-4c8c-bb2a-1c665adcfa75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c\" (UID: \"06539da2-47aa-4c8c-bb2a-1c665adcfa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c"
Jul 16 21:03:31.680733 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.680707    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c"
Jul 16 21:03:31.810265 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:31.810242    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c"]
Jul 16 21:03:31.812237 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:03:31.812209    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06539da2_47aa_4c8c_bb2a_1c665adcfa75.slice/crio-a082ecc9dabccf49108c0fd2e201bf322a9fd299f4c5d583384052c2f1f39dda WatchSource:0}: Error finding container a082ecc9dabccf49108c0fd2e201bf322a9fd299f4c5d583384052c2f1f39dda: Status 404 returned error can't find the container with id a082ecc9dabccf49108c0fd2e201bf322a9fd299f4c5d583384052c2f1f39dda
Jul 16 21:03:32.214641 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:32.214608    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c" event={"ID":"06539da2-47aa-4c8c-bb2a-1c665adcfa75","Type":"ContainerStarted","Data":"51dec90721299f2a2baec6fb459adfa82e768fecd352e5cebc7a6a27db2e2d64"}
Jul 16 21:03:32.214801 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:32.214671    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c" event={"ID":"06539da2-47aa-4c8c-bb2a-1c665adcfa75","Type":"ContainerStarted","Data":"a082ecc9dabccf49108c0fd2e201bf322a9fd299f4c5d583384052c2f1f39dda"}
Jul 16 21:03:32.231167 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:32.231121    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c" podStartSLOduration=1.2310874410000001 podStartE2EDuration="1.231087441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:03:32.229734169 +0000 UTC m=+1850.251321936" watchObservedRunningTime="2026-07-16 21:03:32.231087441 +0000 UTC m=+1850.252675206"
Jul 16 21:03:33.220279 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:33.220241    2530 generic.go:356] "Generic (PLEG): container finished" podID="06539da2-47aa-4c8c-bb2a-1c665adcfa75" containerID="51dec90721299f2a2baec6fb459adfa82e768fecd352e5cebc7a6a27db2e2d64" exitCode=7
Jul 16 21:03:33.220632 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:33.220333    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c" event={"ID":"06539da2-47aa-4c8c-bb2a-1c665adcfa75","Type":"ContainerDied","Data":"51dec90721299f2a2baec6fb459adfa82e768fecd352e5cebc7a6a27db2e2d64"}
Jul 16 21:03:33.220632 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:33.220442    2530 scope.go:117] "RemoveContainer" containerID="51dec90721299f2a2baec6fb459adfa82e768fecd352e5cebc7a6a27db2e2d64"
Jul 16 21:03:34.227458 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:34.227419    2530 generic.go:356] "Generic (PLEG): container finished" podID="06539da2-47aa-4c8c-bb2a-1c665adcfa75" containerID="f2d180bde52119c3feb4a9d1aa240355158cb649e5d8e8d6411ec98ce18c3a92" exitCode=7
Jul 16 21:03:34.227866 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:34.227503    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c" event={"ID":"06539da2-47aa-4c8c-bb2a-1c665adcfa75","Type":"ContainerDied","Data":"f2d180bde52119c3feb4a9d1aa240355158cb649e5d8e8d6411ec98ce18c3a92"}
Jul 16 21:03:34.227866 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:34.227600    2530 scope.go:117] "RemoveContainer" containerID="51dec90721299f2a2baec6fb459adfa82e768fecd352e5cebc7a6a27db2e2d64"
Jul 16 21:03:34.227866 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:34.227658    2530 scope.go:117] "RemoveContainer" containerID="f2d180bde52119c3feb4a9d1aa240355158cb649e5d8e8d6411ec98ce18c3a92"
Jul 16 21:03:34.227866 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:03:34.227860    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c_odh-ai-gateway-infra(06539da2-47aa-4c8c-bb2a-1c665adcfa75)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c" podUID="06539da2-47aa-4c8c-bb2a-1c665adcfa75"
Jul 16 21:03:34.502805 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:34.502720    2530 scope.go:117] "RemoveContainer" containerID="0bcb89352f529190a05d7eaefd92d0de10e21e7055a76abbaa0738010c931510"
Jul 16 21:03:35.235084 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:35.235044    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" event={"ID":"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4","Type":"ContainerStarted","Data":"02c8f546ae6149539a0fc2cd53ef72365f9856a1d05f1dcb132d79f5af9cb058"}
Jul 16 21:03:35.235602 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:35.235314    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:03:35.237057 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:35.237036    2530 scope.go:117] "RemoveContainer" containerID="f2d180bde52119c3feb4a9d1aa240355158cb649e5d8e8d6411ec98ce18c3a92"
Jul 16 21:03:35.237298 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:03:35.237258    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c_odh-ai-gateway-infra(06539da2-47aa-4c8c-bb2a-1c665adcfa75)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c" podUID="06539da2-47aa-4c8c-bb2a-1c665adcfa75"
Jul 16 21:03:36.242901 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:36.242867    2530 generic.go:356] "Generic (PLEG): container finished" podID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" containerID="02c8f546ae6149539a0fc2cd53ef72365f9856a1d05f1dcb132d79f5af9cb058" exitCode=1
Jul 16 21:03:36.243370 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:36.242949    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" event={"ID":"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4","Type":"ContainerDied","Data":"02c8f546ae6149539a0fc2cd53ef72365f9856a1d05f1dcb132d79f5af9cb058"}
Jul 16 21:03:36.243370 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:36.242992    2530 scope.go:117] "RemoveContainer" containerID="0bcb89352f529190a05d7eaefd92d0de10e21e7055a76abbaa0738010c931510"
Jul 16 21:03:36.243370 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:36.243213    2530 scope.go:117] "RemoveContainer" containerID="02c8f546ae6149539a0fc2cd53ef72365f9856a1d05f1dcb132d79f5af9cb058"
Jul 16 21:03:36.243502 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:03:36.243401    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7_odh-ai-gateway-infra(0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4"
Jul 16 21:03:37.249267 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:37.249237    2530 scope.go:117] "RemoveContainer" containerID="02c8f546ae6149539a0fc2cd53ef72365f9856a1d05f1dcb132d79f5af9cb058"
Jul 16 21:03:37.249696 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:03:37.249434    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7_odh-ai-gateway-infra(0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4"
Jul 16 21:03:37.280903 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:37.280855    2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:03:38.254623 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:38.254596    2530 scope.go:117] "RemoveContainer" containerID="02c8f546ae6149539a0fc2cd53ef72365f9856a1d05f1dcb132d79f5af9cb058"
Jul 16 21:03:38.255039 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:03:38.254768    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7_odh-ai-gateway-infra(0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4"
Jul 16 21:03:45.502769 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:45.502731    2530 scope.go:117] "RemoveContainer" containerID="f2d180bde52119c3feb4a9d1aa240355158cb649e5d8e8d6411ec98ce18c3a92"
Jul 16 21:03:46.301731 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:46.301691    2530 generic.go:356] "Generic (PLEG): container finished" podID="06539da2-47aa-4c8c-bb2a-1c665adcfa75" containerID="1acd57b06f9901da2a35c6d334ae9021bd0aba8c0f63a05b215f0c5504c0d32a" exitCode=7
Jul 16 21:03:46.301927 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:46.301767    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c" event={"ID":"06539da2-47aa-4c8c-bb2a-1c665adcfa75","Type":"ContainerDied","Data":"1acd57b06f9901da2a35c6d334ae9021bd0aba8c0f63a05b215f0c5504c0d32a"}
Jul 16 21:03:46.301927 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:46.301860    2530 scope.go:117] "RemoveContainer" containerID="f2d180bde52119c3feb4a9d1aa240355158cb649e5d8e8d6411ec98ce18c3a92"
Jul 16 21:03:46.302184 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:46.302162    2530 scope.go:117] "RemoveContainer" containerID="1acd57b06f9901da2a35c6d334ae9021bd0aba8c0f63a05b215f0c5504c0d32a"
Jul 16 21:03:46.302405 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:03:46.302386    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c_odh-ai-gateway-infra(06539da2-47aa-4c8c-bb2a-1c665adcfa75)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c" podUID="06539da2-47aa-4c8c-bb2a-1c665adcfa75"
Jul 16 21:03:46.828402 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:46.828363    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c"]
Jul 16 21:03:47.445368 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:47.445339    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c"
Jul 16 21:03:47.493785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:47.493729    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06539da2-47aa-4c8c-bb2a-1c665adcfa75-maas-api-service-ca\") pod \"06539da2-47aa-4c8c-bb2a-1c665adcfa75\" (UID: \"06539da2-47aa-4c8c-bb2a-1c665adcfa75\") "
Jul 16 21:03:47.494126 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:47.494075    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06539da2-47aa-4c8c-bb2a-1c665adcfa75-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "06539da2-47aa-4c8c-bb2a-1c665adcfa75" (UID: "06539da2-47aa-4c8c-bb2a-1c665adcfa75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:03:47.595349 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:47.595252    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/06539da2-47aa-4c8c-bb2a-1c665adcfa75-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:03:47.853214 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:47.853130    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"]
Jul 16 21:03:47.995115 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:47.995068    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:03:48.098791 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.098755    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4-maas-api-tls\") pod \"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4\" (UID: \"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4\") "
Jul 16 21:03:48.098977 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.098823    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bw4f\" (UniqueName: \"kubernetes.io/projected/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4-kube-api-access-8bw4f\") pod \"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4\" (UID: \"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4\") "
Jul 16 21:03:48.098977 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.098844    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4-openshift-service-ca\") pod \"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4\" (UID: \"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4\") "
Jul 16 21:03:48.099273 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.099248    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" (UID: "0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:03:48.101147 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.101120    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" (UID: "0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:03:48.101255 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.101230    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4-kube-api-access-8bw4f" (OuterVolumeSpecName: "kube-api-access-8bw4f") pod "0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" (UID: "0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4"). InnerVolumeSpecName "kube-api-access-8bw4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:03:48.199668 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.199625    2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bw4f\" (UniqueName: \"kubernetes.io/projected/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4-kube-api-access-8bw4f\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:03:48.199668 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.199661    2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4-openshift-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:03:48.199668 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.199671    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4-maas-api-tls\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:03:48.313394 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.313362    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"
Jul 16 21:03:48.313591 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.313363    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7" event={"ID":"0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4","Type":"ContainerDied","Data":"ec7a78f9cda55d41e5448a790fd3065005c670e50c7c85c0b7603d1e872c96d4"}
Jul 16 21:03:48.313591 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.313500    2530 scope.go:117] "RemoveContainer" containerID="02c8f546ae6149539a0fc2cd53ef72365f9856a1d05f1dcb132d79f5af9cb058"
Jul 16 21:03:48.314725 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.314714    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c"
Jul 16 21:03:48.314792 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.314718    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c" event={"ID":"06539da2-47aa-4c8c-bb2a-1c665adcfa75","Type":"ContainerDied","Data":"a082ecc9dabccf49108c0fd2e201bf322a9fd299f4c5d583384052c2f1f39dda"}
Jul 16 21:03:48.324329 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.324305    2530 scope.go:117] "RemoveContainer" containerID="1acd57b06f9901da2a35c6d334ae9021bd0aba8c0f63a05b215f0c5504c0d32a"
Jul 16 21:03:48.342187 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.342154    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"]
Jul 16 21:03:48.347932 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.347902    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ad09ff70-5c7b984745-pwnx7"]
Jul 16 21:03:48.362378 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.362288    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c"]
Jul 16 21:03:48.366427 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.366400    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-9s97c"]
Jul 16 21:03:48.508219 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.508178    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06539da2-47aa-4c8c-bb2a-1c665adcfa75" path="/var/lib/kubelet/pods/06539da2-47aa-4c8c-bb2a-1c665adcfa75/volumes"
Jul 16 21:03:48.508663 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:48.508643    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" path="/var/lib/kubelet/pods/0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4/volumes"
Jul 16 21:03:55.827425 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827390    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"]
Jul 16 21:03:55.827941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827704    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" containerName="maas-api"
Jul 16 21:03:55.827941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827715    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" containerName="maas-api"
Jul 16 21:03:55.827941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827725    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" containerName="maas-api"
Jul 16 21:03:55.827941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827732    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" containerName="maas-api"
Jul 16 21:03:55.827941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827742    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06539da2-47aa-4c8c-bb2a-1c665adcfa75" containerName="revoke-keys"
Jul 16 21:03:55.827941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827747    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="06539da2-47aa-4c8c-bb2a-1c665adcfa75" containerName="revoke-keys"
Jul 16 21:03:55.827941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827753    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" containerName="maas-api"
Jul 16 21:03:55.827941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827757    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" containerName="maas-api"
Jul 16 21:03:55.827941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827773    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06539da2-47aa-4c8c-bb2a-1c665adcfa75" containerName="revoke-keys"
Jul 16 21:03:55.827941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827778    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="06539da2-47aa-4c8c-bb2a-1c665adcfa75" containerName="revoke-keys"
Jul 16 21:03:55.827941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827784    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06539da2-47aa-4c8c-bb2a-1c665adcfa75" containerName="revoke-keys"
Jul 16 21:03:55.827941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827789    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="06539da2-47aa-4c8c-bb2a-1c665adcfa75" containerName="revoke-keys"
Jul 16 21:03:55.827941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827834    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="06539da2-47aa-4c8c-bb2a-1c665adcfa75" containerName="revoke-keys"
Jul 16 21:03:55.827941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827840    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" containerName="maas-api"
Jul 16 21:03:55.827941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827845    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="06539da2-47aa-4c8c-bb2a-1c665adcfa75" containerName="revoke-keys"
Jul 16 21:03:55.827941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827851    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5f3df62-653b-4a8c-a29a-42bae9c214d2" containerName="revoke-keys"
Jul 16 21:03:55.827941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827856    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" containerName="maas-api"
Jul 16 21:03:55.827941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.827864    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" containerName="maas-api"
Jul 16 21:03:55.832249 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.832226    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:03:55.835314 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.835286    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e65739ab-serving-cert\""
Jul 16 21:03:55.849774 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.849731    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"]
Jul 16 21:03:55.968322 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.968033    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa609bb2-28bf-44e9-8c1c-3125dea60565-openshift-service-ca\") pod \"maas-api-e2e-ait-e65739ab-f74c8f957-5slqt\" (UID: \"aa609bb2-28bf-44e9-8c1c-3125dea60565\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:03:55.968322 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.968133    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjnk2\" (UniqueName: \"kubernetes.io/projected/aa609bb2-28bf-44e9-8c1c-3125dea60565-kube-api-access-rjnk2\") pod \"maas-api-e2e-ait-e65739ab-f74c8f957-5slqt\" (UID: \"aa609bb2-28bf-44e9-8c1c-3125dea60565\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:03:55.968322 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:55.968194    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa609bb2-28bf-44e9-8c1c-3125dea60565-maas-api-tls\") pod \"maas-api-e2e-ait-e65739ab-f74c8f957-5slqt\" (UID: \"aa609bb2-28bf-44e9-8c1c-3125dea60565\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:03:56.068622 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:56.068586    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjnk2\" (UniqueName: \"kubernetes.io/projected/aa609bb2-28bf-44e9-8c1c-3125dea60565-kube-api-access-rjnk2\") pod \"maas-api-e2e-ait-e65739ab-f74c8f957-5slqt\" (UID: \"aa609bb2-28bf-44e9-8c1c-3125dea60565\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:03:56.068835 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:56.068632    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa609bb2-28bf-44e9-8c1c-3125dea60565-maas-api-tls\") pod \"maas-api-e2e-ait-e65739ab-f74c8f957-5slqt\" (UID: \"aa609bb2-28bf-44e9-8c1c-3125dea60565\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:03:56.068835 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:56.068691    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa609bb2-28bf-44e9-8c1c-3125dea60565-openshift-service-ca\") pod \"maas-api-e2e-ait-e65739ab-f74c8f957-5slqt\" (UID: \"aa609bb2-28bf-44e9-8c1c-3125dea60565\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:03:56.069410 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:56.069387    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa609bb2-28bf-44e9-8c1c-3125dea60565-openshift-service-ca\") pod \"maas-api-e2e-ait-e65739ab-f74c8f957-5slqt\" (UID: \"aa609bb2-28bf-44e9-8c1c-3125dea60565\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:03:56.071376 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:56.071353    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa609bb2-28bf-44e9-8c1c-3125dea60565-maas-api-tls\") pod \"maas-api-e2e-ait-e65739ab-f74c8f957-5slqt\" (UID: \"aa609bb2-28bf-44e9-8c1c-3125dea60565\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:03:56.100283 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:56.100211    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjnk2\" (UniqueName: \"kubernetes.io/projected/aa609bb2-28bf-44e9-8c1c-3125dea60565-kube-api-access-rjnk2\") pod \"maas-api-e2e-ait-e65739ab-f74c8f957-5slqt\" (UID: \"aa609bb2-28bf-44e9-8c1c-3125dea60565\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:03:56.143209 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:56.143171    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:03:56.309175 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:56.309152    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"]
Jul 16 21:03:56.311511 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:03:56.311483    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa609bb2_28bf_44e9_8c1c_3125dea60565.slice/crio-cef5ebe63f701f2370faf12eeb4af1a1105cb31e4fc983988e80c2fdbc1fc6ba WatchSource:0}: Error finding container cef5ebe63f701f2370faf12eeb4af1a1105cb31e4fc983988e80c2fdbc1fc6ba: Status 404 returned error can't find the container with id cef5ebe63f701f2370faf12eeb4af1a1105cb31e4fc983988e80c2fdbc1fc6ba
Jul 16 21:03:56.351599 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:56.351509    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" event={"ID":"aa609bb2-28bf-44e9-8c1c-3125dea60565","Type":"ContainerStarted","Data":"cef5ebe63f701f2370faf12eeb4af1a1105cb31e4fc983988e80c2fdbc1fc6ba"}
Jul 16 21:03:57.357323 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:57.357280    2530 generic.go:356] "Generic (PLEG): container finished" podID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerID="395f3a57a2017ea38b30247a4aa49bcb8aa0c33210ef8c7fbb4015f5aaa7ce74" exitCode=1
Jul 16 21:03:57.357742 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:57.357371    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" event={"ID":"aa609bb2-28bf-44e9-8c1c-3125dea60565","Type":"ContainerDied","Data":"395f3a57a2017ea38b30247a4aa49bcb8aa0c33210ef8c7fbb4015f5aaa7ce74"}
Jul 16 21:03:57.357742 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:57.357576    2530 scope.go:117] "RemoveContainer" containerID="395f3a57a2017ea38b30247a4aa49bcb8aa0c33210ef8c7fbb4015f5aaa7ce74"
Jul 16 21:03:58.363621 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:58.363527    2530 generic.go:356] "Generic (PLEG): container finished" podID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerID="1e4a43bc105d7e2a88c1bb93a83c4be559ad9306b51566190fb66b0e18f7faa6" exitCode=1
Jul 16 21:03:58.364073 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:58.363610    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" event={"ID":"aa609bb2-28bf-44e9-8c1c-3125dea60565","Type":"ContainerDied","Data":"1e4a43bc105d7e2a88c1bb93a83c4be559ad9306b51566190fb66b0e18f7faa6"}
Jul 16 21:03:58.364073 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:58.363677    2530 scope.go:117] "RemoveContainer" containerID="395f3a57a2017ea38b30247a4aa49bcb8aa0c33210ef8c7fbb4015f5aaa7ce74"
Jul 16 21:03:58.364073 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:58.364020    2530 scope.go:117] "RemoveContainer" containerID="1e4a43bc105d7e2a88c1bb93a83c4be559ad9306b51566190fb66b0e18f7faa6"
Jul 16 21:03:58.364295 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:03:58.364274    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:03:59.369770 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:03:59.369740    2530 scope.go:117] "RemoveContainer" containerID="1e4a43bc105d7e2a88c1bb93a83c4be559ad9306b51566190fb66b0e18f7faa6"
Jul 16 21:03:59.370283 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:03:59.369987    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:04:00.640585 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:00.640547    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2"]
Jul 16 21:04:00.641025 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:00.640890    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" containerName="maas-api"
Jul 16 21:04:00.641025 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:00.640902    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" containerName="maas-api"
Jul 16 21:04:00.641025 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:00.640977    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="06539da2-47aa-4c8c-bb2a-1c665adcfa75" containerName="revoke-keys"
Jul 16 21:04:00.641025 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:00.640988    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e04ef8c-3f1b-4c0d-9b2a-234c5a76c1b4" containerName="maas-api"
Jul 16 21:04:00.645173 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:00.645148    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2"
Jul 16 21:04:00.648274 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:00.648251    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:04:00.651542 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:00.651511    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2"]
Jul 16 21:04:00.713336 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:00.713298    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec29ff9-d49c-4501-a06c-2e0db591f525-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-tskv2\" (UID: \"0ec29ff9-d49c-4501-a06c-2e0db591f525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2"
Jul 16 21:04:00.813808 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:00.813765    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec29ff9-d49c-4501-a06c-2e0db591f525-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-tskv2\" (UID: \"0ec29ff9-d49c-4501-a06c-2e0db591f525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2"
Jul 16 21:04:00.814466 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:00.814440    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec29ff9-d49c-4501-a06c-2e0db591f525-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-tskv2\" (UID: \"0ec29ff9-d49c-4501-a06c-2e0db591f525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2"
Jul 16 21:04:00.956693 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:00.956659    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2"
Jul 16 21:04:01.101797 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:01.101770    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2"]
Jul 16 21:04:01.104030 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:04:01.104002    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec29ff9_d49c_4501_a06c_2e0db591f525.slice/crio-46d0f04e93b4e173c7ead420406e3fc8b032cfcb49035d9b979fca391f2b2ff7 WatchSource:0}: Error finding container 46d0f04e93b4e173c7ead420406e3fc8b032cfcb49035d9b979fca391f2b2ff7: Status 404 returned error can't find the container with id 46d0f04e93b4e173c7ead420406e3fc8b032cfcb49035d9b979fca391f2b2ff7
Jul 16 21:04:01.143935 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:01.143906    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:04:01.144305 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:01.144292    2530 scope.go:117] "RemoveContainer" containerID="1e4a43bc105d7e2a88c1bb93a83c4be559ad9306b51566190fb66b0e18f7faa6"
Jul 16 21:04:01.144504 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:04:01.144486    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:04:01.379474 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:01.379442    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2" event={"ID":"0ec29ff9-d49c-4501-a06c-2e0db591f525","Type":"ContainerStarted","Data":"46d0f04e93b4e173c7ead420406e3fc8b032cfcb49035d9b979fca391f2b2ff7"}
Jul 16 21:04:02.385032 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:02.384992    2530 generic.go:356] "Generic (PLEG): container finished" podID="0ec29ff9-d49c-4501-a06c-2e0db591f525" containerID="455b15c631e7f13babbdbf3cb0f1a5d13d1bcc40fe5621e7cc9981b68872c97d" exitCode=7
Jul 16 21:04:02.385492 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:02.385075    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2" event={"ID":"0ec29ff9-d49c-4501-a06c-2e0db591f525","Type":"ContainerDied","Data":"455b15c631e7f13babbdbf3cb0f1a5d13d1bcc40fe5621e7cc9981b68872c97d"}
Jul 16 21:04:02.385492 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:02.385309    2530 scope.go:117] "RemoveContainer" containerID="455b15c631e7f13babbdbf3cb0f1a5d13d1bcc40fe5621e7cc9981b68872c97d"
Jul 16 21:04:03.392308 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:03.392223    2530 generic.go:356] "Generic (PLEG): container finished" podID="0ec29ff9-d49c-4501-a06c-2e0db591f525" containerID="a5c58809c92712c3a63b5bf782d3527b082f76634622e3f0bbc8e0887decb05b" exitCode=7
Jul 16 21:04:03.392719 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:03.392304    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2" event={"ID":"0ec29ff9-d49c-4501-a06c-2e0db591f525","Type":"ContainerDied","Data":"a5c58809c92712c3a63b5bf782d3527b082f76634622e3f0bbc8e0887decb05b"}
Jul 16 21:04:03.392719 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:03.392362    2530 scope.go:117] "RemoveContainer" containerID="455b15c631e7f13babbdbf3cb0f1a5d13d1bcc40fe5621e7cc9981b68872c97d"
Jul 16 21:04:03.392719 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:03.392564    2530 scope.go:117] "RemoveContainer" containerID="a5c58809c92712c3a63b5bf782d3527b082f76634622e3f0bbc8e0887decb05b"
Jul 16 21:04:03.392819 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:04:03.392793    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-tskv2_odh-ai-gateway-infra(0ec29ff9-d49c-4501-a06c-2e0db591f525)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2" podUID="0ec29ff9-d49c-4501-a06c-2e0db591f525"
Jul 16 21:04:04.398691 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:04.398658    2530 scope.go:117] "RemoveContainer" containerID="a5c58809c92712c3a63b5bf782d3527b082f76634622e3f0bbc8e0887decb05b"
Jul 16 21:04:04.399078 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:04:04.398846    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-tskv2_odh-ai-gateway-infra(0ec29ff9-d49c-4501-a06c-2e0db591f525)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2" podUID="0ec29ff9-d49c-4501-a06c-2e0db591f525"
Jul 16 21:04:06.143291 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:06.143251    2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:04:06.143783 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:06.143753    2530 scope.go:117] "RemoveContainer" containerID="1e4a43bc105d7e2a88c1bb93a83c4be559ad9306b51566190fb66b0e18f7faa6"
Jul 16 21:04:06.143986 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:04:06.143963    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:04:13.364015 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:13.363970    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb"]
Jul 16 21:04:13.367122 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:13.367082    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb"
Jul 16 21:04:13.399492 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:13.399455    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb"]
Jul 16 21:04:13.530700 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:13.530652    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ac6bd2-1122-41a6-ada8-f4e12b9a29e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb\" (UID: \"48ac6bd2-1122-41a6-ada8-f4e12b9a29e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb"
Jul 16 21:04:13.632127 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:13.632009    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ac6bd2-1122-41a6-ada8-f4e12b9a29e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb\" (UID: \"48ac6bd2-1122-41a6-ada8-f4e12b9a29e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb"
Jul 16 21:04:13.632714 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:13.632693    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ac6bd2-1122-41a6-ada8-f4e12b9a29e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb\" (UID: \"48ac6bd2-1122-41a6-ada8-f4e12b9a29e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb"
Jul 16 21:04:13.677585 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:13.677548    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb"
Jul 16 21:04:13.842455 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:13.842421    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb"]
Jul 16 21:04:13.846154 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:04:13.846128    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ac6bd2_1122_41a6_ada8_f4e12b9a29e0.slice/crio-7a7ecb99e4cde55426f702897c5064b4405f99c5e7144c65388d0c55c87a37cd WatchSource:0}: Error finding container 7a7ecb99e4cde55426f702897c5064b4405f99c5e7144c65388d0c55c87a37cd: Status 404 returned error can't find the container with id 7a7ecb99e4cde55426f702897c5064b4405f99c5e7144c65388d0c55c87a37cd
Jul 16 21:04:14.445829 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:14.445797    2530 generic.go:356] "Generic (PLEG): container finished" podID="48ac6bd2-1122-41a6-ada8-f4e12b9a29e0" containerID="cf7a992c47f4871341eb3a01c5f9e75a551a904c58b5acebbe62102a427763b1" exitCode=6
Jul 16 21:04:14.446191 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:14.445874    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb" event={"ID":"48ac6bd2-1122-41a6-ada8-f4e12b9a29e0","Type":"ContainerDied","Data":"cf7a992c47f4871341eb3a01c5f9e75a551a904c58b5acebbe62102a427763b1"}
Jul 16 21:04:14.446191 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:14.445931    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb" event={"ID":"48ac6bd2-1122-41a6-ada8-f4e12b9a29e0","Type":"ContainerStarted","Data":"7a7ecb99e4cde55426f702897c5064b4405f99c5e7144c65388d0c55c87a37cd"}
Jul 16 21:04:14.446191 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:14.446043    2530 scope.go:117] "RemoveContainer" containerID="cf7a992c47f4871341eb3a01c5f9e75a551a904c58b5acebbe62102a427763b1"
Jul 16 21:04:15.455657 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:15.455624    2530 generic.go:356] "Generic (PLEG): container finished" podID="48ac6bd2-1122-41a6-ada8-f4e12b9a29e0" containerID="cccde00caecbd03497d0a51a1a69a80c824fdf59c54512b5e23330e71b842742" exitCode=6
Jul 16 21:04:15.456137 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:15.455709    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb" event={"ID":"48ac6bd2-1122-41a6-ada8-f4e12b9a29e0","Type":"ContainerDied","Data":"cccde00caecbd03497d0a51a1a69a80c824fdf59c54512b5e23330e71b842742"}
Jul 16 21:04:15.456137 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:15.455769    2530 scope.go:117] "RemoveContainer" containerID="cf7a992c47f4871341eb3a01c5f9e75a551a904c58b5acebbe62102a427763b1"
Jul 16 21:04:15.456137 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:15.455951    2530 scope.go:117] "RemoveContainer" containerID="cccde00caecbd03497d0a51a1a69a80c824fdf59c54512b5e23330e71b842742"
Jul 16 21:04:15.456305 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:04:15.456218    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb_odh-ai-gateway-infra(48ac6bd2-1122-41a6-ada8-f4e12b9a29e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb" podUID="48ac6bd2-1122-41a6-ada8-f4e12b9a29e0"
Jul 16 21:04:16.461373 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:16.461332    2530 scope.go:117] "RemoveContainer" containerID="cccde00caecbd03497d0a51a1a69a80c824fdf59c54512b5e23330e71b842742"
Jul 16 21:04:16.461812 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:04:16.461575    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb_odh-ai-gateway-infra(48ac6bd2-1122-41a6-ada8-f4e12b9a29e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb" podUID="48ac6bd2-1122-41a6-ada8-f4e12b9a29e0"
Jul 16 21:04:18.507918 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:18.507886    2530 scope.go:117] "RemoveContainer" containerID="a5c58809c92712c3a63b5bf782d3527b082f76634622e3f0bbc8e0887decb05b"
Jul 16 21:04:18.510332 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:18.508129    2530 scope.go:117] "RemoveContainer" containerID="1e4a43bc105d7e2a88c1bb93a83c4be559ad9306b51566190fb66b0e18f7faa6"
Jul 16 21:04:19.476695 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:19.476599    2530 generic.go:356] "Generic (PLEG): container finished" podID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerID="073991ccaf3daa9a902738558233ddbd044fa8f66b75be27dafa9389b0e85c06" exitCode=1
Jul 16 21:04:19.476695 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:19.476681    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" event={"ID":"aa609bb2-28bf-44e9-8c1c-3125dea60565","Type":"ContainerDied","Data":"073991ccaf3daa9a902738558233ddbd044fa8f66b75be27dafa9389b0e85c06"}
Jul 16 21:04:19.476908 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:19.476748    2530 scope.go:117] "RemoveContainer" containerID="1e4a43bc105d7e2a88c1bb93a83c4be559ad9306b51566190fb66b0e18f7faa6"
Jul 16 21:04:19.477161 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:19.477139    2530 scope.go:117] "RemoveContainer" containerID="073991ccaf3daa9a902738558233ddbd044fa8f66b75be27dafa9389b0e85c06"
Jul 16 21:04:19.477392 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:04:19.477367    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:04:19.478683 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:19.478660    2530 generic.go:356] "Generic (PLEG): container finished" podID="0ec29ff9-d49c-4501-a06c-2e0db591f525" containerID="0e45709516bb6db132dc53108c605991e809431531d6357c216ec36da54af9c2" exitCode=7
Jul 16 21:04:19.478772 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:19.478707    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2" event={"ID":"0ec29ff9-d49c-4501-a06c-2e0db591f525","Type":"ContainerDied","Data":"0e45709516bb6db132dc53108c605991e809431531d6357c216ec36da54af9c2"}
Jul 16 21:04:19.478953 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:19.478941    2530 scope.go:117] "RemoveContainer" containerID="0e45709516bb6db132dc53108c605991e809431531d6357c216ec36da54af9c2"
Jul 16 21:04:19.479155 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:04:19.479140    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-tskv2_odh-ai-gateway-infra(0ec29ff9-d49c-4501-a06c-2e0db591f525)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2" podUID="0ec29ff9-d49c-4501-a06c-2e0db591f525"
Jul 16 21:04:19.489213 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:19.489134    2530 scope.go:117] "RemoveContainer" containerID="a5c58809c92712c3a63b5bf782d3527b082f76634622e3f0bbc8e0887decb05b"
Jul 16 21:04:20.574915 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:20.574878    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2"]
Jul 16 21:04:20.715084 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:20.715050    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2"
Jul 16 21:04:20.800988 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:20.800955    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec29ff9-d49c-4501-a06c-2e0db591f525-maas-api-service-ca\") pod \"0ec29ff9-d49c-4501-a06c-2e0db591f525\" (UID: \"0ec29ff9-d49c-4501-a06c-2e0db591f525\") "
Jul 16 21:04:20.801348 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:20.801323    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec29ff9-d49c-4501-a06c-2e0db591f525-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ec29ff9-d49c-4501-a06c-2e0db591f525" (UID: "0ec29ff9-d49c-4501-a06c-2e0db591f525"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:04:20.902004 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:20.901912    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec29ff9-d49c-4501-a06c-2e0db591f525-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:04:21.144151 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:21.144087    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:04:21.144511 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:21.144496    2530 scope.go:117] "RemoveContainer" containerID="073991ccaf3daa9a902738558233ddbd044fa8f66b75be27dafa9389b0e85c06"
Jul 16 21:04:21.144694 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:04:21.144674    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:04:21.493230 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:21.493196    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2"
Jul 16 21:04:21.493230 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:21.493207    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2" event={"ID":"0ec29ff9-d49c-4501-a06c-2e0db591f525","Type":"ContainerDied","Data":"46d0f04e93b4e173c7ead420406e3fc8b032cfcb49035d9b979fca391f2b2ff7"}
Jul 16 21:04:21.493471 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:21.493276    2530 scope.go:117] "RemoveContainer" containerID="0e45709516bb6db132dc53108c605991e809431531d6357c216ec36da54af9c2"
Jul 16 21:04:21.537322 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:21.537279    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2"]
Jul 16 21:04:21.544408 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:21.544379    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-tskv2"]
Jul 16 21:04:22.507308 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:22.507274    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ec29ff9-d49c-4501-a06c-2e0db591f525" path="/var/lib/kubelet/pods/0ec29ff9-d49c-4501-a06c-2e0db591f525/volumes"
Jul 16 21:04:26.143355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:26.143307    2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:04:26.143850 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:26.143735    2530 scope.go:117] "RemoveContainer" containerID="073991ccaf3daa9a902738558233ddbd044fa8f66b75be27dafa9389b0e85c06"
Jul 16 21:04:26.143911 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:04:26.143889    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:04:31.502977 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:31.502947    2530 scope.go:117] "RemoveContainer" containerID="cccde00caecbd03497d0a51a1a69a80c824fdf59c54512b5e23330e71b842742"
Jul 16 21:04:32.547219 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:32.547124    2530 generic.go:356] "Generic (PLEG): container finished" podID="48ac6bd2-1122-41a6-ada8-f4e12b9a29e0" containerID="d25db37733c1672f9dca0a6d44462dc266a1f5eddcd835e58ae3f0817dd7bfdd" exitCode=6
Jul 16 21:04:32.547219 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:32.547193    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb" event={"ID":"48ac6bd2-1122-41a6-ada8-f4e12b9a29e0","Type":"ContainerDied","Data":"d25db37733c1672f9dca0a6d44462dc266a1f5eddcd835e58ae3f0817dd7bfdd"}
Jul 16 21:04:32.547738 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:32.547271    2530 scope.go:117] "RemoveContainer" containerID="cccde00caecbd03497d0a51a1a69a80c824fdf59c54512b5e23330e71b842742"
Jul 16 21:04:32.547738 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:32.547617    2530 scope.go:117] "RemoveContainer" containerID="d25db37733c1672f9dca0a6d44462dc266a1f5eddcd835e58ae3f0817dd7bfdd"
Jul 16 21:04:32.547878 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:04:32.547857    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb_odh-ai-gateway-infra(48ac6bd2-1122-41a6-ada8-f4e12b9a29e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb" podUID="48ac6bd2-1122-41a6-ada8-f4e12b9a29e0"
Jul 16 21:04:33.573170 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:33.573126    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb"]
Jul 16 21:04:33.715034 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:33.715003    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb"
Jul 16 21:04:33.804086 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:33.804046    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ac6bd2-1122-41a6-ada8-f4e12b9a29e0-maas-api-service-ca\") pod \"48ac6bd2-1122-41a6-ada8-f4e12b9a29e0\" (UID: \"48ac6bd2-1122-41a6-ada8-f4e12b9a29e0\") "
Jul 16 21:04:33.804424 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:33.804399    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ac6bd2-1122-41a6-ada8-f4e12b9a29e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48ac6bd2-1122-41a6-ada8-f4e12b9a29e0" (UID: "48ac6bd2-1122-41a6-ada8-f4e12b9a29e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:04:33.905519 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:33.905434    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ac6bd2-1122-41a6-ada8-f4e12b9a29e0-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:04:34.558705 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:34.558674    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb"
Jul 16 21:04:34.558705 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:34.558677    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb" event={"ID":"48ac6bd2-1122-41a6-ada8-f4e12b9a29e0","Type":"ContainerDied","Data":"7a7ecb99e4cde55426f702897c5064b4405f99c5e7144c65388d0c55c87a37cd"}
Jul 16 21:04:34.558936 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:34.558749    2530 scope.go:117] "RemoveContainer" containerID="d25db37733c1672f9dca0a6d44462dc266a1f5eddcd835e58ae3f0817dd7bfdd"
Jul 16 21:04:34.578119 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:34.578069    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb"]
Jul 16 21:04:34.585265 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:34.585220    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-q2zcb"]
Jul 16 21:04:36.507504 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:36.507463    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48ac6bd2-1122-41a6-ada8-f4e12b9a29e0" path="/var/lib/kubelet/pods/48ac6bd2-1122-41a6-ada8-f4e12b9a29e0/volumes"
Jul 16 21:04:37.502125 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:37.502070    2530 scope.go:117] "RemoveContainer" containerID="073991ccaf3daa9a902738558233ddbd044fa8f66b75be27dafa9389b0e85c06"
Jul 16 21:04:37.502306 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:04:37.502292    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:04:50.502476 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:50.502437    2530 scope.go:117] "RemoveContainer" containerID="073991ccaf3daa9a902738558233ddbd044fa8f66b75be27dafa9389b0e85c06"
Jul 16 21:04:50.502919 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:50.502690    2530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:04:51.631838 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:51.631757    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" event={"ID":"aa609bb2-28bf-44e9-8c1c-3125dea60565","Type":"ContainerStarted","Data":"73a307004d4f4c78ea002275fc4edee7453a26a091206bfb6282c5d9bc9d4ab6"}
Jul 16 21:04:51.632273 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:51.631956    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:04:51.653843 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:51.653801    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podStartSLOduration=54.701406162 podStartE2EDuration="56.653786143s" totalImagesPullingTime="1.952379981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:03:55 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:04:51.653437558 +0000 UTC m=+1929.675025326" watchObservedRunningTime="2026-07-16 21:04:51.653786143 +0000 UTC m=+1929.675373916"
Jul 16 21:04:52.637799 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:52.637760    2530 generic.go:356] "Generic (PLEG): container finished" podID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerID="73a307004d4f4c78ea002275fc4edee7453a26a091206bfb6282c5d9bc9d4ab6" exitCode=1
Jul 16 21:04:52.638269 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:52.637843    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" event={"ID":"aa609bb2-28bf-44e9-8c1c-3125dea60565","Type":"ContainerDied","Data":"73a307004d4f4c78ea002275fc4edee7453a26a091206bfb6282c5d9bc9d4ab6"}
Jul 16 21:04:52.638269 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:52.637918    2530 scope.go:117] "RemoveContainer" containerID="073991ccaf3daa9a902738558233ddbd044fa8f66b75be27dafa9389b0e85c06"
Jul 16 21:04:52.638269 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:52.638176    2530 scope.go:117] "RemoveContainer" containerID="73a307004d4f4c78ea002275fc4edee7453a26a091206bfb6282c5d9bc9d4ab6"
Jul 16 21:04:52.638436 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:04:52.638372    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:04:53.643548 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:53.643513    2530 scope.go:117] "RemoveContainer" containerID="73a307004d4f4c78ea002275fc4edee7453a26a091206bfb6282c5d9bc9d4ab6"
Jul 16 21:04:53.644014 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:04:53.643722    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:04:56.143713 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:56.143664    2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:04:56.144154 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:04:56.144052    2530 scope.go:117] "RemoveContainer" containerID="73a307004d4f4c78ea002275fc4edee7453a26a091206bfb6282c5d9bc9d4ab6"
Jul 16 21:04:56.144260 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:04:56.144239    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:05:00.709766 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.709729    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j"]
Jul 16 21:05:00.710261 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.710068    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ac6bd2-1122-41a6-ada8-f4e12b9a29e0" containerName="revoke-keys"
Jul 16 21:05:00.710261 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.710079    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ac6bd2-1122-41a6-ada8-f4e12b9a29e0" containerName="revoke-keys"
Jul 16 21:05:00.710261 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.710087    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ac6bd2-1122-41a6-ada8-f4e12b9a29e0" containerName="revoke-keys"
Jul 16 21:05:00.710261 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.710093    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ac6bd2-1122-41a6-ada8-f4e12b9a29e0" containerName="revoke-keys"
Jul 16 21:05:00.710261 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.710112    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ec29ff9-d49c-4501-a06c-2e0db591f525" containerName="revoke-keys"
Jul 16 21:05:00.710261 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.710118    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ec29ff9-d49c-4501-a06c-2e0db591f525" containerName="revoke-keys"
Jul 16 21:05:00.710261 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.710124    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ec29ff9-d49c-4501-a06c-2e0db591f525" containerName="revoke-keys"
Jul 16 21:05:00.710261 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.710129    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ec29ff9-d49c-4501-a06c-2e0db591f525" containerName="revoke-keys"
Jul 16 21:05:00.710261 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.710196    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ac6bd2-1122-41a6-ada8-f4e12b9a29e0" containerName="revoke-keys"
Jul 16 21:05:00.710261 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.710206    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ec29ff9-d49c-4501-a06c-2e0db591f525" containerName="revoke-keys"
Jul 16 21:05:00.710261 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.710214    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ec29ff9-d49c-4501-a06c-2e0db591f525" containerName="revoke-keys"
Jul 16 21:05:00.710261 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.710220    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ec29ff9-d49c-4501-a06c-2e0db591f525" containerName="revoke-keys"
Jul 16 21:05:00.710261 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.710226    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ac6bd2-1122-41a6-ada8-f4e12b9a29e0" containerName="revoke-keys"
Jul 16 21:05:00.714817 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.714799    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j"
Jul 16 21:05:00.717644 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.717626    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:05:00.725811 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.723266    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j"]
Jul 16 21:05:00.833691 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.833648    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac54228-8035-4863-9fa8-a4276793653c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-csl9j\" (UID: \"dac54228-8035-4863-9fa8-a4276793653c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j"
Jul 16 21:05:00.934717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.934671    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac54228-8035-4863-9fa8-a4276793653c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-csl9j\" (UID: \"dac54228-8035-4863-9fa8-a4276793653c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j"
Jul 16 21:05:00.935443 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:00.935415    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac54228-8035-4863-9fa8-a4276793653c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-csl9j\" (UID: \"dac54228-8035-4863-9fa8-a4276793653c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j"
Jul 16 21:05:01.034681 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:01.034580    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j"
Jul 16 21:05:01.172920 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:01.172894    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j"]
Jul 16 21:05:01.175487 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:05:01.175453    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac54228_8035_4863_9fa8_a4276793653c.slice/crio-e2aba41677f4da337085d4eca8ab39179691c102cbbdd52aa25cadbcb3b7d876 WatchSource:0}: Error finding container e2aba41677f4da337085d4eca8ab39179691c102cbbdd52aa25cadbcb3b7d876: Status 404 returned error can't find the container with id e2aba41677f4da337085d4eca8ab39179691c102cbbdd52aa25cadbcb3b7d876
Jul 16 21:05:01.679762 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:01.679661    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j" event={"ID":"dac54228-8035-4863-9fa8-a4276793653c","Type":"ContainerStarted","Data":"bec4f5f37fdd0da10a581351dcac489e15792f6cb5ea81fce84fea459cba14d6"}
Jul 16 21:05:01.679762 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:01.679715    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j" event={"ID":"dac54228-8035-4863-9fa8-a4276793653c","Type":"ContainerStarted","Data":"e2aba41677f4da337085d4eca8ab39179691c102cbbdd52aa25cadbcb3b7d876"}
Jul 16 21:05:01.697997 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:01.697936    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j" podStartSLOduration=1.697916271 podStartE2EDuration="1.697916271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:05:01.696847735 +0000 UTC m=+1939.718435501" watchObservedRunningTime="2026-07-16 21:05:01.697916271 +0000 UTC m=+1939.719504041"
Jul 16 21:05:02.685233 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:02.685199    2530 generic.go:356] "Generic (PLEG): container finished" podID="dac54228-8035-4863-9fa8-a4276793653c" containerID="bec4f5f37fdd0da10a581351dcac489e15792f6cb5ea81fce84fea459cba14d6" exitCode=7
Jul 16 21:05:02.685655 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:02.685302    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j" event={"ID":"dac54228-8035-4863-9fa8-a4276793653c","Type":"ContainerDied","Data":"bec4f5f37fdd0da10a581351dcac489e15792f6cb5ea81fce84fea459cba14d6"}
Jul 16 21:05:02.685655 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:02.685488    2530 scope.go:117] "RemoveContainer" containerID="bec4f5f37fdd0da10a581351dcac489e15792f6cb5ea81fce84fea459cba14d6"
Jul 16 21:05:03.691183 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:03.691083    2530 generic.go:356] "Generic (PLEG): container finished" podID="dac54228-8035-4863-9fa8-a4276793653c" containerID="85ce32212da01cae79823144c2d4566c01475d8243a36fd27cd7b0809fffe2dc" exitCode=7
Jul 16 21:05:03.691183 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:03.691170    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j" event={"ID":"dac54228-8035-4863-9fa8-a4276793653c","Type":"ContainerDied","Data":"85ce32212da01cae79823144c2d4566c01475d8243a36fd27cd7b0809fffe2dc"}
Jul 16 21:05:03.691632 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:03.691234    2530 scope.go:117] "RemoveContainer" containerID="bec4f5f37fdd0da10a581351dcac489e15792f6cb5ea81fce84fea459cba14d6"
Jul 16 21:05:03.691632 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:03.691451    2530 scope.go:117] "RemoveContainer" containerID="85ce32212da01cae79823144c2d4566c01475d8243a36fd27cd7b0809fffe2dc"
Jul 16 21:05:03.691697 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:05:03.691664    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-csl9j_odh-ai-gateway-infra(dac54228-8035-4863-9fa8-a4276793653c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j" podUID="dac54228-8035-4863-9fa8-a4276793653c"
Jul 16 21:05:04.697116 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:04.697079    2530 scope.go:117] "RemoveContainer" containerID="85ce32212da01cae79823144c2d4566c01475d8243a36fd27cd7b0809fffe2dc"
Jul 16 21:05:04.697574 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:05:04.697344    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-csl9j_odh-ai-gateway-infra(dac54228-8035-4863-9fa8-a4276793653c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j" podUID="dac54228-8035-4863-9fa8-a4276793653c"
Jul 16 21:05:11.502464 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:11.502431    2530 scope.go:117] "RemoveContainer" containerID="73a307004d4f4c78ea002275fc4edee7453a26a091206bfb6282c5d9bc9d4ab6"
Jul 16 21:05:11.502892 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:05:11.502614    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:05:13.392153 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:13.392095    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c"]
Jul 16 21:05:13.392534 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:13.392458    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ac6bd2-1122-41a6-ada8-f4e12b9a29e0" containerName="revoke-keys"
Jul 16 21:05:13.392534 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:13.392470    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ac6bd2-1122-41a6-ada8-f4e12b9a29e0" containerName="revoke-keys"
Jul 16 21:05:13.392534 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:13.392479    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ec29ff9-d49c-4501-a06c-2e0db591f525" containerName="revoke-keys"
Jul 16 21:05:13.392534 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:13.392486    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ec29ff9-d49c-4501-a06c-2e0db591f525" containerName="revoke-keys"
Jul 16 21:05:13.392662 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:13.392538    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ac6bd2-1122-41a6-ada8-f4e12b9a29e0" containerName="revoke-keys"
Jul 16 21:05:13.396689 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:13.396667    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c"
Jul 16 21:05:13.403446 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:13.403262    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c"]
Jul 16 21:05:13.448951 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:13.448908    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d47b9c0-d78d-4acc-a059-8ebf9be432f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c\" (UID: \"6d47b9c0-d78d-4acc-a059-8ebf9be432f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c"
Jul 16 21:05:13.550269 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:13.550229    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d47b9c0-d78d-4acc-a059-8ebf9be432f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c\" (UID: \"6d47b9c0-d78d-4acc-a059-8ebf9be432f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c"
Jul 16 21:05:13.550878 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:13.550851    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d47b9c0-d78d-4acc-a059-8ebf9be432f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c\" (UID: \"6d47b9c0-d78d-4acc-a059-8ebf9be432f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c"
Jul 16 21:05:13.708637 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:13.708596    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c"
Jul 16 21:05:13.851517 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:13.851480    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c"]
Jul 16 21:05:13.853990 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:05:13.853960    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d47b9c0_d78d_4acc_a059_8ebf9be432f3.slice/crio-8116f720396fd9ace6d8389f958f2918fb77f13999605c3b838f5bc10ac0ef63 WatchSource:0}: Error finding container 8116f720396fd9ace6d8389f958f2918fb77f13999605c3b838f5bc10ac0ef63: Status 404 returned error can't find the container with id 8116f720396fd9ace6d8389f958f2918fb77f13999605c3b838f5bc10ac0ef63
Jul 16 21:05:14.744920 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:14.744881    2530 generic.go:356] "Generic (PLEG): container finished" podID="6d47b9c0-d78d-4acc-a059-8ebf9be432f3" containerID="8e13e56be2d039fbeb1ba1ed298b41b0eaa4310f2502ef54f0179ab50c193814" exitCode=6
Jul 16 21:05:14.745379 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:14.744941    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c" event={"ID":"6d47b9c0-d78d-4acc-a059-8ebf9be432f3","Type":"ContainerDied","Data":"8e13e56be2d039fbeb1ba1ed298b41b0eaa4310f2502ef54f0179ab50c193814"}
Jul 16 21:05:14.745379 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:14.744987    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c" event={"ID":"6d47b9c0-d78d-4acc-a059-8ebf9be432f3","Type":"ContainerStarted","Data":"8116f720396fd9ace6d8389f958f2918fb77f13999605c3b838f5bc10ac0ef63"}
Jul 16 21:05:14.745379 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:14.745212    2530 scope.go:117] "RemoveContainer" containerID="8e13e56be2d039fbeb1ba1ed298b41b0eaa4310f2502ef54f0179ab50c193814"
Jul 16 21:05:15.751512 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:15.751475    2530 generic.go:356] "Generic (PLEG): container finished" podID="6d47b9c0-d78d-4acc-a059-8ebf9be432f3" containerID="edd683ed33fbaf4bc7236be93fd774847bc44e47870f125b5007b1f0e261f648" exitCode=6
Jul 16 21:05:15.751923 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:15.751563    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c" event={"ID":"6d47b9c0-d78d-4acc-a059-8ebf9be432f3","Type":"ContainerDied","Data":"edd683ed33fbaf4bc7236be93fd774847bc44e47870f125b5007b1f0e261f648"}
Jul 16 21:05:15.751923 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:15.751630    2530 scope.go:117] "RemoveContainer" containerID="8e13e56be2d039fbeb1ba1ed298b41b0eaa4310f2502ef54f0179ab50c193814"
Jul 16 21:05:15.751923 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:15.751851    2530 scope.go:117] "RemoveContainer" containerID="edd683ed33fbaf4bc7236be93fd774847bc44e47870f125b5007b1f0e261f648"
Jul 16 21:05:15.752122 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:05:15.752078    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c_odh-ai-gateway-infra(6d47b9c0-d78d-4acc-a059-8ebf9be432f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c" podUID="6d47b9c0-d78d-4acc-a059-8ebf9be432f3"
Jul 16 21:05:16.757008 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:16.756983    2530 scope.go:117] "RemoveContainer" containerID="edd683ed33fbaf4bc7236be93fd774847bc44e47870f125b5007b1f0e261f648"
Jul 16 21:05:16.757424 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:05:16.757182    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c_odh-ai-gateway-infra(6d47b9c0-d78d-4acc-a059-8ebf9be432f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c" podUID="6d47b9c0-d78d-4acc-a059-8ebf9be432f3"
Jul 16 21:05:18.502927 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:18.502891    2530 scope.go:117] "RemoveContainer" containerID="85ce32212da01cae79823144c2d4566c01475d8243a36fd27cd7b0809fffe2dc"
Jul 16 21:05:19.771340 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:19.771294    2530 generic.go:356] "Generic (PLEG): container finished" podID="dac54228-8035-4863-9fa8-a4276793653c" containerID="07c2f6ca757e9cc357f7e1a09cf13e01ef09f3391b62d5213c80b0cb8b28bac9" exitCode=7
Jul 16 21:05:19.771781 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:19.771366    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j" event={"ID":"dac54228-8035-4863-9fa8-a4276793653c","Type":"ContainerDied","Data":"07c2f6ca757e9cc357f7e1a09cf13e01ef09f3391b62d5213c80b0cb8b28bac9"}
Jul 16 21:05:19.771781 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:19.771436    2530 scope.go:117] "RemoveContainer" containerID="85ce32212da01cae79823144c2d4566c01475d8243a36fd27cd7b0809fffe2dc"
Jul 16 21:05:19.771781 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:19.771739    2530 scope.go:117] "RemoveContainer" containerID="07c2f6ca757e9cc357f7e1a09cf13e01ef09f3391b62d5213c80b0cb8b28bac9"
Jul 16 21:05:19.771995 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:05:19.771974    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-csl9j_odh-ai-gateway-infra(dac54228-8035-4863-9fa8-a4276793653c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j" podUID="dac54228-8035-4863-9fa8-a4276793653c"
Jul 16 21:05:20.801288 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:20.801247    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j"]
Jul 16 21:05:20.940871 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:20.940836    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j"
Jul 16 21:05:21.009884 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:21.009848    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac54228-8035-4863-9fa8-a4276793653c-maas-api-service-ca\") pod \"dac54228-8035-4863-9fa8-a4276793653c\" (UID: \"dac54228-8035-4863-9fa8-a4276793653c\") "
Jul 16 21:05:21.010294 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:21.010268    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac54228-8035-4863-9fa8-a4276793653c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dac54228-8035-4863-9fa8-a4276793653c" (UID: "dac54228-8035-4863-9fa8-a4276793653c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:05:21.110941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:21.110846    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac54228-8035-4863-9fa8-a4276793653c-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:05:21.783389 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:21.783348    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j" event={"ID":"dac54228-8035-4863-9fa8-a4276793653c","Type":"ContainerDied","Data":"e2aba41677f4da337085d4eca8ab39179691c102cbbdd52aa25cadbcb3b7d876"}
Jul 16 21:05:21.783389 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:21.783389    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j"
Jul 16 21:05:21.783655 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:21.783420    2530 scope.go:117] "RemoveContainer" containerID="07c2f6ca757e9cc357f7e1a09cf13e01ef09f3391b62d5213c80b0cb8b28bac9"
Jul 16 21:05:21.804883 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:21.804846    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j"]
Jul 16 21:05:21.808795 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:21.808766    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-csl9j"]
Jul 16 21:05:22.504255 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:22.504223    2530 scope.go:117] "RemoveContainer" containerID="73a307004d4f4c78ea002275fc4edee7453a26a091206bfb6282c5d9bc9d4ab6"
Jul 16 21:05:22.504490 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:05:22.504450    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:05:22.507549 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:22.507523    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dac54228-8035-4863-9fa8-a4276793653c" path="/var/lib/kubelet/pods/dac54228-8035-4863-9fa8-a4276793653c/volumes"
Jul 16 21:05:29.502420 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:29.502378    2530 scope.go:117] "RemoveContainer" containerID="edd683ed33fbaf4bc7236be93fd774847bc44e47870f125b5007b1f0e261f648"
Jul 16 21:05:29.820902 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:29.820808    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c" event={"ID":"6d47b9c0-d78d-4acc-a059-8ebf9be432f3","Type":"ContainerStarted","Data":"4a49778b16c285989e049ba3dca2bd5024a94880f4cfafc123aa6abcd715f9cd"}
Jul 16 21:05:29.837264 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:29.837204    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c" podStartSLOduration=16.837186046 podStartE2EDuration="16.837186046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:05:29.835958686 +0000 UTC m=+1967.857546454" watchObservedRunningTime="2026-07-16 21:05:29.837186046 +0000 UTC m=+1967.858773813"
Jul 16 21:05:30.827160 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:30.827092    2530 generic.go:356] "Generic (PLEG): container finished" podID="6d47b9c0-d78d-4acc-a059-8ebf9be432f3" containerID="4a49778b16c285989e049ba3dca2bd5024a94880f4cfafc123aa6abcd715f9cd" exitCode=6
Jul 16 21:05:30.827673 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:30.827171    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c" event={"ID":"6d47b9c0-d78d-4acc-a059-8ebf9be432f3","Type":"ContainerDied","Data":"4a49778b16c285989e049ba3dca2bd5024a94880f4cfafc123aa6abcd715f9cd"}
Jul 16 21:05:30.827673 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:30.827235    2530 scope.go:117] "RemoveContainer" containerID="edd683ed33fbaf4bc7236be93fd774847bc44e47870f125b5007b1f0e261f648"
Jul 16 21:05:30.827673 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:30.827607    2530 scope.go:117] "RemoveContainer" containerID="4a49778b16c285989e049ba3dca2bd5024a94880f4cfafc123aa6abcd715f9cd"
Jul 16 21:05:30.827831 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:05:30.827810    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c_odh-ai-gateway-infra(6d47b9c0-d78d-4acc-a059-8ebf9be432f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c" podUID="6d47b9c0-d78d-4acc-a059-8ebf9be432f3"
Jul 16 21:05:30.853541 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:30.853506    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c"]
Jul 16 21:05:31.971929 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:31.971901    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c"
Jul 16 21:05:32.012502 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:32.012457    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d47b9c0-d78d-4acc-a059-8ebf9be432f3-maas-api-service-ca\") pod \"6d47b9c0-d78d-4acc-a059-8ebf9be432f3\" (UID: \"6d47b9c0-d78d-4acc-a059-8ebf9be432f3\") "
Jul 16 21:05:32.012840 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:32.012815    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d47b9c0-d78d-4acc-a059-8ebf9be432f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d47b9c0-d78d-4acc-a059-8ebf9be432f3" (UID: "6d47b9c0-d78d-4acc-a059-8ebf9be432f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:05:32.114293 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:32.114199    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d47b9c0-d78d-4acc-a059-8ebf9be432f3-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:05:32.839068 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:32.838985    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c"
Jul 16 21:05:32.839068 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:32.839000    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c" event={"ID":"6d47b9c0-d78d-4acc-a059-8ebf9be432f3","Type":"ContainerDied","Data":"8116f720396fd9ace6d8389f958f2918fb77f13999605c3b838f5bc10ac0ef63"}
Jul 16 21:05:32.839068 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:32.839058    2530 scope.go:117] "RemoveContainer" containerID="4a49778b16c285989e049ba3dca2bd5024a94880f4cfafc123aa6abcd715f9cd"
Jul 16 21:05:32.855443 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:32.855410    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c"]
Jul 16 21:05:32.863610 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:32.863580    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jqk9c"]
Jul 16 21:05:34.507989 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:34.507902    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d47b9c0-d78d-4acc-a059-8ebf9be432f3" path="/var/lib/kubelet/pods/6d47b9c0-d78d-4acc-a059-8ebf9be432f3/volumes"
Jul 16 21:05:36.501994 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:36.501951    2530 scope.go:117] "RemoveContainer" containerID="73a307004d4f4c78ea002275fc4edee7453a26a091206bfb6282c5d9bc9d4ab6"
Jul 16 21:05:37.865462 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:37.865434    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" event={"ID":"aa609bb2-28bf-44e9-8c1c-3125dea60565","Type":"ContainerStarted","Data":"e3cf844042ae232d8b719f318f08c2af965290ddd4263b9c12b06df961ded70f"}
Jul 16 21:05:37.865781 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:37.865756    2530 scope.go:117] "RemoveContainer" containerID="e3cf844042ae232d8b719f318f08c2af965290ddd4263b9c12b06df961ded70f"
Jul 16 21:05:37.865926 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:05:37.865909    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:05:38.872009 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:38.871974    2530 generic.go:356] "Generic (PLEG): container finished" podID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerID="e3cf844042ae232d8b719f318f08c2af965290ddd4263b9c12b06df961ded70f" exitCode=1
Jul 16 21:05:38.872503 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:38.872041    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" event={"ID":"aa609bb2-28bf-44e9-8c1c-3125dea60565","Type":"ContainerDied","Data":"e3cf844042ae232d8b719f318f08c2af965290ddd4263b9c12b06df961ded70f"}
Jul 16 21:05:38.872503 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:38.872124    2530 scope.go:117] "RemoveContainer" containerID="73a307004d4f4c78ea002275fc4edee7453a26a091206bfb6282c5d9bc9d4ab6"
Jul 16 21:05:38.872503 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:38.872486    2530 scope.go:117] "RemoveContainer" containerID="e3cf844042ae232d8b719f318f08c2af965290ddd4263b9c12b06df961ded70f"
Jul 16 21:05:38.872730 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:05:38.872709    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:05:41.632834 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:41.632770    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:05:41.633287 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:41.633254    2530 scope.go:117] "RemoveContainer" containerID="e3cf844042ae232d8b719f318f08c2af965290ddd4263b9c12b06df961ded70f"
Jul 16 21:05:41.633461 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:05:41.633441    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:05:46.143554 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:46.143513    2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:05:46.143963 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:46.143871    2530 scope.go:117] "RemoveContainer" containerID="e3cf844042ae232d8b719f318f08c2af965290ddd4263b9c12b06df961ded70f"
Jul 16 21:05:46.144046 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:05:46.144025    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:05:59.502332 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:05:59.502295    2530 scope.go:117] "RemoveContainer" containerID="e3cf844042ae232d8b719f318f08c2af965290ddd4263b9c12b06df961ded70f"
Jul 16 21:05:59.502733 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:05:59.502515    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:06:01.519006 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.518970    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz"]
Jul 16 21:06:01.519420 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.519308    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac54228-8035-4863-9fa8-a4276793653c" containerName="revoke-keys"
Jul 16 21:06:01.519420 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.519320    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac54228-8035-4863-9fa8-a4276793653c" containerName="revoke-keys"
Jul 16 21:06:01.519420 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.519340    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d47b9c0-d78d-4acc-a059-8ebf9be432f3" containerName="revoke-keys"
Jul 16 21:06:01.519420 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.519346    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d47b9c0-d78d-4acc-a059-8ebf9be432f3" containerName="revoke-keys"
Jul 16 21:06:01.519420 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.519356    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac54228-8035-4863-9fa8-a4276793653c" containerName="revoke-keys"
Jul 16 21:06:01.519420 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.519361    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac54228-8035-4863-9fa8-a4276793653c" containerName="revoke-keys"
Jul 16 21:06:01.519420 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.519369    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d47b9c0-d78d-4acc-a059-8ebf9be432f3" containerName="revoke-keys"
Jul 16 21:06:01.519420 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.519374    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d47b9c0-d78d-4acc-a059-8ebf9be432f3" containerName="revoke-keys"
Jul 16 21:06:01.519669 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.519427    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="dac54228-8035-4863-9fa8-a4276793653c" containerName="revoke-keys"
Jul 16 21:06:01.519669 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.519434    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d47b9c0-d78d-4acc-a059-8ebf9be432f3" containerName="revoke-keys"
Jul 16 21:06:01.519669 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.519441    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="dac54228-8035-4863-9fa8-a4276793653c" containerName="revoke-keys"
Jul 16 21:06:01.519669 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.519447    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d47b9c0-d78d-4acc-a059-8ebf9be432f3" containerName="revoke-keys"
Jul 16 21:06:01.522249 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.522228    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz"
Jul 16 21:06:01.525067 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.525046    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:06:01.533091 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.533063    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz"]
Jul 16 21:06:01.581151 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.581125    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb899c95-212c-47f5-887a-63a07865fa5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz\" (UID: \"bb899c95-212c-47f5-887a-63a07865fa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz"
Jul 16 21:06:01.682285 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.682251    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb899c95-212c-47f5-887a-63a07865fa5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz\" (UID: \"bb899c95-212c-47f5-887a-63a07865fa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz"
Jul 16 21:06:01.682853 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.682833    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb899c95-212c-47f5-887a-63a07865fa5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz\" (UID: \"bb899c95-212c-47f5-887a-63a07865fa5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz"
Jul 16 21:06:01.833608 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.833536    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz"
Jul 16 21:06:01.959778 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.959745    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz"]
Jul 16 21:06:01.962817 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:06:01.962791    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb899c95_212c_47f5_887a_63a07865fa5e.slice/crio-5d140cdc5928f5adab1ed7c8fbdc39703cd5dd903ecd377c59e47e7d4eba2720 WatchSource:0}: Error finding container 5d140cdc5928f5adab1ed7c8fbdc39703cd5dd903ecd377c59e47e7d4eba2720: Status 404 returned error can't find the container with id 5d140cdc5928f5adab1ed7c8fbdc39703cd5dd903ecd377c59e47e7d4eba2720
Jul 16 21:06:01.973918 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:01.973893    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz" event={"ID":"bb899c95-212c-47f5-887a-63a07865fa5e","Type":"ContainerStarted","Data":"5d140cdc5928f5adab1ed7c8fbdc39703cd5dd903ecd377c59e47e7d4eba2720"}
Jul 16 21:06:02.978738 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:02.978705    2530 generic.go:356] "Generic (PLEG): container finished" podID="bb899c95-212c-47f5-887a-63a07865fa5e" containerID="85a47e8550453b513c0290af1eb3f26ba4af81129f8ca73bee34ad079060cfb7" exitCode=6
Jul 16 21:06:02.979187 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:02.978763    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz" event={"ID":"bb899c95-212c-47f5-887a-63a07865fa5e","Type":"ContainerDied","Data":"85a47e8550453b513c0290af1eb3f26ba4af81129f8ca73bee34ad079060cfb7"}
Jul 16 21:06:02.979187 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:02.978980    2530 scope.go:117] "RemoveContainer" containerID="85a47e8550453b513c0290af1eb3f26ba4af81129f8ca73bee34ad079060cfb7"
Jul 16 21:06:03.427466 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.427442    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg"]
Jul 16 21:06:03.427794 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.427781    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac54228-8035-4863-9fa8-a4276793653c" containerName="revoke-keys"
Jul 16 21:06:03.427866 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.427796    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac54228-8035-4863-9fa8-a4276793653c" containerName="revoke-keys"
Jul 16 21:06:03.427866 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.427812    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d47b9c0-d78d-4acc-a059-8ebf9be432f3" containerName="revoke-keys"
Jul 16 21:06:03.427866 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.427819    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d47b9c0-d78d-4acc-a059-8ebf9be432f3" containerName="revoke-keys"
Jul 16 21:06:03.428011 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.427873    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="dac54228-8035-4863-9fa8-a4276793653c" containerName="revoke-keys"
Jul 16 21:06:03.428011 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.427886    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d47b9c0-d78d-4acc-a059-8ebf9be432f3" containerName="revoke-keys"
Jul 16 21:06:03.430757 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.430736    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg"
Jul 16 21:06:03.438272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.438249    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg"]
Jul 16 21:06:03.497641 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.497613    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94568fca-ecd2-4502-96fc-76f7ea6609f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg\" (UID: \"94568fca-ecd2-4502-96fc-76f7ea6609f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg"
Jul 16 21:06:03.598798 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.598769    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94568fca-ecd2-4502-96fc-76f7ea6609f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg\" (UID: \"94568fca-ecd2-4502-96fc-76f7ea6609f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg"
Jul 16 21:06:03.599331 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.599316    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94568fca-ecd2-4502-96fc-76f7ea6609f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg\" (UID: \"94568fca-ecd2-4502-96fc-76f7ea6609f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg"
Jul 16 21:06:03.742503 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.742479    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg"
Jul 16 21:06:03.873881 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.870219    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg"]
Jul 16 21:06:03.878502 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:06:03.878458    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94568fca_ecd2_4502_96fc_76f7ea6609f2.slice/crio-d1b8ca2ae60cbe432d099965946777f39691f5f9683a09bc58713464f730f744 WatchSource:0}: Error finding container d1b8ca2ae60cbe432d099965946777f39691f5f9683a09bc58713464f730f744: Status 404 returned error can't find the container with id d1b8ca2ae60cbe432d099965946777f39691f5f9683a09bc58713464f730f744
Jul 16 21:06:03.984288 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.984256    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg" event={"ID":"94568fca-ecd2-4502-96fc-76f7ea6609f2","Type":"ContainerStarted","Data":"d1b8ca2ae60cbe432d099965946777f39691f5f9683a09bc58713464f730f744"}
Jul 16 21:06:03.985968 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.985942    2530 generic.go:356] "Generic (PLEG): container finished" podID="bb899c95-212c-47f5-887a-63a07865fa5e" containerID="20652cad7e497c9de2c4e98b0ac2974c3933512333469d9fbdf39bd8007200dc" exitCode=6
Jul 16 21:06:03.986056 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.986017    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz" event={"ID":"bb899c95-212c-47f5-887a-63a07865fa5e","Type":"ContainerDied","Data":"20652cad7e497c9de2c4e98b0ac2974c3933512333469d9fbdf39bd8007200dc"}
Jul 16 21:06:03.986139 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.986070    2530 scope.go:117] "RemoveContainer" containerID="85a47e8550453b513c0290af1eb3f26ba4af81129f8ca73bee34ad079060cfb7"
Jul 16 21:06:03.986317 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:03.986302    2530 scope.go:117] "RemoveContainer" containerID="20652cad7e497c9de2c4e98b0ac2974c3933512333469d9fbdf39bd8007200dc"
Jul 16 21:06:03.986507 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:06:03.986489    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz_odh-ai-gateway-infra(bb899c95-212c-47f5-887a-63a07865fa5e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz" podUID="bb899c95-212c-47f5-887a-63a07865fa5e"
Jul 16 21:06:04.992198 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:04.992170    2530 scope.go:117] "RemoveContainer" containerID="20652cad7e497c9de2c4e98b0ac2974c3933512333469d9fbdf39bd8007200dc"
Jul 16 21:06:04.992603 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:06:04.992367    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz_odh-ai-gateway-infra(bb899c95-212c-47f5-887a-63a07865fa5e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz" podUID="bb899c95-212c-47f5-887a-63a07865fa5e"
Jul 16 21:06:04.993383 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:04.993358    2530 generic.go:356] "Generic (PLEG): container finished" podID="94568fca-ecd2-4502-96fc-76f7ea6609f2" containerID="94da372579ae7bad4000329fadfee22203b8133d93832d77e8f72dc02d3a6cc0" exitCode=6
Jul 16 21:06:04.993480 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:04.993431    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg" event={"ID":"94568fca-ecd2-4502-96fc-76f7ea6609f2","Type":"ContainerDied","Data":"94da372579ae7bad4000329fadfee22203b8133d93832d77e8f72dc02d3a6cc0"}
Jul 16 21:06:04.993588 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:04.993574    2530 scope.go:117] "RemoveContainer" containerID="94da372579ae7bad4000329fadfee22203b8133d93832d77e8f72dc02d3a6cc0"
Jul 16 21:06:05.999258 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:05.999220    2530 generic.go:356] "Generic (PLEG): container finished" podID="94568fca-ecd2-4502-96fc-76f7ea6609f2" containerID="d3c6c2a320a2998904a403ddeccb304ee846a8e1bb495b52e27e7a0f50464c1b" exitCode=6
Jul 16 21:06:05.999647 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:05.999266    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg" event={"ID":"94568fca-ecd2-4502-96fc-76f7ea6609f2","Type":"ContainerDied","Data":"d3c6c2a320a2998904a403ddeccb304ee846a8e1bb495b52e27e7a0f50464c1b"}
Jul 16 21:06:05.999647 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:05.999321    2530 scope.go:117] "RemoveContainer" containerID="94da372579ae7bad4000329fadfee22203b8133d93832d77e8f72dc02d3a6cc0"
Jul 16 21:06:05.999647 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:05.999478    2530 scope.go:117] "RemoveContainer" containerID="d3c6c2a320a2998904a403ddeccb304ee846a8e1bb495b52e27e7a0f50464c1b"
Jul 16 21:06:05.999775 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:06:05.999696    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg_odh-ai-gateway-infra(94568fca-ecd2-4502-96fc-76f7ea6609f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg" podUID="94568fca-ecd2-4502-96fc-76f7ea6609f2"
Jul 16 21:06:07.004792 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:07.004763    2530 scope.go:117] "RemoveContainer" containerID="d3c6c2a320a2998904a403ddeccb304ee846a8e1bb495b52e27e7a0f50464c1b"
Jul 16 21:06:07.005215 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:06:07.004952    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg_odh-ai-gateway-infra(94568fca-ecd2-4502-96fc-76f7ea6609f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg" podUID="94568fca-ecd2-4502-96fc-76f7ea6609f2"
Jul 16 21:06:13.502802 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:13.502767    2530 scope.go:117] "RemoveContainer" containerID="e3cf844042ae232d8b719f318f08c2af965290ddd4263b9c12b06df961ded70f"
Jul 16 21:06:13.503229 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:06:13.502932    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:06:18.502143 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:18.502096    2530 scope.go:117] "RemoveContainer" containerID="d3c6c2a320a2998904a403ddeccb304ee846a8e1bb495b52e27e7a0f50464c1b"
Jul 16 21:06:19.062176 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:19.062146    2530 generic.go:356] "Generic (PLEG): container finished" podID="94568fca-ecd2-4502-96fc-76f7ea6609f2" containerID="66b4fc735a6dccdb6e3f2f032265f4bab7dcc386268b1122556dd664969d44a7" exitCode=6
Jul 16 21:06:19.062278 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:19.062194    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg" event={"ID":"94568fca-ecd2-4502-96fc-76f7ea6609f2","Type":"ContainerDied","Data":"66b4fc735a6dccdb6e3f2f032265f4bab7dcc386268b1122556dd664969d44a7"}
Jul 16 21:06:19.062278 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:19.062240    2530 scope.go:117] "RemoveContainer" containerID="d3c6c2a320a2998904a403ddeccb304ee846a8e1bb495b52e27e7a0f50464c1b"
Jul 16 21:06:19.062597 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:19.062579    2530 scope.go:117] "RemoveContainer" containerID="66b4fc735a6dccdb6e3f2f032265f4bab7dcc386268b1122556dd664969d44a7"
Jul 16 21:06:19.062787 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:06:19.062753    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg_odh-ai-gateway-infra(94568fca-ecd2-4502-96fc-76f7ea6609f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg" podUID="94568fca-ecd2-4502-96fc-76f7ea6609f2"
Jul 16 21:06:19.502526 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:19.502500    2530 scope.go:117] "RemoveContainer" containerID="20652cad7e497c9de2c4e98b0ac2974c3933512333469d9fbdf39bd8007200dc"
Jul 16 21:06:20.073029 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:20.072994    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz" event={"ID":"bb899c95-212c-47f5-887a-63a07865fa5e","Type":"ContainerStarted","Data":"ca3be03df4a97d11913aaeb3c01393850c62c03e23a0979cc16a74afcbff5236"}
Jul 16 21:06:20.089260 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:20.089235    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg"]
Jul 16 21:06:20.091521 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:20.091483    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz" podStartSLOduration=19.091470493 podStartE2EDuration="19.091470493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:06:20.090367961 +0000 UTC m=+2018.111955760" watchObservedRunningTime="2026-07-16 21:06:20.091470493 +0000 UTC m=+2018.113058309"
Jul 16 21:06:20.222670 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:20.222646    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg"
Jul 16 21:06:20.329841 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:20.329748    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94568fca-ecd2-4502-96fc-76f7ea6609f2-maas-api-service-ca\") pod \"94568fca-ecd2-4502-96fc-76f7ea6609f2\" (UID: \"94568fca-ecd2-4502-96fc-76f7ea6609f2\") "
Jul 16 21:06:20.330056 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:20.330032    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94568fca-ecd2-4502-96fc-76f7ea6609f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94568fca-ecd2-4502-96fc-76f7ea6609f2" (UID: "94568fca-ecd2-4502-96fc-76f7ea6609f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:06:20.430438 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:20.430410    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94568fca-ecd2-4502-96fc-76f7ea6609f2-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:06:21.079045 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:21.079012    2530 generic.go:356] "Generic (PLEG): container finished" podID="bb899c95-212c-47f5-887a-63a07865fa5e" containerID="ca3be03df4a97d11913aaeb3c01393850c62c03e23a0979cc16a74afcbff5236" exitCode=6
Jul 16 21:06:21.079545 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:21.079086    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz" event={"ID":"bb899c95-212c-47f5-887a-63a07865fa5e","Type":"ContainerDied","Data":"ca3be03df4a97d11913aaeb3c01393850c62c03e23a0979cc16a74afcbff5236"}
Jul 16 21:06:21.079545 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:21.079181    2530 scope.go:117] "RemoveContainer" containerID="20652cad7e497c9de2c4e98b0ac2974c3933512333469d9fbdf39bd8007200dc"
Jul 16 21:06:21.079545 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:21.079462    2530 scope.go:117] "RemoveContainer" containerID="ca3be03df4a97d11913aaeb3c01393850c62c03e23a0979cc16a74afcbff5236"
Jul 16 21:06:21.079726 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:06:21.079700    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz_odh-ai-gateway-infra(bb899c95-212c-47f5-887a-63a07865fa5e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz" podUID="bb899c95-212c-47f5-887a-63a07865fa5e"
Jul 16 21:06:21.080552 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:21.080537    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg"
Jul 16 21:06:21.080627 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:21.080538    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg" event={"ID":"94568fca-ecd2-4502-96fc-76f7ea6609f2","Type":"ContainerDied","Data":"d1b8ca2ae60cbe432d099965946777f39691f5f9683a09bc58713464f730f744"}
Jul 16 21:06:21.089803 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:21.089789    2530 scope.go:117] "RemoveContainer" containerID="66b4fc735a6dccdb6e3f2f032265f4bab7dcc386268b1122556dd664969d44a7"
Jul 16 21:06:21.102887 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:21.102864    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz"]
Jul 16 21:06:21.111187 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:21.111166    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg"]
Jul 16 21:06:21.113209 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:21.113187    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-n92pg"]
Jul 16 21:06:22.216632 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:22.216607    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz"
Jul 16 21:06:22.347265 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:22.347189    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb899c95-212c-47f5-887a-63a07865fa5e-maas-api-service-ca\") pod \"bb899c95-212c-47f5-887a-63a07865fa5e\" (UID: \"bb899c95-212c-47f5-887a-63a07865fa5e\") "
Jul 16 21:06:22.347504 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:22.347482    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb899c95-212c-47f5-887a-63a07865fa5e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bb899c95-212c-47f5-887a-63a07865fa5e" (UID: "bb899c95-212c-47f5-887a-63a07865fa5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:06:22.448416 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:22.448388    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb899c95-212c-47f5-887a-63a07865fa5e-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:06:22.507168 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:22.507138    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94568fca-ecd2-4502-96fc-76f7ea6609f2" path="/var/lib/kubelet/pods/94568fca-ecd2-4502-96fc-76f7ea6609f2/volumes"
Jul 16 21:06:23.092585 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:23.092547    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz" event={"ID":"bb899c95-212c-47f5-887a-63a07865fa5e","Type":"ContainerDied","Data":"5d140cdc5928f5adab1ed7c8fbdc39703cd5dd903ecd377c59e47e7d4eba2720"}
Jul 16 21:06:23.092585 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:23.092570    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz"
Jul 16 21:06:23.092848 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:23.092620    2530 scope.go:117] "RemoveContainer" containerID="ca3be03df4a97d11913aaeb3c01393850c62c03e23a0979cc16a74afcbff5236"
Jul 16 21:06:23.110254 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:23.110232    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz"]
Jul 16 21:06:23.116278 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:23.116258    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dnvfz"]
Jul 16 21:06:24.507540 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:24.507504    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb899c95-212c-47f5-887a-63a07865fa5e" path="/var/lib/kubelet/pods/bb899c95-212c-47f5-887a-63a07865fa5e/volumes"
Jul 16 21:06:27.501973 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:27.501944    2530 scope.go:117] "RemoveContainer" containerID="e3cf844042ae232d8b719f318f08c2af965290ddd4263b9c12b06df961ded70f"
Jul 16 21:06:27.502346 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:06:27.502141    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:06:38.501987 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:38.501956    2530 scope.go:117] "RemoveContainer" containerID="e3cf844042ae232d8b719f318f08c2af965290ddd4263b9c12b06df961ded70f"
Jul 16 21:06:38.502619 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:06:38.502140    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:06:49.502005 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:49.501975    2530 scope.go:117] "RemoveContainer" containerID="e3cf844042ae232d8b719f318f08c2af965290ddd4263b9c12b06df961ded70f"
Jul 16 21:06:49.502391 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:06:49.502170    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:06:53.474743 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.474707    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt"]
Jul 16 21:06:53.475210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.475037    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb899c95-212c-47f5-887a-63a07865fa5e" containerName="revoke-keys"
Jul 16 21:06:53.475210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.475048    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb899c95-212c-47f5-887a-63a07865fa5e" containerName="revoke-keys"
Jul 16 21:06:53.475210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.475072    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb899c95-212c-47f5-887a-63a07865fa5e" containerName="revoke-keys"
Jul 16 21:06:53.475210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.475078    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb899c95-212c-47f5-887a-63a07865fa5e" containerName="revoke-keys"
Jul 16 21:06:53.475210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.475085    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94568fca-ecd2-4502-96fc-76f7ea6609f2" containerName="revoke-keys"
Jul 16 21:06:53.475210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.475091    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="94568fca-ecd2-4502-96fc-76f7ea6609f2" containerName="revoke-keys"
Jul 16 21:06:53.475210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.475116    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94568fca-ecd2-4502-96fc-76f7ea6609f2" containerName="revoke-keys"
Jul 16 21:06:53.475210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.475122    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="94568fca-ecd2-4502-96fc-76f7ea6609f2" containerName="revoke-keys"
Jul 16 21:06:53.475210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.475130    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94568fca-ecd2-4502-96fc-76f7ea6609f2" containerName="revoke-keys"
Jul 16 21:06:53.475210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.475135    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="94568fca-ecd2-4502-96fc-76f7ea6609f2" containerName="revoke-keys"
Jul 16 21:06:53.475210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.475184    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="94568fca-ecd2-4502-96fc-76f7ea6609f2" containerName="revoke-keys"
Jul 16 21:06:53.475210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.475192    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb899c95-212c-47f5-887a-63a07865fa5e" containerName="revoke-keys"
Jul 16 21:06:53.475210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.475197    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="94568fca-ecd2-4502-96fc-76f7ea6609f2" containerName="revoke-keys"
Jul 16 21:06:53.475210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.475204    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb899c95-212c-47f5-887a-63a07865fa5e" containerName="revoke-keys"
Jul 16 21:06:53.475210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.475211    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb899c95-212c-47f5-887a-63a07865fa5e" containerName="revoke-keys"
Jul 16 21:06:53.477905 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.477886    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt"
Jul 16 21:06:53.480398 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.480379    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:06:53.485959 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.485928    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt"]
Jul 16 21:06:53.504017 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.503980    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df048b2-21c9-4a81-8bcc-203fcb1054bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt\" (UID: \"9df048b2-21c9-4a81-8bcc-203fcb1054bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt"
Jul 16 21:06:53.604491 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.604454    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df048b2-21c9-4a81-8bcc-203fcb1054bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt\" (UID: \"9df048b2-21c9-4a81-8bcc-203fcb1054bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt"
Jul 16 21:06:53.605043 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.605025    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df048b2-21c9-4a81-8bcc-203fcb1054bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt\" (UID: \"9df048b2-21c9-4a81-8bcc-203fcb1054bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt"
Jul 16 21:06:53.788764 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.788695    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt"
Jul 16 21:06:53.915341 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:53.915318    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt"]
Jul 16 21:06:53.917535 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:06:53.917508    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df048b2_21c9_4a81_8bcc_203fcb1054bd.slice/crio-f6a32838a7460799833aa6fd7ce0421df99b5d3bcf88b17e67cb7b0dd2c8b2bc WatchSource:0}: Error finding container f6a32838a7460799833aa6fd7ce0421df99b5d3bcf88b17e67cb7b0dd2c8b2bc: Status 404 returned error can't find the container with id f6a32838a7460799833aa6fd7ce0421df99b5d3bcf88b17e67cb7b0dd2c8b2bc
Jul 16 21:06:54.228783 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:54.228756    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt" event={"ID":"9df048b2-21c9-4a81-8bcc-203fcb1054bd","Type":"ContainerStarted","Data":"ffe3aa866d449d6fc78d4c05d88571cd932b0d660c10d4d68f0a3ad3727dcbc9"}
Jul 16 21:06:54.228886 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:54.228804    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt" event={"ID":"9df048b2-21c9-4a81-8bcc-203fcb1054bd","Type":"ContainerStarted","Data":"f6a32838a7460799833aa6fd7ce0421df99b5d3bcf88b17e67cb7b0dd2c8b2bc"}
Jul 16 21:06:54.247942 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:54.247897    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt" podStartSLOduration=1.2478822840000001 podStartE2EDuration="1.247882284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:06:54.245889077 +0000 UTC m=+2052.267476844" watchObservedRunningTime="2026-07-16 21:06:54.247882284 +0000 UTC m=+2052.269470049"
Jul 16 21:06:55.235178 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:55.235148    2530 generic.go:356] "Generic (PLEG): container finished" podID="9df048b2-21c9-4a81-8bcc-203fcb1054bd" containerID="ffe3aa866d449d6fc78d4c05d88571cd932b0d660c10d4d68f0a3ad3727dcbc9" exitCode=6
Jul 16 21:06:55.235559 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:55.235214    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt" event={"ID":"9df048b2-21c9-4a81-8bcc-203fcb1054bd","Type":"ContainerDied","Data":"ffe3aa866d449d6fc78d4c05d88571cd932b0d660c10d4d68f0a3ad3727dcbc9"}
Jul 16 21:06:55.235559 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:55.235351    2530 scope.go:117] "RemoveContainer" containerID="ffe3aa866d449d6fc78d4c05d88571cd932b0d660c10d4d68f0a3ad3727dcbc9"
Jul 16 21:06:56.242201 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:56.242168    2530 generic.go:356] "Generic (PLEG): container finished" podID="9df048b2-21c9-4a81-8bcc-203fcb1054bd" containerID="11f346de163e5f86e4f753665a699e4a86945c618325f2afa70b6ed4f8b95376" exitCode=6
Jul 16 21:06:56.242612 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:56.242260    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt" event={"ID":"9df048b2-21c9-4a81-8bcc-203fcb1054bd","Type":"ContainerDied","Data":"11f346de163e5f86e4f753665a699e4a86945c618325f2afa70b6ed4f8b95376"}
Jul 16 21:06:56.242612 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:56.242318    2530 scope.go:117] "RemoveContainer" containerID="ffe3aa866d449d6fc78d4c05d88571cd932b0d660c10d4d68f0a3ad3727dcbc9"
Jul 16 21:06:56.242612 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:56.242415    2530 scope.go:117] "RemoveContainer" containerID="11f346de163e5f86e4f753665a699e4a86945c618325f2afa70b6ed4f8b95376"
Jul 16 21:06:56.242757 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:06:56.242632    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt_odh-ai-gateway-infra(9df048b2-21c9-4a81-8bcc-203fcb1054bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt" podUID="9df048b2-21c9-4a81-8bcc-203fcb1054bd"
Jul 16 21:06:57.248349 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:06:57.248324    2530 scope.go:117] "RemoveContainer" containerID="11f346de163e5f86e4f753665a699e4a86945c618325f2afa70b6ed4f8b95376"
Jul 16 21:06:57.248711 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:06:57.248497    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt_odh-ai-gateway-infra(9df048b2-21c9-4a81-8bcc-203fcb1054bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt" podUID="9df048b2-21c9-4a81-8bcc-203fcb1054bd"
Jul 16 21:07:01.565390 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:01.565356    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj"]
Jul 16 21:07:01.565769 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:01.565695    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb899c95-212c-47f5-887a-63a07865fa5e" containerName="revoke-keys"
Jul 16 21:07:01.565769 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:01.565705    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb899c95-212c-47f5-887a-63a07865fa5e" containerName="revoke-keys"
Jul 16 21:07:01.565769 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:01.565760    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="94568fca-ecd2-4502-96fc-76f7ea6609f2" containerName="revoke-keys"
Jul 16 21:07:01.569912 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:01.569891    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj"
Jul 16 21:07:01.577482 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:01.577454    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj"]
Jul 16 21:07:01.663825 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:01.663797    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54c450a-b1ad-4ca0-b763-e2d1416a33a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj\" (UID: \"c54c450a-b1ad-4ca0-b763-e2d1416a33a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj"
Jul 16 21:07:01.764609 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:01.764572    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54c450a-b1ad-4ca0-b763-e2d1416a33a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj\" (UID: \"c54c450a-b1ad-4ca0-b763-e2d1416a33a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj"
Jul 16 21:07:01.765169 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:01.765151    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54c450a-b1ad-4ca0-b763-e2d1416a33a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj\" (UID: \"c54c450a-b1ad-4ca0-b763-e2d1416a33a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj"
Jul 16 21:07:01.881280 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:01.881226    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj"
Jul 16 21:07:02.007681 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:02.007610    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj"]
Jul 16 21:07:02.011149 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:07:02.011121    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54c450a_b1ad_4ca0_b763_e2d1416a33a2.slice/crio-a61fbf3acc0c35d1f7c96b97751c5b8c292a19d028472c25c136b307c44b8c70 WatchSource:0}: Error finding container a61fbf3acc0c35d1f7c96b97751c5b8c292a19d028472c25c136b307c44b8c70: Status 404 returned error can't find the container with id a61fbf3acc0c35d1f7c96b97751c5b8c292a19d028472c25c136b307c44b8c70
Jul 16 21:07:02.271442 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:02.271418    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj" event={"ID":"c54c450a-b1ad-4ca0-b763-e2d1416a33a2","Type":"ContainerStarted","Data":"3cb4db802b60bf616b5fcb720d30278ba931767884cfdb78320752f2816b1962"}
Jul 16 21:07:02.271560 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:02.271474    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj" event={"ID":"c54c450a-b1ad-4ca0-b763-e2d1416a33a2","Type":"ContainerStarted","Data":"a61fbf3acc0c35d1f7c96b97751c5b8c292a19d028472c25c136b307c44b8c70"}
Jul 16 21:07:02.288486 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:02.288440    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj" podStartSLOduration=1.288423057 podStartE2EDuration="1.288423057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:07:02.286767648 +0000 UTC m=+2060.308355414" watchObservedRunningTime="2026-07-16 21:07:02.288423057 +0000 UTC m=+2060.310010824"
Jul 16 21:07:02.503888 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:02.503836    2530 scope.go:117] "RemoveContainer" containerID="e3cf844042ae232d8b719f318f08c2af965290ddd4263b9c12b06df961ded70f"
Jul 16 21:07:03.283326 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:03.283255    2530 generic.go:356] "Generic (PLEG): container finished" podID="c54c450a-b1ad-4ca0-b763-e2d1416a33a2" containerID="3cb4db802b60bf616b5fcb720d30278ba931767884cfdb78320752f2816b1962" exitCode=6
Jul 16 21:07:03.283722 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:03.283345    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj" event={"ID":"c54c450a-b1ad-4ca0-b763-e2d1416a33a2","Type":"ContainerDied","Data":"3cb4db802b60bf616b5fcb720d30278ba931767884cfdb78320752f2816b1962"}
Jul 16 21:07:03.283722 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:03.283599    2530 scope.go:117] "RemoveContainer" containerID="3cb4db802b60bf616b5fcb720d30278ba931767884cfdb78320752f2816b1962"
Jul 16 21:07:03.285463 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:03.285436    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" event={"ID":"aa609bb2-28bf-44e9-8c1c-3125dea60565","Type":"ContainerStarted","Data":"46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc"}
Jul 16 21:07:03.285685 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:03.285634    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:07:04.292603 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:04.292562    2530 generic.go:356] "Generic (PLEG): container finished" podID="c54c450a-b1ad-4ca0-b763-e2d1416a33a2" containerID="0cd4a744b22233fa756ddb7e36e4cbb3121396391cb14e6f4d62c9cb4c0c0df8" exitCode=6
Jul 16 21:07:04.293085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:04.292647    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj" event={"ID":"c54c450a-b1ad-4ca0-b763-e2d1416a33a2","Type":"ContainerDied","Data":"0cd4a744b22233fa756ddb7e36e4cbb3121396391cb14e6f4d62c9cb4c0c0df8"}
Jul 16 21:07:04.293085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:04.292734    2530 scope.go:117] "RemoveContainer" containerID="3cb4db802b60bf616b5fcb720d30278ba931767884cfdb78320752f2816b1962"
Jul 16 21:07:04.293085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:04.292958    2530 scope.go:117] "RemoveContainer" containerID="0cd4a744b22233fa756ddb7e36e4cbb3121396391cb14e6f4d62c9cb4c0c0df8"
Jul 16 21:07:04.293313 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:07:04.293173    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj_odh-ai-gateway-infra(c54c450a-b1ad-4ca0-b763-e2d1416a33a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj" podUID="c54c450a-b1ad-4ca0-b763-e2d1416a33a2"
Jul 16 21:07:04.294814 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:04.294791    2530 generic.go:356] "Generic (PLEG): container finished" podID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerID="46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc" exitCode=1
Jul 16 21:07:04.294949 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:04.294869    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" event={"ID":"aa609bb2-28bf-44e9-8c1c-3125dea60565","Type":"ContainerDied","Data":"46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc"}
Jul 16 21:07:04.295137 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:04.295092    2530 scope.go:117] "RemoveContainer" containerID="46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc"
Jul 16 21:07:04.295326 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:07:04.295304    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:07:04.303861 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:04.303843    2530 scope.go:117] "RemoveContainer" containerID="e3cf844042ae232d8b719f318f08c2af965290ddd4263b9c12b06df961ded70f"
Jul 16 21:07:05.300656 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:05.300627    2530 scope.go:117] "RemoveContainer" containerID="0cd4a744b22233fa756ddb7e36e4cbb3121396391cb14e6f4d62c9cb4c0c0df8"
Jul 16 21:07:05.301073 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:07:05.300809    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj_odh-ai-gateway-infra(c54c450a-b1ad-4ca0-b763-e2d1416a33a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj" podUID="c54c450a-b1ad-4ca0-b763-e2d1416a33a2"
Jul 16 21:07:05.302231 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:05.302211    2530 scope.go:117] "RemoveContainer" containerID="46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc"
Jul 16 21:07:05.302357 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:07:05.302340    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:07:06.143782 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:06.143740    2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:07:06.306205 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:06.306177    2530 scope.go:117] "RemoveContainer" containerID="46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc"
Jul 16 21:07:06.306564 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:07:06.306344    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:07:08.502832 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:08.502786    2530 scope.go:117] "RemoveContainer" containerID="11f346de163e5f86e4f753665a699e4a86945c618325f2afa70b6ed4f8b95376"
Jul 16 21:07:09.326873 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:09.326840    2530 generic.go:356] "Generic (PLEG): container finished" podID="9df048b2-21c9-4a81-8bcc-203fcb1054bd" containerID="171d825657d3f89b948df68ba7effc595cfa0764833b984489f80808e8b3c170" exitCode=6
Jul 16 21:07:09.327122 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:09.326891    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt" event={"ID":"9df048b2-21c9-4a81-8bcc-203fcb1054bd","Type":"ContainerDied","Data":"171d825657d3f89b948df68ba7effc595cfa0764833b984489f80808e8b3c170"}
Jul 16 21:07:09.327122 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:09.326938    2530 scope.go:117] "RemoveContainer" containerID="11f346de163e5f86e4f753665a699e4a86945c618325f2afa70b6ed4f8b95376"
Jul 16 21:07:09.327350 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:09.327325    2530 scope.go:117] "RemoveContainer" containerID="171d825657d3f89b948df68ba7effc595cfa0764833b984489f80808e8b3c170"
Jul 16 21:07:09.327559 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:07:09.327531    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt_odh-ai-gateway-infra(9df048b2-21c9-4a81-8bcc-203fcb1054bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt" podUID="9df048b2-21c9-4a81-8bcc-203fcb1054bd"
Jul 16 21:07:10.353648 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:10.353615    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt"]
Jul 16 21:07:10.483119 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:10.483084    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt"
Jul 16 21:07:10.537029 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:10.537007    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df048b2-21c9-4a81-8bcc-203fcb1054bd-maas-api-service-ca\") pod \"9df048b2-21c9-4a81-8bcc-203fcb1054bd\" (UID: \"9df048b2-21c9-4a81-8bcc-203fcb1054bd\") "
Jul 16 21:07:10.537354 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:10.537331    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df048b2-21c9-4a81-8bcc-203fcb1054bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9df048b2-21c9-4a81-8bcc-203fcb1054bd" (UID: "9df048b2-21c9-4a81-8bcc-203fcb1054bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:07:10.638211 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:10.638148    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9df048b2-21c9-4a81-8bcc-203fcb1054bd-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:07:11.338715 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:11.338688    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt"
Jul 16 21:07:11.338916 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:11.338696    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt" event={"ID":"9df048b2-21c9-4a81-8bcc-203fcb1054bd","Type":"ContainerDied","Data":"f6a32838a7460799833aa6fd7ce0421df99b5d3bcf88b17e67cb7b0dd2c8b2bc"}
Jul 16 21:07:11.338916 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:11.338816    2530 scope.go:117] "RemoveContainer" containerID="171d825657d3f89b948df68ba7effc595cfa0764833b984489f80808e8b3c170"
Jul 16 21:07:11.360325 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:11.360302    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt"]
Jul 16 21:07:11.366830 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:11.366810    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-v7hkt"]
Jul 16 21:07:12.507908 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:12.507869    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9df048b2-21c9-4a81-8bcc-203fcb1054bd" path="/var/lib/kubelet/pods/9df048b2-21c9-4a81-8bcc-203fcb1054bd/volumes"
Jul 16 21:07:17.501781 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:17.501745    2530 scope.go:117] "RemoveContainer" containerID="0cd4a744b22233fa756ddb7e36e4cbb3121396391cb14e6f4d62c9cb4c0c0df8"
Jul 16 21:07:18.372669 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:18.372590    2530 generic.go:356] "Generic (PLEG): container finished" podID="c54c450a-b1ad-4ca0-b763-e2d1416a33a2" containerID="0441d42f7a9f7a62a26f2c0614517f6313d0b8c3c7352828e0e47ccb90e058f4" exitCode=6
Jul 16 21:07:18.372819 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:18.372665    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj" event={"ID":"c54c450a-b1ad-4ca0-b763-e2d1416a33a2","Type":"ContainerDied","Data":"0441d42f7a9f7a62a26f2c0614517f6313d0b8c3c7352828e0e47ccb90e058f4"}
Jul 16 21:07:18.372819 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:18.372729    2530 scope.go:117] "RemoveContainer" containerID="0cd4a744b22233fa756ddb7e36e4cbb3121396391cb14e6f4d62c9cb4c0c0df8"
Jul 16 21:07:18.373165 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:18.373143    2530 scope.go:117] "RemoveContainer" containerID="0441d42f7a9f7a62a26f2c0614517f6313d0b8c3c7352828e0e47ccb90e058f4"
Jul 16 21:07:18.373401 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:07:18.373379    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj_odh-ai-gateway-infra(c54c450a-b1ad-4ca0-b763-e2d1416a33a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj" podUID="c54c450a-b1ad-4ca0-b763-e2d1416a33a2"
Jul 16 21:07:19.397617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:19.397586    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj"]
Jul 16 21:07:19.502944 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:19.502922    2530 scope.go:117] "RemoveContainer" containerID="46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc"
Jul 16 21:07:19.503092 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:07:19.503070    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:07:19.533023 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:19.533004    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj"
Jul 16 21:07:19.614213 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:19.614176    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54c450a-b1ad-4ca0-b763-e2d1416a33a2-maas-api-service-ca\") pod \"c54c450a-b1ad-4ca0-b763-e2d1416a33a2\" (UID: \"c54c450a-b1ad-4ca0-b763-e2d1416a33a2\") "
Jul 16 21:07:19.614523 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:19.614490    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c54c450a-b1ad-4ca0-b763-e2d1416a33a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c54c450a-b1ad-4ca0-b763-e2d1416a33a2" (UID: "c54c450a-b1ad-4ca0-b763-e2d1416a33a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:07:19.614668 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:19.614572    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c54c450a-b1ad-4ca0-b763-e2d1416a33a2-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:07:20.384992 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:20.384959    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj" event={"ID":"c54c450a-b1ad-4ca0-b763-e2d1416a33a2","Type":"ContainerDied","Data":"a61fbf3acc0c35d1f7c96b97751c5b8c292a19d028472c25c136b307c44b8c70"}
Jul 16 21:07:20.384992 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:20.384989    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj"
Jul 16 21:07:20.385274 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:20.385021    2530 scope.go:117] "RemoveContainer" containerID="0441d42f7a9f7a62a26f2c0614517f6313d0b8c3c7352828e0e47ccb90e058f4"
Jul 16 21:07:20.409431 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:20.409404    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj"]
Jul 16 21:07:20.412345 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:20.412323    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-kl4nj"]
Jul 16 21:07:20.507220 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:20.507200    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c54c450a-b1ad-4ca0-b763-e2d1416a33a2" path="/var/lib/kubelet/pods/c54c450a-b1ad-4ca0-b763-e2d1416a33a2/volumes"
Jul 16 21:07:34.502140 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:34.502088    2530 scope.go:117] "RemoveContainer" containerID="46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc"
Jul 16 21:07:34.502656 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:07:34.502287    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:07:49.502653 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:49.502627    2530 scope.go:117] "RemoveContainer" containerID="46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc"
Jul 16 21:07:49.503051 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:07:49.502775    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:07:50.853437 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.853389    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k"]
Jul 16 21:07:50.853809 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.853736    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9df048b2-21c9-4a81-8bcc-203fcb1054bd" containerName="revoke-keys"
Jul 16 21:07:50.853809 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.853748    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="9df048b2-21c9-4a81-8bcc-203fcb1054bd" containerName="revoke-keys"
Jul 16 21:07:50.853809 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.853756    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c54c450a-b1ad-4ca0-b763-e2d1416a33a2" containerName="revoke-keys"
Jul 16 21:07:50.853809 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.853761    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c54c450a-b1ad-4ca0-b763-e2d1416a33a2" containerName="revoke-keys"
Jul 16 21:07:50.853809 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.853768    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9df048b2-21c9-4a81-8bcc-203fcb1054bd" containerName="revoke-keys"
Jul 16 21:07:50.853809 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.853774    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="9df048b2-21c9-4a81-8bcc-203fcb1054bd" containerName="revoke-keys"
Jul 16 21:07:50.853809 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.853787    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c54c450a-b1ad-4ca0-b763-e2d1416a33a2" containerName="revoke-keys"
Jul 16 21:07:50.853809 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.853795    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c54c450a-b1ad-4ca0-b763-e2d1416a33a2" containerName="revoke-keys"
Jul 16 21:07:50.854058 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.853858    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c54c450a-b1ad-4ca0-b763-e2d1416a33a2" containerName="revoke-keys"
Jul 16 21:07:50.854058 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.853866    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c54c450a-b1ad-4ca0-b763-e2d1416a33a2" containerName="revoke-keys"
Jul 16 21:07:50.854058 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.853876    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="9df048b2-21c9-4a81-8bcc-203fcb1054bd" containerName="revoke-keys"
Jul 16 21:07:50.854058 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.853885    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c54c450a-b1ad-4ca0-b763-e2d1416a33a2" containerName="revoke-keys"
Jul 16 21:07:50.854058 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.853891    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="9df048b2-21c9-4a81-8bcc-203fcb1054bd" containerName="revoke-keys"
Jul 16 21:07:50.854058 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.853896    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="9df048b2-21c9-4a81-8bcc-203fcb1054bd" containerName="revoke-keys"
Jul 16 21:07:50.856796 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.856781    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k"
Jul 16 21:07:50.859321 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.859305    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:07:50.864457 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.864424    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ddba0f-4018-4688-ad83-3e505de9bf34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k\" (UID: \"65ddba0f-4018-4688-ad83-3e505de9bf34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k"
Jul 16 21:07:50.865210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.865184    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k"]
Jul 16 21:07:50.965453 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.965424    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ddba0f-4018-4688-ad83-3e505de9bf34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k\" (UID: \"65ddba0f-4018-4688-ad83-3e505de9bf34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k"
Jul 16 21:07:50.966004 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:50.965986    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ddba0f-4018-4688-ad83-3e505de9bf34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k\" (UID: \"65ddba0f-4018-4688-ad83-3e505de9bf34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k"
Jul 16 21:07:51.168056 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:51.167969    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k"
Jul 16 21:07:51.299302 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:51.299277    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k"]
Jul 16 21:07:51.301259 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:07:51.301236    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ddba0f_4018_4688_ad83_3e505de9bf34.slice/crio-95c0d008bcb9b4457d6c781800c7b52205223ee02437100dc31ef7a710f887de WatchSource:0}: Error finding container 95c0d008bcb9b4457d6c781800c7b52205223ee02437100dc31ef7a710f887de: Status 404 returned error can't find the container with id 95c0d008bcb9b4457d6c781800c7b52205223ee02437100dc31ef7a710f887de
Jul 16 21:07:51.520129 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:51.520073    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k" event={"ID":"65ddba0f-4018-4688-ad83-3e505de9bf34","Type":"ContainerStarted","Data":"95c0d008bcb9b4457d6c781800c7b52205223ee02437100dc31ef7a710f887de"}
Jul 16 21:07:51.609085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:51.609052    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk"]
Jul 16 21:07:51.609480 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:51.609454    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9df048b2-21c9-4a81-8bcc-203fcb1054bd" containerName="revoke-keys"
Jul 16 21:07:51.609480 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:51.609478    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="9df048b2-21c9-4a81-8bcc-203fcb1054bd" containerName="revoke-keys"
Jul 16 21:07:51.609617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:51.609494    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c54c450a-b1ad-4ca0-b763-e2d1416a33a2" containerName="revoke-keys"
Jul 16 21:07:51.609617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:51.609500    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c54c450a-b1ad-4ca0-b763-e2d1416a33a2" containerName="revoke-keys"
Jul 16 21:07:51.613486 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:51.613470    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk"
Jul 16 21:07:51.620386 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:51.620126    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk"]
Jul 16 21:07:51.671359 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:51.671330    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7442f25-d66c-42f2-8fef-eaabcb47fe60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk\" (UID: \"c7442f25-d66c-42f2-8fef-eaabcb47fe60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk"
Jul 16 21:07:51.771924 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:51.771865    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7442f25-d66c-42f2-8fef-eaabcb47fe60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk\" (UID: \"c7442f25-d66c-42f2-8fef-eaabcb47fe60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk"
Jul 16 21:07:51.772480 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:51.772461    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7442f25-d66c-42f2-8fef-eaabcb47fe60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk\" (UID: \"c7442f25-d66c-42f2-8fef-eaabcb47fe60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk"
Jul 16 21:07:51.924795 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:51.924766    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk"
Jul 16 21:07:52.052837 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:52.052815    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk"]
Jul 16 21:07:52.054846 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:07:52.054819    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7442f25_d66c_42f2_8fef_eaabcb47fe60.slice/crio-80ccd06872741d1a42723a25e2f42d205e6bf07550fdd949f3353303d414cabf WatchSource:0}: Error finding container 80ccd06872741d1a42723a25e2f42d205e6bf07550fdd949f3353303d414cabf: Status 404 returned error can't find the container with id 80ccd06872741d1a42723a25e2f42d205e6bf07550fdd949f3353303d414cabf
Jul 16 21:07:52.524738 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:52.524707    2530 generic.go:356] "Generic (PLEG): container finished" podID="65ddba0f-4018-4688-ad83-3e505de9bf34" containerID="91df15fea20a8ce1df820e2c3f51806e974eee593c099fb8a99156e35704397e" exitCode=7
Jul 16 21:07:52.524921 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:52.524788    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k" event={"ID":"65ddba0f-4018-4688-ad83-3e505de9bf34","Type":"ContainerDied","Data":"91df15fea20a8ce1df820e2c3f51806e974eee593c099fb8a99156e35704397e"}
Jul 16 21:07:52.524993 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:52.524972    2530 scope.go:117] "RemoveContainer" containerID="91df15fea20a8ce1df820e2c3f51806e974eee593c099fb8a99156e35704397e"
Jul 16 21:07:52.526318 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:52.526289    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk" event={"ID":"c7442f25-d66c-42f2-8fef-eaabcb47fe60","Type":"ContainerStarted","Data":"7c23f9221cb3c8da57dc1c745dde93d590927a05b83756bb2a43abc519114894"}
Jul 16 21:07:52.526431 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:52.526333    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk" event={"ID":"c7442f25-d66c-42f2-8fef-eaabcb47fe60","Type":"ContainerStarted","Data":"80ccd06872741d1a42723a25e2f42d205e6bf07550fdd949f3353303d414cabf"}
Jul 16 21:07:52.557200 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:52.557162    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk" podStartSLOduration=1.557149221 podStartE2EDuration="1.557149221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:07:52.555777842 +0000 UTC m=+2110.577365625" watchObservedRunningTime="2026-07-16 21:07:52.557149221 +0000 UTC m=+2110.578736988"
Jul 16 21:07:53.531936 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:53.531901    2530 generic.go:356] "Generic (PLEG): container finished" podID="c7442f25-d66c-42f2-8fef-eaabcb47fe60" containerID="7c23f9221cb3c8da57dc1c745dde93d590927a05b83756bb2a43abc519114894" exitCode=6
Jul 16 21:07:53.532344 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:53.531994    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk" event={"ID":"c7442f25-d66c-42f2-8fef-eaabcb47fe60","Type":"ContainerDied","Data":"7c23f9221cb3c8da57dc1c745dde93d590927a05b83756bb2a43abc519114894"}
Jul 16 21:07:53.532344 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:53.532176    2530 scope.go:117] "RemoveContainer" containerID="7c23f9221cb3c8da57dc1c745dde93d590927a05b83756bb2a43abc519114894"
Jul 16 21:07:53.533725 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:53.533704    2530 generic.go:356] "Generic (PLEG): container finished" podID="65ddba0f-4018-4688-ad83-3e505de9bf34" containerID="255cf257ec39ea6990c654c2da7c76a49e00c862f9a82ac9abf2773e820e9e71" exitCode=7
Jul 16 21:07:53.533817 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:53.533738    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k" event={"ID":"65ddba0f-4018-4688-ad83-3e505de9bf34","Type":"ContainerDied","Data":"255cf257ec39ea6990c654c2da7c76a49e00c862f9a82ac9abf2773e820e9e71"}
Jul 16 21:07:53.533817 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:53.533781    2530 scope.go:117] "RemoveContainer" containerID="91df15fea20a8ce1df820e2c3f51806e974eee593c099fb8a99156e35704397e"
Jul 16 21:07:53.533961 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:53.533925    2530 scope.go:117] "RemoveContainer" containerID="255cf257ec39ea6990c654c2da7c76a49e00c862f9a82ac9abf2773e820e9e71"
Jul 16 21:07:53.534253 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:07:53.534136    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k_odh-ai-gateway-infra(65ddba0f-4018-4688-ad83-3e505de9bf34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k" podUID="65ddba0f-4018-4688-ad83-3e505de9bf34"
Jul 16 21:07:54.540164 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:54.540067    2530 generic.go:356] "Generic (PLEG): container finished" podID="c7442f25-d66c-42f2-8fef-eaabcb47fe60" containerID="62ea8a322cf0aec43824cb94bdd98dddb1387e56003231cac9937978f32b936c" exitCode=6
Jul 16 21:07:54.540164 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:54.540139    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk" event={"ID":"c7442f25-d66c-42f2-8fef-eaabcb47fe60","Type":"ContainerDied","Data":"62ea8a322cf0aec43824cb94bdd98dddb1387e56003231cac9937978f32b936c"}
Jul 16 21:07:54.540592 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:54.540202    2530 scope.go:117] "RemoveContainer" containerID="7c23f9221cb3c8da57dc1c745dde93d590927a05b83756bb2a43abc519114894"
Jul 16 21:07:54.540592 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:54.540354    2530 scope.go:117] "RemoveContainer" containerID="62ea8a322cf0aec43824cb94bdd98dddb1387e56003231cac9937978f32b936c"
Jul 16 21:07:54.540592 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:07:54.540536    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk_odh-ai-gateway-infra(c7442f25-d66c-42f2-8fef-eaabcb47fe60)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk" podUID="c7442f25-d66c-42f2-8fef-eaabcb47fe60"
Jul 16 21:07:54.542001 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:54.541986    2530 scope.go:117] "RemoveContainer" containerID="255cf257ec39ea6990c654c2da7c76a49e00c862f9a82ac9abf2773e820e9e71"
Jul 16 21:07:54.542159 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:07:54.542142    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k_odh-ai-gateway-infra(65ddba0f-4018-4688-ad83-3e505de9bf34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k" podUID="65ddba0f-4018-4688-ad83-3e505de9bf34"
Jul 16 21:07:55.547373 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:07:55.547345    2530 scope.go:117] "RemoveContainer" containerID="62ea8a322cf0aec43824cb94bdd98dddb1387e56003231cac9937978f32b936c"
Jul 16 21:07:55.547716 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:07:55.547519    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk_odh-ai-gateway-infra(c7442f25-d66c-42f2-8fef-eaabcb47fe60)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk" podUID="c7442f25-d66c-42f2-8fef-eaabcb47fe60"
Jul 16 21:08:01.502401 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:01.502368    2530 scope.go:117] "RemoveContainer" containerID="46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc"
Jul 16 21:08:01.502784 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:08:01.502576    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:08:07.502487 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:07.502461    2530 scope.go:117] "RemoveContainer" containerID="62ea8a322cf0aec43824cb94bdd98dddb1387e56003231cac9937978f32b936c"
Jul 16 21:08:08.607400 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:08.607361    2530 generic.go:356] "Generic (PLEG): container finished" podID="c7442f25-d66c-42f2-8fef-eaabcb47fe60" containerID="4c215c26e17945cbbc3521da2416af32e6d9d3630e20e501a0cecd64d1a389b9" exitCode=6
Jul 16 21:08:08.607839 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:08.607405    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk" event={"ID":"c7442f25-d66c-42f2-8fef-eaabcb47fe60","Type":"ContainerDied","Data":"4c215c26e17945cbbc3521da2416af32e6d9d3630e20e501a0cecd64d1a389b9"}
Jul 16 21:08:08.607839 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:08.607466    2530 scope.go:117] "RemoveContainer" containerID="62ea8a322cf0aec43824cb94bdd98dddb1387e56003231cac9937978f32b936c"
Jul 16 21:08:08.607839 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:08.607786    2530 scope.go:117] "RemoveContainer" containerID="4c215c26e17945cbbc3521da2416af32e6d9d3630e20e501a0cecd64d1a389b9"
Jul 16 21:08:08.608025 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:08:08.607998    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk_odh-ai-gateway-infra(c7442f25-d66c-42f2-8fef-eaabcb47fe60)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk" podUID="c7442f25-d66c-42f2-8fef-eaabcb47fe60"
Jul 16 21:08:09.502399 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:09.502367    2530 scope.go:117] "RemoveContainer" containerID="255cf257ec39ea6990c654c2da7c76a49e00c862f9a82ac9abf2773e820e9e71"
Jul 16 21:08:09.639642 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:09.639609    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk"]
Jul 16 21:08:09.766686 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:09.766629    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk"
Jul 16 21:08:09.806625 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:09.806598    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7442f25-d66c-42f2-8fef-eaabcb47fe60-maas-api-service-ca\") pod \"c7442f25-d66c-42f2-8fef-eaabcb47fe60\" (UID: \"c7442f25-d66c-42f2-8fef-eaabcb47fe60\") "
Jul 16 21:08:09.806901 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:09.806880    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7442f25-d66c-42f2-8fef-eaabcb47fe60-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c7442f25-d66c-42f2-8fef-eaabcb47fe60" (UID: "c7442f25-d66c-42f2-8fef-eaabcb47fe60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:08:09.907644 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:09.907623    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7442f25-d66c-42f2-8fef-eaabcb47fe60-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:08:10.618197 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:10.618163    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk"
Jul 16 21:08:10.618197 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:10.618187    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk" event={"ID":"c7442f25-d66c-42f2-8fef-eaabcb47fe60","Type":"ContainerDied","Data":"80ccd06872741d1a42723a25e2f42d205e6bf07550fdd949f3353303d414cabf"}
Jul 16 21:08:10.618405 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:10.618257    2530 scope.go:117] "RemoveContainer" containerID="4c215c26e17945cbbc3521da2416af32e6d9d3630e20e501a0cecd64d1a389b9"
Jul 16 21:08:10.620198 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:10.620162    2530 generic.go:356] "Generic (PLEG): container finished" podID="65ddba0f-4018-4688-ad83-3e505de9bf34" containerID="8a9656908d660837ac963358ad6eba1ba10fb3076bf0c0b276795bf9f1eb1816" exitCode=7
Jul 16 21:08:10.620281 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:10.620251    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k" event={"ID":"65ddba0f-4018-4688-ad83-3e505de9bf34","Type":"ContainerDied","Data":"8a9656908d660837ac963358ad6eba1ba10fb3076bf0c0b276795bf9f1eb1816"}
Jul 16 21:08:10.620564 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:10.620544    2530 scope.go:117] "RemoveContainer" containerID="8a9656908d660837ac963358ad6eba1ba10fb3076bf0c0b276795bf9f1eb1816"
Jul 16 21:08:10.620741 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:08:10.620723    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k_odh-ai-gateway-infra(65ddba0f-4018-4688-ad83-3e505de9bf34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k" podUID="65ddba0f-4018-4688-ad83-3e505de9bf34"
Jul 16 21:08:10.626749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:10.626734    2530 scope.go:117] "RemoveContainer" containerID="255cf257ec39ea6990c654c2da7c76a49e00c862f9a82ac9abf2773e820e9e71"
Jul 16 21:08:10.639147 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:10.639129    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk"]
Jul 16 21:08:10.646488 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:10.646465    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-gsbzk"]
Jul 16 21:08:11.682885 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:11.682850    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k"]
Jul 16 21:08:11.816565 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:11.816539    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k"
Jul 16 21:08:11.923832 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:11.923810    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ddba0f-4018-4688-ad83-3e505de9bf34-maas-api-service-ca\") pod \"65ddba0f-4018-4688-ad83-3e505de9bf34\" (UID: \"65ddba0f-4018-4688-ad83-3e505de9bf34\") "
Jul 16 21:08:11.924140 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:11.924095    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ddba0f-4018-4688-ad83-3e505de9bf34-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "65ddba0f-4018-4688-ad83-3e505de9bf34" (UID: "65ddba0f-4018-4688-ad83-3e505de9bf34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:08:12.024785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:12.024764    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ddba0f-4018-4688-ad83-3e505de9bf34-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:08:12.512872 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:12.512840    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7442f25-d66c-42f2-8fef-eaabcb47fe60" path="/var/lib/kubelet/pods/c7442f25-d66c-42f2-8fef-eaabcb47fe60/volumes"
Jul 16 21:08:12.633180 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:12.633150    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k" event={"ID":"65ddba0f-4018-4688-ad83-3e505de9bf34","Type":"ContainerDied","Data":"95c0d008bcb9b4457d6c781800c7b52205223ee02437100dc31ef7a710f887de"}
Jul 16 21:08:12.633326 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:12.633185    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k"
Jul 16 21:08:12.633326 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:12.633205    2530 scope.go:117] "RemoveContainer" containerID="8a9656908d660837ac963358ad6eba1ba10fb3076bf0c0b276795bf9f1eb1816"
Jul 16 21:08:12.651224 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:12.651202    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k"]
Jul 16 21:08:12.656139 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:12.656118    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-kvg4k"]
Jul 16 21:08:14.502744 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:14.502709    2530 scope.go:117] "RemoveContainer" containerID="46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc"
Jul 16 21:08:14.503318 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:08:14.502916    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:08:14.507951 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:14.507924    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65ddba0f-4018-4688-ad83-3e505de9bf34" path="/var/lib/kubelet/pods/65ddba0f-4018-4688-ad83-3e505de9bf34/volumes"
Jul 16 21:08:27.502547 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:27.502513    2530 scope.go:117] "RemoveContainer" containerID="46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc"
Jul 16 21:08:27.502924 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:08:27.502695    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:08:38.503031 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:38.502946    2530 scope.go:117] "RemoveContainer" containerID="46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc"
Jul 16 21:08:38.503453 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:08:38.503160    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:08:43.621017 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.620977    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz"]
Jul 16 21:08:43.621436 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.621341    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7442f25-d66c-42f2-8fef-eaabcb47fe60" containerName="revoke-keys"
Jul 16 21:08:43.621436 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.621353    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7442f25-d66c-42f2-8fef-eaabcb47fe60" containerName="revoke-keys"
Jul 16 21:08:43.621436 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.621367    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65ddba0f-4018-4688-ad83-3e505de9bf34" containerName="revoke-keys"
Jul 16 21:08:43.621436 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.621372    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="65ddba0f-4018-4688-ad83-3e505de9bf34" containerName="revoke-keys"
Jul 16 21:08:43.621436 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.621379    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65ddba0f-4018-4688-ad83-3e505de9bf34" containerName="revoke-keys"
Jul 16 21:08:43.621436 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.621384    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="65ddba0f-4018-4688-ad83-3e505de9bf34" containerName="revoke-keys"
Jul 16 21:08:43.621436 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.621391    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7442f25-d66c-42f2-8fef-eaabcb47fe60" containerName="revoke-keys"
Jul 16 21:08:43.621436 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.621396    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7442f25-d66c-42f2-8fef-eaabcb47fe60" containerName="revoke-keys"
Jul 16 21:08:43.621436 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.621403    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65ddba0f-4018-4688-ad83-3e505de9bf34" containerName="revoke-keys"
Jul 16 21:08:43.621436 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.621407    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="65ddba0f-4018-4688-ad83-3e505de9bf34" containerName="revoke-keys"
Jul 16 21:08:43.621436 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.621416    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7442f25-d66c-42f2-8fef-eaabcb47fe60" containerName="revoke-keys"
Jul 16 21:08:43.621436 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.621420    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7442f25-d66c-42f2-8fef-eaabcb47fe60" containerName="revoke-keys"
Jul 16 21:08:43.621792 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.621474    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7442f25-d66c-42f2-8fef-eaabcb47fe60" containerName="revoke-keys"
Jul 16 21:08:43.621792 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.621482    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7442f25-d66c-42f2-8fef-eaabcb47fe60" containerName="revoke-keys"
Jul 16 21:08:43.621792 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.621488    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="65ddba0f-4018-4688-ad83-3e505de9bf34" containerName="revoke-keys"
Jul 16 21:08:43.621792 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.621495    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="65ddba0f-4018-4688-ad83-3e505de9bf34" containerName="revoke-keys"
Jul 16 21:08:43.625744 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.625724    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz"
Jul 16 21:08:43.632180 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.631966    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:08:43.640697 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.640674    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz"]
Jul 16 21:08:43.788235 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.788192    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc473c3-22e1-4d46-b08c-3734f7e9782a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz\" (UID: \"4cc473c3-22e1-4d46-b08c-3734f7e9782a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz"
Jul 16 21:08:43.889740 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.889658    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc473c3-22e1-4d46-b08c-3734f7e9782a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz\" (UID: \"4cc473c3-22e1-4d46-b08c-3734f7e9782a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz"
Jul 16 21:08:43.890299 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.890282    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc473c3-22e1-4d46-b08c-3734f7e9782a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz\" (UID: \"4cc473c3-22e1-4d46-b08c-3734f7e9782a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz"
Jul 16 21:08:43.935265 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:43.935227    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz"
Jul 16 21:08:44.077915 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:44.077880    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz"]
Jul 16 21:08:44.080929 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:08:44.080895    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc473c3_22e1_4d46_b08c_3734f7e9782a.slice/crio-42af655f962900ee5f00fc537d7efcd5143419ed40f7e5a9e9f698662e8553b7 WatchSource:0}: Error finding container 42af655f962900ee5f00fc537d7efcd5143419ed40f7e5a9e9f698662e8553b7: Status 404 returned error can't find the container with id 42af655f962900ee5f00fc537d7efcd5143419ed40f7e5a9e9f698662e8553b7
Jul 16 21:08:44.767197 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:44.767153    2530 generic.go:356] "Generic (PLEG): container finished" podID="4cc473c3-22e1-4d46-b08c-3734f7e9782a" containerID="88a2b019256fc4efabf4eeccc634e57e46729c332b96f3f4fa72159bed316b34" exitCode=6
Jul 16 21:08:44.767658 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:44.767250    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz" event={"ID":"4cc473c3-22e1-4d46-b08c-3734f7e9782a","Type":"ContainerDied","Data":"88a2b019256fc4efabf4eeccc634e57e46729c332b96f3f4fa72159bed316b34"}
Jul 16 21:08:44.767658 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:44.767306    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz" event={"ID":"4cc473c3-22e1-4d46-b08c-3734f7e9782a","Type":"ContainerStarted","Data":"42af655f962900ee5f00fc537d7efcd5143419ed40f7e5a9e9f698662e8553b7"}
Jul 16 21:08:44.767658 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:44.767415    2530 scope.go:117] "RemoveContainer" containerID="88a2b019256fc4efabf4eeccc634e57e46729c332b96f3f4fa72159bed316b34"
Jul 16 21:08:45.773903 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:45.773862    2530 generic.go:356] "Generic (PLEG): container finished" podID="4cc473c3-22e1-4d46-b08c-3734f7e9782a" containerID="3bbf303a81a76c5fdc7168bdf23e2d694f5d0b585f4b0f56c81bf10ff58e897c" exitCode=6
Jul 16 21:08:45.774421 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:45.773949    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz" event={"ID":"4cc473c3-22e1-4d46-b08c-3734f7e9782a","Type":"ContainerDied","Data":"3bbf303a81a76c5fdc7168bdf23e2d694f5d0b585f4b0f56c81bf10ff58e897c"}
Jul 16 21:08:45.774421 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:45.774002    2530 scope.go:117] "RemoveContainer" containerID="88a2b019256fc4efabf4eeccc634e57e46729c332b96f3f4fa72159bed316b34"
Jul 16 21:08:45.774421 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:45.774235    2530 scope.go:117] "RemoveContainer" containerID="3bbf303a81a76c5fdc7168bdf23e2d694f5d0b585f4b0f56c81bf10ff58e897c"
Jul 16 21:08:45.774533 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:08:45.774449    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz_odh-ai-gateway-infra(4cc473c3-22e1-4d46-b08c-3734f7e9782a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz" podUID="4cc473c3-22e1-4d46-b08c-3734f7e9782a"
Jul 16 21:08:46.780370 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:46.780340    2530 scope.go:117] "RemoveContainer" containerID="3bbf303a81a76c5fdc7168bdf23e2d694f5d0b585f4b0f56c81bf10ff58e897c"
Jul 16 21:08:46.780787 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:08:46.780543    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz_odh-ai-gateway-infra(4cc473c3-22e1-4d46-b08c-3734f7e9782a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz" podUID="4cc473c3-22e1-4d46-b08c-3734f7e9782a"
Jul 16 21:08:50.502064 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:50.502029    2530 scope.go:117] "RemoveContainer" containerID="46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc"
Jul 16 21:08:50.502502 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:08:50.502250    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:08:50.914575 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:50.914480    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq"]
Jul 16 21:08:50.914965 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:50.914945    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="65ddba0f-4018-4688-ad83-3e505de9bf34" containerName="revoke-keys"
Jul 16 21:08:50.915151 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:50.914969    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7442f25-d66c-42f2-8fef-eaabcb47fe60" containerName="revoke-keys"
Jul 16 21:08:50.919095 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:50.919072    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq"
Jul 16 21:08:50.927039 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:50.927009    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq"]
Jul 16 21:08:50.942873 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:50.942833    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89df93de-82f3-45ec-82aa-1d02deb4cd62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq\" (UID: \"89df93de-82f3-45ec-82aa-1d02deb4cd62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq"
Jul 16 21:08:51.044286 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:51.044243    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89df93de-82f3-45ec-82aa-1d02deb4cd62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq\" (UID: \"89df93de-82f3-45ec-82aa-1d02deb4cd62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq"
Jul 16 21:08:51.044873 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:51.044850    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89df93de-82f3-45ec-82aa-1d02deb4cd62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq\" (UID: \"89df93de-82f3-45ec-82aa-1d02deb4cd62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq"
Jul 16 21:08:51.231405 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:51.231369    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq"
Jul 16 21:08:51.362955 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:51.362899    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq"]
Jul 16 21:08:51.365145 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:08:51.365117    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89df93de_82f3_45ec_82aa_1d02deb4cd62.slice/crio-04b63de2308f56f770bfea321f907cb857f6122e74ab564b29397b00404c4474 WatchSource:0}: Error finding container 04b63de2308f56f770bfea321f907cb857f6122e74ab564b29397b00404c4474: Status 404 returned error can't find the container with id 04b63de2308f56f770bfea321f907cb857f6122e74ab564b29397b00404c4474
Jul 16 21:08:51.811567 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:51.811467    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq" event={"ID":"89df93de-82f3-45ec-82aa-1d02deb4cd62","Type":"ContainerStarted","Data":"469c750cf2fdc5ed39f6ccc811f5137c455a2283fad0ec9fa4c09937e1796eb7"}
Jul 16 21:08:51.811567 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:51.811538    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq" event={"ID":"89df93de-82f3-45ec-82aa-1d02deb4cd62","Type":"ContainerStarted","Data":"04b63de2308f56f770bfea321f907cb857f6122e74ab564b29397b00404c4474"}
Jul 16 21:08:51.830582 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:51.830531    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq" podStartSLOduration=1.830515697 podStartE2EDuration="1.830515697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:08:51.830223476 +0000 UTC m=+2169.851811244" watchObservedRunningTime="2026-07-16 21:08:51.830515697 +0000 UTC m=+2169.852103464"
Jul 16 21:08:52.817764 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:52.817727    2530 generic.go:356] "Generic (PLEG): container finished" podID="89df93de-82f3-45ec-82aa-1d02deb4cd62" containerID="469c750cf2fdc5ed39f6ccc811f5137c455a2283fad0ec9fa4c09937e1796eb7" exitCode=7
Jul 16 21:08:52.818221 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:52.817809    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq" event={"ID":"89df93de-82f3-45ec-82aa-1d02deb4cd62","Type":"ContainerDied","Data":"469c750cf2fdc5ed39f6ccc811f5137c455a2283fad0ec9fa4c09937e1796eb7"}
Jul 16 21:08:52.818221 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:52.817955    2530 scope.go:117] "RemoveContainer" containerID="469c750cf2fdc5ed39f6ccc811f5137c455a2283fad0ec9fa4c09937e1796eb7"
Jul 16 21:08:53.823587 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:53.823547    2530 generic.go:356] "Generic (PLEG): container finished" podID="89df93de-82f3-45ec-82aa-1d02deb4cd62" containerID="90f8ee74016e53bcccd1d5cf5ffe6f4cca4777fbd40c42ed6c3d43ce6dbdcf8f" exitCode=7
Jul 16 21:08:53.824008 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:53.823633    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq" event={"ID":"89df93de-82f3-45ec-82aa-1d02deb4cd62","Type":"ContainerDied","Data":"90f8ee74016e53bcccd1d5cf5ffe6f4cca4777fbd40c42ed6c3d43ce6dbdcf8f"}
Jul 16 21:08:53.824008 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:53.823696    2530 scope.go:117] "RemoveContainer" containerID="469c750cf2fdc5ed39f6ccc811f5137c455a2283fad0ec9fa4c09937e1796eb7"
Jul 16 21:08:53.824008 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:53.823942    2530 scope.go:117] "RemoveContainer" containerID="90f8ee74016e53bcccd1d5cf5ffe6f4cca4777fbd40c42ed6c3d43ce6dbdcf8f"
Jul 16 21:08:53.824188 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:08:53.824160    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq_odh-ai-gateway-infra(89df93de-82f3-45ec-82aa-1d02deb4cd62)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq" podUID="89df93de-82f3-45ec-82aa-1d02deb4cd62"
Jul 16 21:08:54.829472 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:54.829446    2530 scope.go:117] "RemoveContainer" containerID="90f8ee74016e53bcccd1d5cf5ffe6f4cca4777fbd40c42ed6c3d43ce6dbdcf8f"
Jul 16 21:08:54.829845 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:08:54.829629    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq_odh-ai-gateway-infra(89df93de-82f3-45ec-82aa-1d02deb4cd62)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq" podUID="89df93de-82f3-45ec-82aa-1d02deb4cd62"
Jul 16 21:08:59.501834 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:59.501794    2530 scope.go:117] "RemoveContainer" containerID="3bbf303a81a76c5fdc7168bdf23e2d694f5d0b585f4b0f56c81bf10ff58e897c"
Jul 16 21:08:59.852238 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:59.852200    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz" event={"ID":"4cc473c3-22e1-4d46-b08c-3734f7e9782a","Type":"ContainerStarted","Data":"2ef59f15bf307e4502acd369d53ce8fa82b51eb3ee532af5eba83c8fb2241867"}
Jul 16 21:08:59.872418 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:08:59.872366    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz" podStartSLOduration=16.872349245 podStartE2EDuration="16.872349245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:08:59.869991719 +0000 UTC m=+2177.891579496" watchObservedRunningTime="2026-07-16 21:08:59.872349245 +0000 UTC m=+2177.893937012"
Jul 16 21:09:00.858276 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:00.858235    2530 generic.go:356] "Generic (PLEG): container finished" podID="4cc473c3-22e1-4d46-b08c-3734f7e9782a" containerID="2ef59f15bf307e4502acd369d53ce8fa82b51eb3ee532af5eba83c8fb2241867" exitCode=6
Jul 16 21:09:00.858755 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:00.858305    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz" event={"ID":"4cc473c3-22e1-4d46-b08c-3734f7e9782a","Type":"ContainerDied","Data":"2ef59f15bf307e4502acd369d53ce8fa82b51eb3ee532af5eba83c8fb2241867"}
Jul 16 21:09:00.858755 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:00.858370    2530 scope.go:117] "RemoveContainer" containerID="3bbf303a81a76c5fdc7168bdf23e2d694f5d0b585f4b0f56c81bf10ff58e897c"
Jul 16 21:09:00.858755 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:00.858662    2530 scope.go:117] "RemoveContainer" containerID="2ef59f15bf307e4502acd369d53ce8fa82b51eb3ee532af5eba83c8fb2241867"
Jul 16 21:09:00.858906 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:09:00.858887    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz_odh-ai-gateway-infra(4cc473c3-22e1-4d46-b08c-3734f7e9782a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz" podUID="4cc473c3-22e1-4d46-b08c-3734f7e9782a"
Jul 16 21:09:00.889971 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:00.889939    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz"]
Jul 16 21:09:02.004030 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:02.004006    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz"
Jul 16 21:09:02.042149 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:02.042069    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc473c3-22e1-4d46-b08c-3734f7e9782a-maas-api-service-ca\") pod \"4cc473c3-22e1-4d46-b08c-3734f7e9782a\" (UID: \"4cc473c3-22e1-4d46-b08c-3734f7e9782a\") "
Jul 16 21:09:02.042524 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:02.042495    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc473c3-22e1-4d46-b08c-3734f7e9782a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4cc473c3-22e1-4d46-b08c-3734f7e9782a" (UID: "4cc473c3-22e1-4d46-b08c-3734f7e9782a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:09:02.143725 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:02.143628    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cc473c3-22e1-4d46-b08c-3734f7e9782a-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:09:02.870587 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:02.870488    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz" event={"ID":"4cc473c3-22e1-4d46-b08c-3734f7e9782a","Type":"ContainerDied","Data":"42af655f962900ee5f00fc537d7efcd5143419ed40f7e5a9e9f698662e8553b7"}
Jul 16 21:09:02.870587 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:02.870516    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz"
Jul 16 21:09:02.870587 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:02.870563    2530 scope.go:117] "RemoveContainer" containerID="2ef59f15bf307e4502acd369d53ce8fa82b51eb3ee532af5eba83c8fb2241867"
Jul 16 21:09:02.888729 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:02.888701    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz"]
Jul 16 21:09:02.892555 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:02.892532    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-jrdmz"]
Jul 16 21:09:04.507748 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:04.507714    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cc473c3-22e1-4d46-b08c-3734f7e9782a" path="/var/lib/kubelet/pods/4cc473c3-22e1-4d46-b08c-3734f7e9782a/volumes"
Jul 16 21:09:05.502223 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:05.502193    2530 scope.go:117] "RemoveContainer" containerID="46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc"
Jul 16 21:09:05.502398 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:09:05.502364    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:09:06.502163 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:06.502137    2530 scope.go:117] "RemoveContainer" containerID="90f8ee74016e53bcccd1d5cf5ffe6f4cca4777fbd40c42ed6c3d43ce6dbdcf8f"
Jul 16 21:09:06.890694 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:06.890664    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq" event={"ID":"89df93de-82f3-45ec-82aa-1d02deb4cd62","Type":"ContainerStarted","Data":"8c8b54ee2bbed820182bb7702b66e3129db005c54b425f5dbffbfded96132acf"}
Jul 16 21:09:07.896891 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:07.896857    2530 generic.go:356] "Generic (PLEG): container finished" podID="89df93de-82f3-45ec-82aa-1d02deb4cd62" containerID="8c8b54ee2bbed820182bb7702b66e3129db005c54b425f5dbffbfded96132acf" exitCode=7
Jul 16 21:09:07.897313 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:07.896928    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq" event={"ID":"89df93de-82f3-45ec-82aa-1d02deb4cd62","Type":"ContainerDied","Data":"8c8b54ee2bbed820182bb7702b66e3129db005c54b425f5dbffbfded96132acf"}
Jul 16 21:09:07.897313 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:07.897017    2530 scope.go:117] "RemoveContainer" containerID="90f8ee74016e53bcccd1d5cf5ffe6f4cca4777fbd40c42ed6c3d43ce6dbdcf8f"
Jul 16 21:09:07.897453 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:07.897336    2530 scope.go:117] "RemoveContainer" containerID="8c8b54ee2bbed820182bb7702b66e3129db005c54b425f5dbffbfded96132acf"
Jul 16 21:09:07.897586 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:09:07.897566    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq_odh-ai-gateway-infra(89df93de-82f3-45ec-82aa-1d02deb4cd62)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq" podUID="89df93de-82f3-45ec-82aa-1d02deb4cd62"
Jul 16 21:09:07.930004 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:07.929970    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq"]
Jul 16 21:09:09.035846 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:09.035819    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq"
Jul 16 21:09:09.106161 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:09.106126    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89df93de-82f3-45ec-82aa-1d02deb4cd62-maas-api-service-ca\") pod \"89df93de-82f3-45ec-82aa-1d02deb4cd62\" (UID: \"89df93de-82f3-45ec-82aa-1d02deb4cd62\") "
Jul 16 21:09:09.106475 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:09.106453    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89df93de-82f3-45ec-82aa-1d02deb4cd62-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "89df93de-82f3-45ec-82aa-1d02deb4cd62" (UID: "89df93de-82f3-45ec-82aa-1d02deb4cd62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:09:09.206894 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:09.206862    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89df93de-82f3-45ec-82aa-1d02deb4cd62-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:09:09.908360 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:09.908324    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq" event={"ID":"89df93de-82f3-45ec-82aa-1d02deb4cd62","Type":"ContainerDied","Data":"04b63de2308f56f770bfea321f907cb857f6122e74ab564b29397b00404c4474"}
Jul 16 21:09:09.908571 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:09.908365    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq"
Jul 16 21:09:09.908571 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:09.908386    2530 scope.go:117] "RemoveContainer" containerID="8c8b54ee2bbed820182bb7702b66e3129db005c54b425f5dbffbfded96132acf"
Jul 16 21:09:09.931227 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:09.931198    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq"]
Jul 16 21:09:09.935494 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:09.935470    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-2jrvq"]
Jul 16 21:09:10.507998 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:10.507961    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89df93de-82f3-45ec-82aa-1d02deb4cd62" path="/var/lib/kubelet/pods/89df93de-82f3-45ec-82aa-1d02deb4cd62/volumes"
Jul 16 21:09:18.502055 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:18.502022    2530 scope.go:117] "RemoveContainer" containerID="46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc"
Jul 16 21:09:18.502693 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:09:18.502227    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:09:32.505444 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:32.505414    2530 scope.go:117] "RemoveContainer" containerID="46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc"
Jul 16 21:09:32.505902 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:09:32.505618    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:09:40.992880 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:40.992845    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb"]
Jul 16 21:09:40.993270 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:40.993189    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc473c3-22e1-4d46-b08c-3734f7e9782a" containerName="revoke-keys"
Jul 16 21:09:40.993270 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:40.993204    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc473c3-22e1-4d46-b08c-3734f7e9782a" containerName="revoke-keys"
Jul 16 21:09:40.993270 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:40.993217    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89df93de-82f3-45ec-82aa-1d02deb4cd62" containerName="revoke-keys"
Jul 16 21:09:40.993270 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:40.993224    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="89df93de-82f3-45ec-82aa-1d02deb4cd62" containerName="revoke-keys"
Jul 16 21:09:40.993270 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:40.993236    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89df93de-82f3-45ec-82aa-1d02deb4cd62" containerName="revoke-keys"
Jul 16 21:09:40.993270 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:40.993244    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="89df93de-82f3-45ec-82aa-1d02deb4cd62" containerName="revoke-keys"
Jul 16 21:09:40.993270 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:40.993260    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc473c3-22e1-4d46-b08c-3734f7e9782a" containerName="revoke-keys"
Jul 16 21:09:40.993270 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:40.993269    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc473c3-22e1-4d46-b08c-3734f7e9782a" containerName="revoke-keys"
Jul 16 21:09:40.993529 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:40.993277    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89df93de-82f3-45ec-82aa-1d02deb4cd62" containerName="revoke-keys"
Jul 16 21:09:40.993529 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:40.993283    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="89df93de-82f3-45ec-82aa-1d02deb4cd62" containerName="revoke-keys"
Jul 16 21:09:40.993529 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:40.993349    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="89df93de-82f3-45ec-82aa-1d02deb4cd62" containerName="revoke-keys"
Jul 16 21:09:40.993529 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:40.993359    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc473c3-22e1-4d46-b08c-3734f7e9782a" containerName="revoke-keys"
Jul 16 21:09:40.993529 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:40.993365    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc473c3-22e1-4d46-b08c-3734f7e9782a" containerName="revoke-keys"
Jul 16 21:09:40.993529 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:40.993371    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="89df93de-82f3-45ec-82aa-1d02deb4cd62" containerName="revoke-keys"
Jul 16 21:09:40.996028 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:40.996012    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb"
Jul 16 21:09:40.999047 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:40.999028    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:09:41.010338 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:41.010316    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb"]
Jul 16 21:09:41.091232 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:41.091202    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38469a60-6fbf-421b-ba7c-03502a18c6d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-78tqb\" (UID: \"38469a60-6fbf-421b-ba7c-03502a18c6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb"
Jul 16 21:09:41.191886 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:41.191844    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38469a60-6fbf-421b-ba7c-03502a18c6d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-78tqb\" (UID: \"38469a60-6fbf-421b-ba7c-03502a18c6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb"
Jul 16 21:09:41.192520 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:41.192500    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38469a60-6fbf-421b-ba7c-03502a18c6d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-78tqb\" (UID: \"38469a60-6fbf-421b-ba7c-03502a18c6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb"
Jul 16 21:09:41.305455 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:41.305363    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb"
Jul 16 21:09:41.440960 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:41.440937    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb"]
Jul 16 21:09:41.443142 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:09:41.443110    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38469a60_6fbf_421b_ba7c_03502a18c6d8.slice/crio-f229cf3fea6dc1165cbc3f4eae21dd19d1486b209939d24d26c6464e5c245de8 WatchSource:0}: Error finding container f229cf3fea6dc1165cbc3f4eae21dd19d1486b209939d24d26c6464e5c245de8: Status 404 returned error can't find the container with id f229cf3fea6dc1165cbc3f4eae21dd19d1486b209939d24d26c6464e5c245de8
Jul 16 21:09:41.746066 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:41.746040    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br"]
Jul 16 21:09:41.746420 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:41.746406    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc473c3-22e1-4d46-b08c-3734f7e9782a" containerName="revoke-keys"
Jul 16 21:09:41.746465 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:41.746422    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc473c3-22e1-4d46-b08c-3734f7e9782a" containerName="revoke-keys"
Jul 16 21:09:41.746500 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:41.746490    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="89df93de-82f3-45ec-82aa-1d02deb4cd62" containerName="revoke-keys"
Jul 16 21:09:41.746534 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:41.746501    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc473c3-22e1-4d46-b08c-3734f7e9782a" containerName="revoke-keys"
Jul 16 21:09:41.749209 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:41.749194    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br"
Jul 16 21:09:41.758540 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:41.758517    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br"]
Jul 16 21:09:41.798024 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:41.797998    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4deca357-d7f8-487b-837b-c6a2c5a1ed4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br\" (UID: \"4deca357-d7f8-487b-837b-c6a2c5a1ed4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br"
Jul 16 21:09:41.898810 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:41.898734    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4deca357-d7f8-487b-837b-c6a2c5a1ed4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br\" (UID: \"4deca357-d7f8-487b-837b-c6a2c5a1ed4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br"
Jul 16 21:09:41.899315 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:41.899298    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4deca357-d7f8-487b-837b-c6a2c5a1ed4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br\" (UID: \"4deca357-d7f8-487b-837b-c6a2c5a1ed4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br"
Jul 16 21:09:42.055871 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:42.055840    2530 generic.go:356] "Generic (PLEG): container finished" podID="38469a60-6fbf-421b-ba7c-03502a18c6d8" containerID="67d21adbd72bd6df810133c46347fcdf548e3c5eae77eb3107f7e2ea290af256" exitCode=7
Jul 16 21:09:42.056226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:42.055922    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb" event={"ID":"38469a60-6fbf-421b-ba7c-03502a18c6d8","Type":"ContainerDied","Data":"67d21adbd72bd6df810133c46347fcdf548e3c5eae77eb3107f7e2ea290af256"}
Jul 16 21:09:42.056226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:42.055978    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb" event={"ID":"38469a60-6fbf-421b-ba7c-03502a18c6d8","Type":"ContainerStarted","Data":"f229cf3fea6dc1165cbc3f4eae21dd19d1486b209939d24d26c6464e5c245de8"}
Jul 16 21:09:42.056226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:42.056140    2530 scope.go:117] "RemoveContainer" containerID="67d21adbd72bd6df810133c46347fcdf548e3c5eae77eb3107f7e2ea290af256"
Jul 16 21:09:42.059436 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:42.059422    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br"
Jul 16 21:09:42.190424 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:42.190401    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br"]
Jul 16 21:09:42.192510 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:09:42.192488    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4deca357_d7f8_487b_837b_c6a2c5a1ed4f.slice/crio-1289855499c2837befded23da7f8b866efeaca0e443df40c817087e91263763a WatchSource:0}: Error finding container 1289855499c2837befded23da7f8b866efeaca0e443df40c817087e91263763a: Status 404 returned error can't find the container with id 1289855499c2837befded23da7f8b866efeaca0e443df40c817087e91263763a
Jul 16 21:09:43.061762 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:43.061690    2530 generic.go:356] "Generic (PLEG): container finished" podID="38469a60-6fbf-421b-ba7c-03502a18c6d8" containerID="3e40c3471419ee9d44692093078eae785010d40676dbb85ae63e61a63383b878" exitCode=7
Jul 16 21:09:43.062159 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:43.061759    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb" event={"ID":"38469a60-6fbf-421b-ba7c-03502a18c6d8","Type":"ContainerDied","Data":"3e40c3471419ee9d44692093078eae785010d40676dbb85ae63e61a63383b878"}
Jul 16 21:09:43.062159 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:43.061816    2530 scope.go:117] "RemoveContainer" containerID="67d21adbd72bd6df810133c46347fcdf548e3c5eae77eb3107f7e2ea290af256"
Jul 16 21:09:43.062159 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:43.061999    2530 scope.go:117] "RemoveContainer" containerID="3e40c3471419ee9d44692093078eae785010d40676dbb85ae63e61a63383b878"
Jul 16 21:09:43.062327 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:09:43.062222    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-78tqb_odh-ai-gateway-infra(38469a60-6fbf-421b-ba7c-03502a18c6d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb" podUID="38469a60-6fbf-421b-ba7c-03502a18c6d8"
Jul 16 21:09:43.063316 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:43.063297    2530 generic.go:356] "Generic (PLEG): container finished" podID="4deca357-d7f8-487b-837b-c6a2c5a1ed4f" containerID="1170f017a5dac1cedd0ec630e286b7513b94fb0dd60dda13634c26e129c5628f" exitCode=6
Jul 16 21:09:43.063399 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:43.063339    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br" event={"ID":"4deca357-d7f8-487b-837b-c6a2c5a1ed4f","Type":"ContainerDied","Data":"1170f017a5dac1cedd0ec630e286b7513b94fb0dd60dda13634c26e129c5628f"}
Jul 16 21:09:43.063399 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:43.063368    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br" event={"ID":"4deca357-d7f8-487b-837b-c6a2c5a1ed4f","Type":"ContainerStarted","Data":"1289855499c2837befded23da7f8b866efeaca0e443df40c817087e91263763a"}
Jul 16 21:09:43.063551 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:43.063538    2530 scope.go:117] "RemoveContainer" containerID="1170f017a5dac1cedd0ec630e286b7513b94fb0dd60dda13634c26e129c5628f"
Jul 16 21:09:43.502414 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:43.502384    2530 scope.go:117] "RemoveContainer" containerID="46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc"
Jul 16 21:09:44.069283 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:44.069259    2530 scope.go:117] "RemoveContainer" containerID="3e40c3471419ee9d44692093078eae785010d40676dbb85ae63e61a63383b878"
Jul 16 21:09:44.069701 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:09:44.069483    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-78tqb_odh-ai-gateway-infra(38469a60-6fbf-421b-ba7c-03502a18c6d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb" podUID="38469a60-6fbf-421b-ba7c-03502a18c6d8"
Jul 16 21:09:44.070782 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:44.070757    2530 generic.go:356] "Generic (PLEG): container finished" podID="4deca357-d7f8-487b-837b-c6a2c5a1ed4f" containerID="bf70dd0ce43bf803b7bfce6169f91cb93d9255d53fa852162f3928331b3fe923" exitCode=6
Jul 16 21:09:44.070904 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:44.070790    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br" event={"ID":"4deca357-d7f8-487b-837b-c6a2c5a1ed4f","Type":"ContainerDied","Data":"bf70dd0ce43bf803b7bfce6169f91cb93d9255d53fa852162f3928331b3fe923"}
Jul 16 21:09:44.070904 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:44.070856    2530 scope.go:117] "RemoveContainer" containerID="1170f017a5dac1cedd0ec630e286b7513b94fb0dd60dda13634c26e129c5628f"
Jul 16 21:09:44.071017 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:44.071006    2530 scope.go:117] "RemoveContainer" containerID="bf70dd0ce43bf803b7bfce6169f91cb93d9255d53fa852162f3928331b3fe923"
Jul 16 21:09:44.071231 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:09:44.071206    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br_odh-ai-gateway-infra(4deca357-d7f8-487b-837b-c6a2c5a1ed4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br" podUID="4deca357-d7f8-487b-837b-c6a2c5a1ed4f"
Jul 16 21:09:44.072763 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:44.072745    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" event={"ID":"aa609bb2-28bf-44e9-8c1c-3125dea60565","Type":"ContainerStarted","Data":"3b0b1d694444265af2b019b34e7152e025d34a640116ae398edeb06c40360b7a"}
Jul 16 21:09:44.072928 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:44.072916    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:09:45.079305 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:45.079270    2530 generic.go:356] "Generic (PLEG): container finished" podID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerID="3b0b1d694444265af2b019b34e7152e025d34a640116ae398edeb06c40360b7a" exitCode=1
Jul 16 21:09:45.079714 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:45.079362    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" event={"ID":"aa609bb2-28bf-44e9-8c1c-3125dea60565","Type":"ContainerDied","Data":"3b0b1d694444265af2b019b34e7152e025d34a640116ae398edeb06c40360b7a"}
Jul 16 21:09:45.079714 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:45.079451    2530 scope.go:117] "RemoveContainer" containerID="46881e6612d1cd5318129c9e0f946f5335b32fb3b61fc3cd3a382ff9dd060bfc"
Jul 16 21:09:45.079714 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:45.079662    2530 scope.go:117] "RemoveContainer" containerID="3b0b1d694444265af2b019b34e7152e025d34a640116ae398edeb06c40360b7a"
Jul 16 21:09:45.079888 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:09:45.079866    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:09:45.081173 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:45.081157    2530 scope.go:117] "RemoveContainer" containerID="bf70dd0ce43bf803b7bfce6169f91cb93d9255d53fa852162f3928331b3fe923"
Jul 16 21:09:45.081306 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:09:45.081290    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br_odh-ai-gateway-infra(4deca357-d7f8-487b-837b-c6a2c5a1ed4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br" podUID="4deca357-d7f8-487b-837b-c6a2c5a1ed4f"
Jul 16 21:09:46.087309 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:46.087277    2530 scope.go:117] "RemoveContainer" containerID="3b0b1d694444265af2b019b34e7152e025d34a640116ae398edeb06c40360b7a"
Jul 16 21:09:46.087684 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:09:46.087442    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:09:46.143422 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:46.143395    2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:09:47.093297 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:47.093268    2530 scope.go:117] "RemoveContainer" containerID="3b0b1d694444265af2b019b34e7152e025d34a640116ae398edeb06c40360b7a"
Jul 16 21:09:47.093739 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:09:47.093465    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:09:55.502430 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:55.502400    2530 scope.go:117] "RemoveContainer" containerID="bf70dd0ce43bf803b7bfce6169f91cb93d9255d53fa852162f3928331b3fe923"
Jul 16 21:09:56.132951 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:56.132923    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br" event={"ID":"4deca357-d7f8-487b-837b-c6a2c5a1ed4f","Type":"ContainerStarted","Data":"2c05ef08bc1b0186dbc73f817a1ff773c880274eb9c30bcdbb8d70d96e2d0861"}
Jul 16 21:09:56.133253 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:56.133239    2530 scope.go:117] "RemoveContainer" containerID="2c05ef08bc1b0186dbc73f817a1ff773c880274eb9c30bcdbb8d70d96e2d0861"
Jul 16 21:09:56.133460 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:09:56.133443    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br_odh-ai-gateway-infra(4deca357-d7f8-487b-837b-c6a2c5a1ed4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br" podUID="4deca357-d7f8-487b-837b-c6a2c5a1ed4f"
Jul 16 21:09:57.138565 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:57.138532    2530 generic.go:356] "Generic (PLEG): container finished" podID="4deca357-d7f8-487b-837b-c6a2c5a1ed4f" containerID="2c05ef08bc1b0186dbc73f817a1ff773c880274eb9c30bcdbb8d70d96e2d0861" exitCode=6
Jul 16 21:09:57.138974 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:57.138609    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br" event={"ID":"4deca357-d7f8-487b-837b-c6a2c5a1ed4f","Type":"ContainerDied","Data":"2c05ef08bc1b0186dbc73f817a1ff773c880274eb9c30bcdbb8d70d96e2d0861"}
Jul 16 21:09:57.138974 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:57.138665    2530 scope.go:117] "RemoveContainer" containerID="bf70dd0ce43bf803b7bfce6169f91cb93d9255d53fa852162f3928331b3fe923"
Jul 16 21:09:57.138974 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:57.138967    2530 scope.go:117] "RemoveContainer" containerID="2c05ef08bc1b0186dbc73f817a1ff773c880274eb9c30bcdbb8d70d96e2d0861"
Jul 16 21:09:57.139201 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:09:57.139181    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br_odh-ai-gateway-infra(4deca357-d7f8-487b-837b-c6a2c5a1ed4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br" podUID="4deca357-d7f8-487b-837b-c6a2c5a1ed4f"
Jul 16 21:09:57.163962 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:57.163937    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br"]
Jul 16 21:09:57.502091 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:57.502061    2530 scope.go:117] "RemoveContainer" containerID="3e40c3471419ee9d44692093078eae785010d40676dbb85ae63e61a63383b878"
Jul 16 21:09:58.144285 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:58.144255    2530 generic.go:356] "Generic (PLEG): container finished" podID="38469a60-6fbf-421b-ba7c-03502a18c6d8" containerID="2b83a88abf963e6e881be99a302c2c48bb6719c0b7d3dc5ce13264030999aa43" exitCode=7
Jul 16 21:09:58.144751 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:58.144327    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb" event={"ID":"38469a60-6fbf-421b-ba7c-03502a18c6d8","Type":"ContainerDied","Data":"2b83a88abf963e6e881be99a302c2c48bb6719c0b7d3dc5ce13264030999aa43"}
Jul 16 21:09:58.144751 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:58.144379    2530 scope.go:117] "RemoveContainer" containerID="3e40c3471419ee9d44692093078eae785010d40676dbb85ae63e61a63383b878"
Jul 16 21:09:58.144751 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:58.144729    2530 scope.go:117] "RemoveContainer" containerID="2b83a88abf963e6e881be99a302c2c48bb6719c0b7d3dc5ce13264030999aa43"
Jul 16 21:09:58.144989 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:09:58.144964    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-78tqb_odh-ai-gateway-infra(38469a60-6fbf-421b-ba7c-03502a18c6d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb" podUID="38469a60-6fbf-421b-ba7c-03502a18c6d8"
Jul 16 21:09:58.276675 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:58.276652    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br"
Jul 16 21:09:58.328014 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:58.327989    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4deca357-d7f8-487b-837b-c6a2c5a1ed4f-maas-api-service-ca\") pod \"4deca357-d7f8-487b-837b-c6a2c5a1ed4f\" (UID: \"4deca357-d7f8-487b-837b-c6a2c5a1ed4f\") "
Jul 16 21:09:58.328331 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:58.328311    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4deca357-d7f8-487b-837b-c6a2c5a1ed4f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4deca357-d7f8-487b-837b-c6a2c5a1ed4f" (UID: "4deca357-d7f8-487b-837b-c6a2c5a1ed4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:09:58.428932 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:58.428873    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4deca357-d7f8-487b-837b-c6a2c5a1ed4f-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:09:59.152455 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:59.152424    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br" event={"ID":"4deca357-d7f8-487b-837b-c6a2c5a1ed4f","Type":"ContainerDied","Data":"1289855499c2837befded23da7f8b866efeaca0e443df40c817087e91263763a"}
Jul 16 21:09:59.152455 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:59.152449    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br"
Jul 16 21:09:59.152894 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:59.152476    2530 scope.go:117] "RemoveContainer" containerID="2c05ef08bc1b0186dbc73f817a1ff773c880274eb9c30bcdbb8d70d96e2d0861"
Jul 16 21:09:59.171517 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:59.171489    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb"]
Jul 16 21:09:59.172984 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:59.172962    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br"]
Jul 16 21:09:59.175509 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:59.175488    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-mw8br"]
Jul 16 21:09:59.300893 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:59.300868    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb"
Jul 16 21:09:59.438575 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:59.438481    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38469a60-6fbf-421b-ba7c-03502a18c6d8-maas-api-service-ca\") pod \"38469a60-6fbf-421b-ba7c-03502a18c6d8\" (UID: \"38469a60-6fbf-421b-ba7c-03502a18c6d8\") "
Jul 16 21:09:59.438910 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:59.438880    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38469a60-6fbf-421b-ba7c-03502a18c6d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "38469a60-6fbf-421b-ba7c-03502a18c6d8" (UID: "38469a60-6fbf-421b-ba7c-03502a18c6d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:09:59.539159 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:09:59.539128    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38469a60-6fbf-421b-ba7c-03502a18c6d8-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:10:00.157788 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:00.157762    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb"
Jul 16 21:10:00.158264 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:00.157762    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb" event={"ID":"38469a60-6fbf-421b-ba7c-03502a18c6d8","Type":"ContainerDied","Data":"f229cf3fea6dc1165cbc3f4eae21dd19d1486b209939d24d26c6464e5c245de8"}
Jul 16 21:10:00.158264 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:00.157903    2530 scope.go:117] "RemoveContainer" containerID="2b83a88abf963e6e881be99a302c2c48bb6719c0b7d3dc5ce13264030999aa43"
Jul 16 21:10:00.179256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:00.179233    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb"]
Jul 16 21:10:00.185217 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:00.185194    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-78tqb"]
Jul 16 21:10:00.506913 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:00.506880    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38469a60-6fbf-421b-ba7c-03502a18c6d8" path="/var/lib/kubelet/pods/38469a60-6fbf-421b-ba7c-03502a18c6d8/volumes"
Jul 16 21:10:00.507230 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:00.507216    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4deca357-d7f8-487b-837b-c6a2c5a1ed4f" path="/var/lib/kubelet/pods/4deca357-d7f8-487b-837b-c6a2c5a1ed4f/volumes"
Jul 16 21:10:01.502165 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:01.502138    2530 scope.go:117] "RemoveContainer" containerID="3b0b1d694444265af2b019b34e7152e025d34a640116ae398edeb06c40360b7a"
Jul 16 21:10:01.502551 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:01.502310    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-e65739ab-f74c8f957-5slqt_odh-ai-gateway-infra(aa609bb2-28bf-44e9-8c1c-3125dea60565)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565"
Jul 16 21:10:06.406147 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:06.406078    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"]
Jul 16 21:10:06.549714 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:06.549680    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:10:06.696597 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:06.696558    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa609bb2-28bf-44e9-8c1c-3125dea60565-maas-api-tls\") pod \"aa609bb2-28bf-44e9-8c1c-3125dea60565\" (UID: \"aa609bb2-28bf-44e9-8c1c-3125dea60565\") "
Jul 16 21:10:06.696597 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:06.696599    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa609bb2-28bf-44e9-8c1c-3125dea60565-openshift-service-ca\") pod \"aa609bb2-28bf-44e9-8c1c-3125dea60565\" (UID: \"aa609bb2-28bf-44e9-8c1c-3125dea60565\") "
Jul 16 21:10:06.696816 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:06.696636    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjnk2\" (UniqueName: \"kubernetes.io/projected/aa609bb2-28bf-44e9-8c1c-3125dea60565-kube-api-access-rjnk2\") pod \"aa609bb2-28bf-44e9-8c1c-3125dea60565\" (UID: \"aa609bb2-28bf-44e9-8c1c-3125dea60565\") "
Jul 16 21:10:06.697038 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:06.697013    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa609bb2-28bf-44e9-8c1c-3125dea60565-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aa609bb2-28bf-44e9-8c1c-3125dea60565" (UID: "aa609bb2-28bf-44e9-8c1c-3125dea60565"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:10:06.699093 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:06.699062    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa609bb2-28bf-44e9-8c1c-3125dea60565-kube-api-access-rjnk2" (OuterVolumeSpecName: "kube-api-access-rjnk2") pod "aa609bb2-28bf-44e9-8c1c-3125dea60565" (UID: "aa609bb2-28bf-44e9-8c1c-3125dea60565"). InnerVolumeSpecName "kube-api-access-rjnk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:10:06.699182 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:06.699062    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa609bb2-28bf-44e9-8c1c-3125dea60565-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aa609bb2-28bf-44e9-8c1c-3125dea60565" (UID: "aa609bb2-28bf-44e9-8c1c-3125dea60565"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:10:06.797556 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:06.797455    2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjnk2\" (UniqueName: \"kubernetes.io/projected/aa609bb2-28bf-44e9-8c1c-3125dea60565-kube-api-access-rjnk2\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:10:06.797556 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:06.797490    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa609bb2-28bf-44e9-8c1c-3125dea60565-maas-api-tls\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:10:06.797556 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:06.797500    2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa609bb2-28bf-44e9-8c1c-3125dea60565-openshift-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:10:07.191559 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:07.191476    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"
Jul 16 21:10:07.191559 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:07.191498    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt" event={"ID":"aa609bb2-28bf-44e9-8c1c-3125dea60565","Type":"ContainerDied","Data":"cef5ebe63f701f2370faf12eeb4af1a1105cb31e4fc983988e80c2fdbc1fc6ba"}
Jul 16 21:10:07.191559 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:07.191558    2530 scope.go:117] "RemoveContainer" containerID="3b0b1d694444265af2b019b34e7152e025d34a640116ae398edeb06c40360b7a"
Jul 16 21:10:07.214193 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:07.214093    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"]
Jul 16 21:10:07.216968 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:07.216930    2530 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa609bb2_28bf_44e9_8c1c_3125dea60565.slice/crio-cef5ebe63f701f2370faf12eeb4af1a1105cb31e4fc983988e80c2fdbc1fc6ba\": RecentStats: unable to find data in memory cache]"
Jul 16 21:10:07.218245 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:07.218223    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e65739ab-f74c8f957-5slqt"]
Jul 16 21:10:08.511594 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:08.511541    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" path="/var/lib/kubelet/pods/aa609bb2-28bf-44e9-8c1c-3125dea60565/volumes"
Jul 16 21:10:14.624067 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624033    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"]
Jul 16 21:10:14.624486 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624420    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38469a60-6fbf-421b-ba7c-03502a18c6d8" containerName="revoke-keys"
Jul 16 21:10:14.624486 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624436    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="38469a60-6fbf-421b-ba7c-03502a18c6d8" containerName="revoke-keys"
Jul 16 21:10:14.624486 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624454    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38469a60-6fbf-421b-ba7c-03502a18c6d8" containerName="revoke-keys"
Jul 16 21:10:14.624486 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624459    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="38469a60-6fbf-421b-ba7c-03502a18c6d8" containerName="revoke-keys"
Jul 16 21:10:14.624486 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624466    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:14.624486 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624472    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:14.624486 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624479    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4deca357-d7f8-487b-837b-c6a2c5a1ed4f" containerName="revoke-keys"
Jul 16 21:10:14.624486 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624484    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4deca357-d7f8-487b-837b-c6a2c5a1ed4f" containerName="revoke-keys"
Jul 16 21:10:14.624486 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624491    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4deca357-d7f8-487b-837b-c6a2c5a1ed4f" containerName="revoke-keys"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624497    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4deca357-d7f8-487b-837b-c6a2c5a1ed4f" containerName="revoke-keys"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624505    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38469a60-6fbf-421b-ba7c-03502a18c6d8" containerName="revoke-keys"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624510    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="38469a60-6fbf-421b-ba7c-03502a18c6d8" containerName="revoke-keys"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624522    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624531    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624538    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624543    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624549    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624553    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624564    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624569    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624575    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624579    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624632    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624640    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624646    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624651    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="38469a60-6fbf-421b-ba7c-03502a18c6d8" containerName="revoke-keys"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624655    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624660    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4deca357-d7f8-487b-837b-c6a2c5a1ed4f" containerName="revoke-keys"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624666    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4deca357-d7f8-487b-837b-c6a2c5a1ed4f" containerName="revoke-keys"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624671    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4deca357-d7f8-487b-837b-c6a2c5a1ed4f" containerName="revoke-keys"
Jul 16 21:10:14.624779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.624680    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="38469a60-6fbf-421b-ba7c-03502a18c6d8" containerName="revoke-keys"
Jul 16 21:10:14.628938 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.628918    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:10:14.633145 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.633125    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-90b68ec7-serving-cert\""
Jul 16 21:10:14.647430 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.647402    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"]
Jul 16 21:10:14.769121 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.769071    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-openshift-service-ca\") pod \"maas-api-e2e-derive-90b68ec7-7d88b68566-47twm\" (UID: \"b25e21cd-ccbd-4870-99aa-ba1b5dc98083\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:10:14.769281 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.769202    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-maas-api-tls\") pod \"maas-api-e2e-derive-90b68ec7-7d88b68566-47twm\" (UID: \"b25e21cd-ccbd-4870-99aa-ba1b5dc98083\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:10:14.769281 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.769236    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn5c7\" (UniqueName: \"kubernetes.io/projected/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-kube-api-access-jn5c7\") pod \"maas-api-e2e-derive-90b68ec7-7d88b68566-47twm\" (UID: \"b25e21cd-ccbd-4870-99aa-ba1b5dc98083\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:10:14.870096 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.870065    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-maas-api-tls\") pod \"maas-api-e2e-derive-90b68ec7-7d88b68566-47twm\" (UID: \"b25e21cd-ccbd-4870-99aa-ba1b5dc98083\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:10:14.870096 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.870118    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn5c7\" (UniqueName: \"kubernetes.io/projected/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-kube-api-access-jn5c7\") pod \"maas-api-e2e-derive-90b68ec7-7d88b68566-47twm\" (UID: \"b25e21cd-ccbd-4870-99aa-ba1b5dc98083\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:10:14.870366 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.870166    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-openshift-service-ca\") pod \"maas-api-e2e-derive-90b68ec7-7d88b68566-47twm\" (UID: \"b25e21cd-ccbd-4870-99aa-ba1b5dc98083\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:10:14.870366 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:14.870210    2530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-serving-cert: secret "maas-api-e2e-derive-90b68ec7-serving-cert" not found
Jul 16 21:10:14.870366 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:14.870290    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-maas-api-tls podName:b25e21cd-ccbd-4870-99aa-ba1b5dc98083 nodeName:}" failed. No retries permitted until 2026-07-16 21:10:15.370271637 +0000 UTC m=+2253.391859380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-maas-api-tls") pod "maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" (UID: "b25e21cd-ccbd-4870-99aa-ba1b5dc98083") : secret "maas-api-e2e-derive-90b68ec7-serving-cert" not found
Jul 16 21:10:14.870688 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.870672    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-openshift-service-ca\") pod \"maas-api-e2e-derive-90b68ec7-7d88b68566-47twm\" (UID: \"b25e21cd-ccbd-4870-99aa-ba1b5dc98083\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:10:14.879412 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:14.879363    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn5c7\" (UniqueName: \"kubernetes.io/projected/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-kube-api-access-jn5c7\") pod \"maas-api-e2e-derive-90b68ec7-7d88b68566-47twm\" (UID: \"b25e21cd-ccbd-4870-99aa-ba1b5dc98083\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:10:15.374847 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:15.374811    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-maas-api-tls\") pod \"maas-api-e2e-derive-90b68ec7-7d88b68566-47twm\" (UID: \"b25e21cd-ccbd-4870-99aa-ba1b5dc98083\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:10:15.377536 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:15.377511    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-maas-api-tls\") pod \"maas-api-e2e-derive-90b68ec7-7d88b68566-47twm\" (UID: \"b25e21cd-ccbd-4870-99aa-ba1b5dc98083\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:10:15.545583 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:15.545547    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:10:15.695574 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:15.695537    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"]
Jul 16 21:10:15.698036 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:10:15.698009    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25e21cd_ccbd_4870_99aa_ba1b5dc98083.slice/crio-293ecd5094b7d0f8d9714d9a8033e976cae1ad31292077e367e4ab00236b3c47 WatchSource:0}: Error finding container 293ecd5094b7d0f8d9714d9a8033e976cae1ad31292077e367e4ab00236b3c47: Status 404 returned error can't find the container with id 293ecd5094b7d0f8d9714d9a8033e976cae1ad31292077e367e4ab00236b3c47
Jul 16 21:10:15.699401 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:15.699385    2530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:10:16.232923 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:16.232889    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" event={"ID":"b25e21cd-ccbd-4870-99aa-ba1b5dc98083","Type":"ContainerStarted","Data":"293ecd5094b7d0f8d9714d9a8033e976cae1ad31292077e367e4ab00236b3c47"}
Jul 16 21:10:17.237969 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:17.237940    2530 generic.go:356] "Generic (PLEG): container finished" podID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerID="9b9d339ca49d893ea054569c51de92909c169c4c871016b1d0f348f2532ec9be" exitCode=1
Jul 16 21:10:17.238383 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:17.237985    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" event={"ID":"b25e21cd-ccbd-4870-99aa-ba1b5dc98083","Type":"ContainerDied","Data":"9b9d339ca49d893ea054569c51de92909c169c4c871016b1d0f348f2532ec9be"}
Jul 16 21:10:17.238383 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:17.238214    2530 scope.go:117] "RemoveContainer" containerID="9b9d339ca49d893ea054569c51de92909c169c4c871016b1d0f348f2532ec9be"
Jul 16 21:10:18.244399 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:18.244362    2530 generic.go:356] "Generic (PLEG): container finished" podID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerID="a04954ffbe69d019cab6ff8725882362dc631b8cab7a0c20897b2ce17a8998b9" exitCode=1
Jul 16 21:10:18.244786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:18.244437    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" event={"ID":"b25e21cd-ccbd-4870-99aa-ba1b5dc98083","Type":"ContainerDied","Data":"a04954ffbe69d019cab6ff8725882362dc631b8cab7a0c20897b2ce17a8998b9"}
Jul 16 21:10:18.244786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:18.244480    2530 scope.go:117] "RemoveContainer" containerID="9b9d339ca49d893ea054569c51de92909c169c4c871016b1d0f348f2532ec9be"
Jul 16 21:10:18.244786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:18.244657    2530 scope.go:117] "RemoveContainer" containerID="a04954ffbe69d019cab6ff8725882362dc631b8cab7a0c20897b2ce17a8998b9"
Jul 16 21:10:18.244933 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:18.244858    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:10:18.932673 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:18.930279    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd"]
Jul 16 21:10:18.932673 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:18.930966    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:18.932673 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:18.930987    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:18.932673 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:18.931019    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4deca357-d7f8-487b-837b-c6a2c5a1ed4f" containerName="revoke-keys"
Jul 16 21:10:18.932673 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:18.931027    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4deca357-d7f8-487b-837b-c6a2c5a1ed4f" containerName="revoke-keys"
Jul 16 21:10:18.932673 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:18.931214    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:18.932673 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:18.931229    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:18.932673 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:18.931244    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="38469a60-6fbf-421b-ba7c-03502a18c6d8" containerName="revoke-keys"
Jul 16 21:10:18.932673 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:18.931255    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa609bb2-28bf-44e9-8c1c-3125dea60565" containerName="maas-api"
Jul 16 21:10:18.935983 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:18.935959    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd"
Jul 16 21:10:18.938894 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:18.938877    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:10:18.941384 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:18.941359    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd"]
Jul 16 21:10:19.107335 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:19.107288    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1de14b-e01c-44d8-a4a5-d07990aa9905-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd\" (UID: \"9a1de14b-e01c-44d8-a4a5-d07990aa9905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd"
Jul 16 21:10:19.208518 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:19.208488    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1de14b-e01c-44d8-a4a5-d07990aa9905-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd\" (UID: \"9a1de14b-e01c-44d8-a4a5-d07990aa9905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd"
Jul 16 21:10:19.209092 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:19.209072    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1de14b-e01c-44d8-a4a5-d07990aa9905-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd\" (UID: \"9a1de14b-e01c-44d8-a4a5-d07990aa9905\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd"
Jul 16 21:10:19.247378 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:19.247351    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd"
Jul 16 21:10:19.250953 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:19.250932    2530 scope.go:117] "RemoveContainer" containerID="a04954ffbe69d019cab6ff8725882362dc631b8cab7a0c20897b2ce17a8998b9"
Jul 16 21:10:19.251117 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:19.251080    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:10:19.383885 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:19.383694    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd"]
Jul 16 21:10:19.386467 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:10:19.386436    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1de14b_e01c_44d8_a4a5_d07990aa9905.slice/crio-ee215b54fb77704bf805c9c7c8cc3041bc31319fcc9b12e1babe7176e962098e WatchSource:0}: Error finding container ee215b54fb77704bf805c9c7c8cc3041bc31319fcc9b12e1babe7176e962098e: Status 404 returned error can't find the container with id ee215b54fb77704bf805c9c7c8cc3041bc31319fcc9b12e1babe7176e962098e
Jul 16 21:10:20.272011 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:20.271911    2530 generic.go:356] "Generic (PLEG): container finished" podID="9a1de14b-e01c-44d8-a4a5-d07990aa9905" containerID="954e37d7b27467a9f07807ac6c0a6673c4f9dbdc851188a0cb7519f1f8c0f03d" exitCode=7
Jul 16 21:10:20.272011 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:20.271986    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd" event={"ID":"9a1de14b-e01c-44d8-a4a5-d07990aa9905","Type":"ContainerDied","Data":"954e37d7b27467a9f07807ac6c0a6673c4f9dbdc851188a0cb7519f1f8c0f03d"}
Jul 16 21:10:20.272506 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:20.272032    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd" event={"ID":"9a1de14b-e01c-44d8-a4a5-d07990aa9905","Type":"ContainerStarted","Data":"ee215b54fb77704bf805c9c7c8cc3041bc31319fcc9b12e1babe7176e962098e"}
Jul 16 21:10:20.272506 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:20.272335    2530 scope.go:117] "RemoveContainer" containerID="954e37d7b27467a9f07807ac6c0a6673c4f9dbdc851188a0cb7519f1f8c0f03d"
Jul 16 21:10:20.546483 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:20.546458    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:10:20.546849 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:20.546833    2530 scope.go:117] "RemoveContainer" containerID="a04954ffbe69d019cab6ff8725882362dc631b8cab7a0c20897b2ce17a8998b9"
Jul 16 21:10:20.547025 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:20.547007    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:10:21.278053 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:21.278020    2530 generic.go:356] "Generic (PLEG): container finished" podID="9a1de14b-e01c-44d8-a4a5-d07990aa9905" containerID="4a80d04927ba8e202257baa6cb0773ccc770a8a1d09a3249d78878aa920498b3" exitCode=7
Jul 16 21:10:21.278542 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:21.278137    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd" event={"ID":"9a1de14b-e01c-44d8-a4a5-d07990aa9905","Type":"ContainerDied","Data":"4a80d04927ba8e202257baa6cb0773ccc770a8a1d09a3249d78878aa920498b3"}
Jul 16 21:10:21.278542 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:21.278198    2530 scope.go:117] "RemoveContainer" containerID="954e37d7b27467a9f07807ac6c0a6673c4f9dbdc851188a0cb7519f1f8c0f03d"
Jul 16 21:10:21.278542 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:21.278333    2530 scope.go:117] "RemoveContainer" containerID="4a80d04927ba8e202257baa6cb0773ccc770a8a1d09a3249d78878aa920498b3"
Jul 16 21:10:21.278674 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:21.278541    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd_odh-ai-gateway-infra(9a1de14b-e01c-44d8-a4a5-d07990aa9905)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd" podUID="9a1de14b-e01c-44d8-a4a5-d07990aa9905"
Jul 16 21:10:22.283948 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:22.283925    2530 scope.go:117] "RemoveContainer" containerID="4a80d04927ba8e202257baa6cb0773ccc770a8a1d09a3249d78878aa920498b3"
Jul 16 21:10:22.284333 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:22.284095    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd_odh-ai-gateway-infra(9a1de14b-e01c-44d8-a4a5-d07990aa9905)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd" podUID="9a1de14b-e01c-44d8-a4a5-d07990aa9905"
Jul 16 21:10:23.699292 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:23.699252    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48"]
Jul 16 21:10:23.704902 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:23.704867    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48"
Jul 16 21:10:23.711369 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:23.711342    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48"]
Jul 16 21:10:23.749899 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:23.749868    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aeebad2-8bf1-4452-b46f-3a408311d112-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48\" (UID: \"8aeebad2-8bf1-4452-b46f-3a408311d112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48"
Jul 16 21:10:23.851006 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:23.850973    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aeebad2-8bf1-4452-b46f-3a408311d112-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48\" (UID: \"8aeebad2-8bf1-4452-b46f-3a408311d112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48"
Jul 16 21:10:23.851602 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:23.851582    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aeebad2-8bf1-4452-b46f-3a408311d112-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48\" (UID: \"8aeebad2-8bf1-4452-b46f-3a408311d112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48"
Jul 16 21:10:24.016779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:24.016750    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48"
Jul 16 21:10:24.150163 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:24.150138    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48"]
Jul 16 21:10:24.151777 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:10:24.151750    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aeebad2_8bf1_4452_b46f_3a408311d112.slice/crio-e1691f5766160792162e396f40fee1a247b96d746b75cab2a1928dd689bb9875 WatchSource:0}: Error finding container e1691f5766160792162e396f40fee1a247b96d746b75cab2a1928dd689bb9875: Status 404 returned error can't find the container with id e1691f5766160792162e396f40fee1a247b96d746b75cab2a1928dd689bb9875
Jul 16 21:10:24.294589 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:24.294508    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48" event={"ID":"8aeebad2-8bf1-4452-b46f-3a408311d112","Type":"ContainerStarted","Data":"e1691f5766160792162e396f40fee1a247b96d746b75cab2a1928dd689bb9875"}
Jul 16 21:10:25.299921 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:25.299881    2530 generic.go:356] "Generic (PLEG): container finished" podID="8aeebad2-8bf1-4452-b46f-3a408311d112" containerID="c37e286601d77eb5eab579b59fc2f705a7c739f5acce6cee8ea39db87656e04e" exitCode=6
Jul 16 21:10:25.300294 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:25.299957    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48" event={"ID":"8aeebad2-8bf1-4452-b46f-3a408311d112","Type":"ContainerDied","Data":"c37e286601d77eb5eab579b59fc2f705a7c739f5acce6cee8ea39db87656e04e"}
Jul 16 21:10:25.300294 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:25.300203    2530 scope.go:117] "RemoveContainer" containerID="c37e286601d77eb5eab579b59fc2f705a7c739f5acce6cee8ea39db87656e04e"
Jul 16 21:10:25.545816 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:25.545790    2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:10:25.546204 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:25.546191    2530 scope.go:117] "RemoveContainer" containerID="a04954ffbe69d019cab6ff8725882362dc631b8cab7a0c20897b2ce17a8998b9"
Jul 16 21:10:25.546359 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:25.546342    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:10:26.305853 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:26.305763    2530 generic.go:356] "Generic (PLEG): container finished" podID="8aeebad2-8bf1-4452-b46f-3a408311d112" containerID="518b3913aeb85635a63353c7a2fc826e5a9d96f327dd3aea1a34a40253ee384a" exitCode=6
Jul 16 21:10:26.306318 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:26.305861    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48" event={"ID":"8aeebad2-8bf1-4452-b46f-3a408311d112","Type":"ContainerDied","Data":"518b3913aeb85635a63353c7a2fc826e5a9d96f327dd3aea1a34a40253ee384a"}
Jul 16 21:10:26.306318 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:26.305931    2530 scope.go:117] "RemoveContainer" containerID="c37e286601d77eb5eab579b59fc2f705a7c739f5acce6cee8ea39db87656e04e"
Jul 16 21:10:26.306318 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:26.306034    2530 scope.go:117] "RemoveContainer" containerID="518b3913aeb85635a63353c7a2fc826e5a9d96f327dd3aea1a34a40253ee384a"
Jul 16 21:10:26.306318 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:26.306289    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48_odh-ai-gateway-infra(8aeebad2-8bf1-4452-b46f-3a408311d112)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48" podUID="8aeebad2-8bf1-4452-b46f-3a408311d112"
Jul 16 21:10:27.312548 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:27.312522    2530 scope.go:117] "RemoveContainer" containerID="518b3913aeb85635a63353c7a2fc826e5a9d96f327dd3aea1a34a40253ee384a"
Jul 16 21:10:27.312982 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:27.312736    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48_odh-ai-gateway-infra(8aeebad2-8bf1-4452-b46f-3a408311d112)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48" podUID="8aeebad2-8bf1-4452-b46f-3a408311d112"
Jul 16 21:10:31.025442 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:31.025398    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh"]
Jul 16 21:10:31.030065 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:31.030042    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh"
Jul 16 21:10:31.039334 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:31.039297    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh"]
Jul 16 21:10:31.114488 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:31.114449    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85bbe7c3-c8c6-429d-b963-b0a0c771de16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh\" (UID: \"85bbe7c3-c8c6-429d-b963-b0a0c771de16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh"
Jul 16 21:10:31.215634 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:31.215600    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85bbe7c3-c8c6-429d-b963-b0a0c771de16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh\" (UID: \"85bbe7c3-c8c6-429d-b963-b0a0c771de16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh"
Jul 16 21:10:31.216322 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:31.216303    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85bbe7c3-c8c6-429d-b963-b0a0c771de16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh\" (UID: \"85bbe7c3-c8c6-429d-b963-b0a0c771de16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh"
Jul 16 21:10:31.341809 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:31.341757    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh"
Jul 16 21:10:31.485961 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:31.485879    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh"]
Jul 16 21:10:31.488913 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:10:31.488881    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bbe7c3_c8c6_429d_b963_b0a0c771de16.slice/crio-98a3ea14df265ae040197eba1f9ce7176e9793007cf690bb335bd86dd013d1a4 WatchSource:0}: Error finding container 98a3ea14df265ae040197eba1f9ce7176e9793007cf690bb335bd86dd013d1a4: Status 404 returned error can't find the container with id 98a3ea14df265ae040197eba1f9ce7176e9793007cf690bb335bd86dd013d1a4
Jul 16 21:10:31.775160 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:31.775058    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx"]
Jul 16 21:10:31.779790 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:31.779767    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx"
Jul 16 21:10:31.787443 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:31.787169    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx"]
Jul 16 21:10:31.820432 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:31.820401    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff485212-ff1b-4df3-91fd-faf80e501f77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx\" (UID: \"ff485212-ff1b-4df3-91fd-faf80e501f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx"
Jul 16 21:10:31.921457 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:31.921422    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff485212-ff1b-4df3-91fd-faf80e501f77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx\" (UID: \"ff485212-ff1b-4df3-91fd-faf80e501f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx"
Jul 16 21:10:31.922074 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:31.922053    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff485212-ff1b-4df3-91fd-faf80e501f77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx\" (UID: \"ff485212-ff1b-4df3-91fd-faf80e501f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx"
Jul 16 21:10:32.092459 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:32.092359    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx"
Jul 16 21:10:32.232413 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:32.232385    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx"]
Jul 16 21:10:32.234739 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:10:32.234711    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff485212_ff1b_4df3_91fd_faf80e501f77.slice/crio-6943ae6f0bce3b0e502a2c145f43ad7e291c121c5f81a287195aad3cdaf823ca WatchSource:0}: Error finding container 6943ae6f0bce3b0e502a2c145f43ad7e291c121c5f81a287195aad3cdaf823ca: Status 404 returned error can't find the container with id 6943ae6f0bce3b0e502a2c145f43ad7e291c121c5f81a287195aad3cdaf823ca
Jul 16 21:10:32.336507 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:32.336458    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx" event={"ID":"ff485212-ff1b-4df3-91fd-faf80e501f77","Type":"ContainerStarted","Data":"6943ae6f0bce3b0e502a2c145f43ad7e291c121c5f81a287195aad3cdaf823ca"}
Jul 16 21:10:32.337897 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:32.337866    2530 generic.go:356] "Generic (PLEG): container finished" podID="85bbe7c3-c8c6-429d-b963-b0a0c771de16" containerID="139a665c2e79c73dd225001d605e07e45cccf22896acf23e2a1e459e9cc1b924" exitCode=6
Jul 16 21:10:32.338021 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:32.337955    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh" event={"ID":"85bbe7c3-c8c6-429d-b963-b0a0c771de16","Type":"ContainerDied","Data":"139a665c2e79c73dd225001d605e07e45cccf22896acf23e2a1e459e9cc1b924"}
Jul 16 21:10:32.338021 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:32.338004    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh" event={"ID":"85bbe7c3-c8c6-429d-b963-b0a0c771de16","Type":"ContainerStarted","Data":"98a3ea14df265ae040197eba1f9ce7176e9793007cf690bb335bd86dd013d1a4"}
Jul 16 21:10:32.338204 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:32.338189    2530 scope.go:117] "RemoveContainer" containerID="139a665c2e79c73dd225001d605e07e45cccf22896acf23e2a1e459e9cc1b924"
Jul 16 21:10:33.344265 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:33.344227    2530 generic.go:356] "Generic (PLEG): container finished" podID="ff485212-ff1b-4df3-91fd-faf80e501f77" containerID="46254193035e4015e27d681cfae94ca80af39ab1e24777a5b3d665fcf1e8b284" exitCode=6
Jul 16 21:10:33.344783 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:33.344317    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx" event={"ID":"ff485212-ff1b-4df3-91fd-faf80e501f77","Type":"ContainerDied","Data":"46254193035e4015e27d681cfae94ca80af39ab1e24777a5b3d665fcf1e8b284"}
Jul 16 21:10:33.344783 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:33.344564    2530 scope.go:117] "RemoveContainer" containerID="46254193035e4015e27d681cfae94ca80af39ab1e24777a5b3d665fcf1e8b284"
Jul 16 21:10:33.346130 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:33.346088    2530 generic.go:356] "Generic (PLEG): container finished" podID="85bbe7c3-c8c6-429d-b963-b0a0c771de16" containerID="da836d12935aff5e53c8fe242638cb994964ce704c80f967176f3997c6fdbe36" exitCode=6
Jul 16 21:10:33.346243 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:33.346132    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh" event={"ID":"85bbe7c3-c8c6-429d-b963-b0a0c771de16","Type":"ContainerDied","Data":"da836d12935aff5e53c8fe242638cb994964ce704c80f967176f3997c6fdbe36"}
Jul 16 21:10:33.346243 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:33.346195    2530 scope.go:117] "RemoveContainer" containerID="139a665c2e79c73dd225001d605e07e45cccf22896acf23e2a1e459e9cc1b924"
Jul 16 21:10:33.346415 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:33.346401    2530 scope.go:117] "RemoveContainer" containerID="da836d12935aff5e53c8fe242638cb994964ce704c80f967176f3997c6fdbe36"
Jul 16 21:10:33.346620 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:33.346595    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh_odh-ai-gateway-infra(85bbe7c3-c8c6-429d-b963-b0a0c771de16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh" podUID="85bbe7c3-c8c6-429d-b963-b0a0c771de16"
Jul 16 21:10:34.354164 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:34.354059    2530 scope.go:117] "RemoveContainer" containerID="da836d12935aff5e53c8fe242638cb994964ce704c80f967176f3997c6fdbe36"
Jul 16 21:10:34.354558 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:34.354331    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh_odh-ai-gateway-infra(85bbe7c3-c8c6-429d-b963-b0a0c771de16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh" podUID="85bbe7c3-c8c6-429d-b963-b0a0c771de16"
Jul 16 21:10:34.355643 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:34.355618    2530 generic.go:356] "Generic (PLEG): container finished" podID="ff485212-ff1b-4df3-91fd-faf80e501f77" containerID="6337ac364c97ce47731848f28b177f7ab07c561cfa750084d2c66bdd8fe9b122" exitCode=6
Jul 16 21:10:34.355764 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:34.355688    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx" event={"ID":"ff485212-ff1b-4df3-91fd-faf80e501f77","Type":"ContainerDied","Data":"6337ac364c97ce47731848f28b177f7ab07c561cfa750084d2c66bdd8fe9b122"}
Jul 16 21:10:34.355764 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:34.355733    2530 scope.go:117] "RemoveContainer" containerID="46254193035e4015e27d681cfae94ca80af39ab1e24777a5b3d665fcf1e8b284"
Jul 16 21:10:34.355886 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:34.355870    2530 scope.go:117] "RemoveContainer" containerID="6337ac364c97ce47731848f28b177f7ab07c561cfa750084d2c66bdd8fe9b122"
Jul 16 21:10:34.356122 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:34.356079    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx_odh-ai-gateway-infra(ff485212-ff1b-4df3-91fd-faf80e501f77)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx" podUID="ff485212-ff1b-4df3-91fd-faf80e501f77"
Jul 16 21:10:35.362344 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:35.362317    2530 scope.go:117] "RemoveContainer" containerID="6337ac364c97ce47731848f28b177f7ab07c561cfa750084d2c66bdd8fe9b122"
Jul 16 21:10:35.362728 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:35.362498    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx_odh-ai-gateway-infra(ff485212-ff1b-4df3-91fd-faf80e501f77)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx" podUID="ff485212-ff1b-4df3-91fd-faf80e501f77"
Jul 16 21:10:37.502888 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:37.502852    2530 scope.go:117] "RemoveContainer" containerID="4a80d04927ba8e202257baa6cb0773ccc770a8a1d09a3249d78878aa920498b3"
Jul 16 21:10:38.378666 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:38.378575    2530 generic.go:356] "Generic (PLEG): container finished" podID="9a1de14b-e01c-44d8-a4a5-d07990aa9905" containerID="b3918b072761fc4aa061676b6150210c902d2da67d9eb0113c8ba3553637e690" exitCode=7
Jul 16 21:10:38.378666 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:38.378650    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd" event={"ID":"9a1de14b-e01c-44d8-a4a5-d07990aa9905","Type":"ContainerDied","Data":"b3918b072761fc4aa061676b6150210c902d2da67d9eb0113c8ba3553637e690"}
Jul 16 21:10:38.378879 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:38.378719    2530 scope.go:117] "RemoveContainer" containerID="4a80d04927ba8e202257baa6cb0773ccc770a8a1d09a3249d78878aa920498b3"
Jul 16 21:10:38.379081 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:38.379058    2530 scope.go:117] "RemoveContainer" containerID="b3918b072761fc4aa061676b6150210c902d2da67d9eb0113c8ba3553637e690"
Jul 16 21:10:38.379358 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:38.379337    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd_odh-ai-gateway-infra(9a1de14b-e01c-44d8-a4a5-d07990aa9905)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd" podUID="9a1de14b-e01c-44d8-a4a5-d07990aa9905"
Jul 16 21:10:39.405782 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:39.405733    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd"]
Jul 16 21:10:39.502871 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:39.502835    2530 scope.go:117] "RemoveContainer" containerID="518b3913aeb85635a63353c7a2fc826e5a9d96f327dd3aea1a34a40253ee384a"
Jul 16 21:10:39.550495 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:39.550458    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd"
Jul 16 21:10:39.596081 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:39.596036    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1de14b-e01c-44d8-a4a5-d07990aa9905-maas-api-service-ca\") pod \"9a1de14b-e01c-44d8-a4a5-d07990aa9905\" (UID: \"9a1de14b-e01c-44d8-a4a5-d07990aa9905\") "
Jul 16 21:10:39.596463 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:39.596438    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1de14b-e01c-44d8-a4a5-d07990aa9905-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a1de14b-e01c-44d8-a4a5-d07990aa9905" (UID: "9a1de14b-e01c-44d8-a4a5-d07990aa9905"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:10:39.697535 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:39.697433    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1de14b-e01c-44d8-a4a5-d07990aa9905-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:10:40.391807 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:40.391716    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd"
Jul 16 21:10:40.391807 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:40.391755    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd" event={"ID":"9a1de14b-e01c-44d8-a4a5-d07990aa9905","Type":"ContainerDied","Data":"ee215b54fb77704bf805c9c7c8cc3041bc31319fcc9b12e1babe7176e962098e"}
Jul 16 21:10:40.392052 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:40.391839    2530 scope.go:117] "RemoveContainer" containerID="b3918b072761fc4aa061676b6150210c902d2da67d9eb0113c8ba3553637e690"
Jul 16 21:10:40.393915 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:40.393883    2530 generic.go:356] "Generic (PLEG): container finished" podID="8aeebad2-8bf1-4452-b46f-3a408311d112" containerID="ec1036a9c561278142496ea8b407d7bd24a8de41bddf099e907497fb952c30d8" exitCode=6
Jul 16 21:10:40.394051 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:40.393951    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48" event={"ID":"8aeebad2-8bf1-4452-b46f-3a408311d112","Type":"ContainerDied","Data":"ec1036a9c561278142496ea8b407d7bd24a8de41bddf099e907497fb952c30d8"}
Jul 16 21:10:40.394293 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:40.394276    2530 scope.go:117] "RemoveContainer" containerID="ec1036a9c561278142496ea8b407d7bd24a8de41bddf099e907497fb952c30d8"
Jul 16 21:10:40.394538 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:40.394516    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48_odh-ai-gateway-infra(8aeebad2-8bf1-4452-b46f-3a408311d112)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48" podUID="8aeebad2-8bf1-4452-b46f-3a408311d112"
Jul 16 21:10:40.401525 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:40.401504    2530 scope.go:117] "RemoveContainer" containerID="518b3913aeb85635a63353c7a2fc826e5a9d96f327dd3aea1a34a40253ee384a"
Jul 16 21:10:40.423614 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:40.423583    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd"]
Jul 16 21:10:40.427866 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:40.427837    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-6ntcd"]
Jul 16 21:10:40.502348 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:40.502314    2530 scope.go:117] "RemoveContainer" containerID="a04954ffbe69d019cab6ff8725882362dc631b8cab7a0c20897b2ce17a8998b9"
Jul 16 21:10:40.508944 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:40.508915    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a1de14b-e01c-44d8-a4a5-d07990aa9905" path="/var/lib/kubelet/pods/9a1de14b-e01c-44d8-a4a5-d07990aa9905/volumes"
Jul 16 21:10:41.402924 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:41.402888    2530 generic.go:356] "Generic (PLEG): container finished" podID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerID="9040b029295c4d98cfbcbb0efcd3916cfcbdbb24951799d889d8bf69b0e98d80" exitCode=1
Jul 16 21:10:41.403123 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:41.402956    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" event={"ID":"b25e21cd-ccbd-4870-99aa-ba1b5dc98083","Type":"ContainerDied","Data":"9040b029295c4d98cfbcbb0efcd3916cfcbdbb24951799d889d8bf69b0e98d80"}
Jul 16 21:10:41.403123 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:41.403013    2530 scope.go:117] "RemoveContainer" containerID="a04954ffbe69d019cab6ff8725882362dc631b8cab7a0c20897b2ce17a8998b9"
Jul 16 21:10:41.403367 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:41.403352    2530 scope.go:117] "RemoveContainer" containerID="9040b029295c4d98cfbcbb0efcd3916cfcbdbb24951799d889d8bf69b0e98d80"
Jul 16 21:10:41.403573 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:41.403554    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:10:41.420386 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:41.420355    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48"]
Jul 16 21:10:41.545974 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:41.545945    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48"
Jul 16 21:10:41.616235 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:41.616147    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aeebad2-8bf1-4452-b46f-3a408311d112-maas-api-service-ca\") pod \"8aeebad2-8bf1-4452-b46f-3a408311d112\" (UID: \"8aeebad2-8bf1-4452-b46f-3a408311d112\") "
Jul 16 21:10:41.616534 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:41.616510    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aeebad2-8bf1-4452-b46f-3a408311d112-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8aeebad2-8bf1-4452-b46f-3a408311d112" (UID: "8aeebad2-8bf1-4452-b46f-3a408311d112"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:10:41.717639 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:41.717587    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aeebad2-8bf1-4452-b46f-3a408311d112-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:10:42.409287 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:42.409250    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48"
Jul 16 21:10:42.409580 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:42.409284    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48" event={"ID":"8aeebad2-8bf1-4452-b46f-3a408311d112","Type":"ContainerDied","Data":"e1691f5766160792162e396f40fee1a247b96d746b75cab2a1928dd689bb9875"}
Jul 16 21:10:42.409580 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:42.409365    2530 scope.go:117] "RemoveContainer" containerID="ec1036a9c561278142496ea8b407d7bd24a8de41bddf099e907497fb952c30d8"
Jul 16 21:10:42.432992 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:42.432956    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48"]
Jul 16 21:10:42.439425 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:42.439388    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-8tj48"]
Jul 16 21:10:42.507477 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:42.507434    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aeebad2-8bf1-4452-b46f-3a408311d112" path="/var/lib/kubelet/pods/8aeebad2-8bf1-4452-b46f-3a408311d112/volumes"
Jul 16 21:10:45.546678 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:45.546641    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:10:45.547092 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:45.546701    2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:10:45.547092 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:45.547022    2530 scope.go:117] "RemoveContainer" containerID="9040b029295c4d98cfbcbb0efcd3916cfcbdbb24951799d889d8bf69b0e98d80"
Jul 16 21:10:45.547229 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:45.547208    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:10:47.502429 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:47.502391    2530 scope.go:117] "RemoveContainer" containerID="6337ac364c97ce47731848f28b177f7ab07c561cfa750084d2c66bdd8fe9b122"
Jul 16 21:10:48.440330 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:48.440298    2530 generic.go:356] "Generic (PLEG): container finished" podID="ff485212-ff1b-4df3-91fd-faf80e501f77" containerID="2eff77bdb95f1463dcda990aeb848ea98da238332c220ffde23f9f64a6cdc54b" exitCode=6
Jul 16 21:10:48.440527 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:48.440367    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx" event={"ID":"ff485212-ff1b-4df3-91fd-faf80e501f77","Type":"ContainerDied","Data":"2eff77bdb95f1463dcda990aeb848ea98da238332c220ffde23f9f64a6cdc54b"}
Jul 16 21:10:48.440527 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:48.440417    2530 scope.go:117] "RemoveContainer" containerID="6337ac364c97ce47731848f28b177f7ab07c561cfa750084d2c66bdd8fe9b122"
Jul 16 21:10:48.440730 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:48.440703    2530 scope.go:117] "RemoveContainer" containerID="2eff77bdb95f1463dcda990aeb848ea98da238332c220ffde23f9f64a6cdc54b"
Jul 16 21:10:48.440944 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:48.440920    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx_odh-ai-gateway-infra(ff485212-ff1b-4df3-91fd-faf80e501f77)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx" podUID="ff485212-ff1b-4df3-91fd-faf80e501f77"
Jul 16 21:10:49.465068 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:49.465023    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx"]
Jul 16 21:10:49.502558 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:49.502526    2530 scope.go:117] "RemoveContainer" containerID="da836d12935aff5e53c8fe242638cb994964ce704c80f967176f3997c6fdbe36"
Jul 16 21:10:49.607069 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:49.607035    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx"
Jul 16 21:10:49.694272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:49.694220    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff485212-ff1b-4df3-91fd-faf80e501f77-maas-api-service-ca\") pod \"ff485212-ff1b-4df3-91fd-faf80e501f77\" (UID: \"ff485212-ff1b-4df3-91fd-faf80e501f77\") "
Jul 16 21:10:49.694639 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:49.694616    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff485212-ff1b-4df3-91fd-faf80e501f77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff485212-ff1b-4df3-91fd-faf80e501f77" (UID: "ff485212-ff1b-4df3-91fd-faf80e501f77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:10:49.795581 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:49.795531    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff485212-ff1b-4df3-91fd-faf80e501f77-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:10:50.453435 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:50.453394    2530 generic.go:356] "Generic (PLEG): container finished" podID="85bbe7c3-c8c6-429d-b963-b0a0c771de16" containerID="2b5429aebba38f3604534f61f4bfe100659d913b2053f65d8c03075b8ed41efd" exitCode=6
Jul 16 21:10:50.453598 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:50.453466    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh" event={"ID":"85bbe7c3-c8c6-429d-b963-b0a0c771de16","Type":"ContainerDied","Data":"2b5429aebba38f3604534f61f4bfe100659d913b2053f65d8c03075b8ed41efd"}
Jul 16 21:10:50.453598 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:50.453562    2530 scope.go:117] "RemoveContainer" containerID="da836d12935aff5e53c8fe242638cb994964ce704c80f967176f3997c6fdbe36"
Jul 16 21:10:50.453901 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:50.453880    2530 scope.go:117] "RemoveContainer" containerID="2b5429aebba38f3604534f61f4bfe100659d913b2053f65d8c03075b8ed41efd"
Jul 16 21:10:50.454190 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:50.454165    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh_odh-ai-gateway-infra(85bbe7c3-c8c6-429d-b963-b0a0c771de16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh" podUID="85bbe7c3-c8c6-429d-b963-b0a0c771de16"
Jul 16 21:10:50.454942 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:50.454921    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx" event={"ID":"ff485212-ff1b-4df3-91fd-faf80e501f77","Type":"ContainerDied","Data":"6943ae6f0bce3b0e502a2c145f43ad7e291c121c5f81a287195aad3cdaf823ca"}
Jul 16 21:10:50.455016 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:50.454969    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx"
Jul 16 21:10:50.466910 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:50.466762    2530 scope.go:117] "RemoveContainer" containerID="2eff77bdb95f1463dcda990aeb848ea98da238332c220ffde23f9f64a6cdc54b"
Jul 16 21:10:50.483320 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:50.483284    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx"]
Jul 16 21:10:50.486133 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:50.486084    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-wnllx"]
Jul 16 21:10:50.508810 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:50.508767    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff485212-ff1b-4df3-91fd-faf80e501f77" path="/var/lib/kubelet/pods/ff485212-ff1b-4df3-91fd-faf80e501f77/volumes"
Jul 16 21:10:51.477815 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:51.477775    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh"]
Jul 16 21:10:51.619721 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:51.619688    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh"
Jul 16 21:10:51.714036 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:51.713999    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85bbe7c3-c8c6-429d-b963-b0a0c771de16-maas-api-service-ca\") pod \"85bbe7c3-c8c6-429d-b963-b0a0c771de16\" (UID: \"85bbe7c3-c8c6-429d-b963-b0a0c771de16\") "
Jul 16 21:10:51.714445 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:51.714422    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85bbe7c3-c8c6-429d-b963-b0a0c771de16-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "85bbe7c3-c8c6-429d-b963-b0a0c771de16" (UID: "85bbe7c3-c8c6-429d-b963-b0a0c771de16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:10:51.815042 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:51.814945    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85bbe7c3-c8c6-429d-b963-b0a0c771de16-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:10:52.468289 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:52.468259    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh"
Jul 16 21:10:52.468467 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:52.468264    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh" event={"ID":"85bbe7c3-c8c6-429d-b963-b0a0c771de16","Type":"ContainerDied","Data":"98a3ea14df265ae040197eba1f9ce7176e9793007cf690bb335bd86dd013d1a4"}
Jul 16 21:10:52.468467 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:52.468401    2530 scope.go:117] "RemoveContainer" containerID="2b5429aebba38f3604534f61f4bfe100659d913b2053f65d8c03075b8ed41efd"
Jul 16 21:10:52.491436 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:52.491399    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh"]
Jul 16 21:10:52.500229 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:52.500194    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-cs8zh"]
Jul 16 21:10:52.507998 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:52.507957    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85bbe7c3-c8c6-429d-b963-b0a0c771de16" path="/var/lib/kubelet/pods/85bbe7c3-c8c6-429d-b963-b0a0c771de16/volumes"
Jul 16 21:10:58.502422 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:10:58.502387    2530 scope.go:117] "RemoveContainer" containerID="9040b029295c4d98cfbcbb0efcd3916cfcbdbb24951799d889d8bf69b0e98d80"
Jul 16 21:10:58.503020 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:10:58.502607    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:11:13.502559 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:13.502527    2530 scope.go:117] "RemoveContainer" containerID="9040b029295c4d98cfbcbb0efcd3916cfcbdbb24951799d889d8bf69b0e98d80"
Jul 16 21:11:14.569925 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:14.569843    2530 generic.go:356] "Generic (PLEG): container finished" podID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerID="ab355bb287b35f902a1f2d710b86425e94344e4566c31d5accd58f25dba6aff5" exitCode=1
Jul 16 21:11:14.570330 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:14.569922    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" event={"ID":"b25e21cd-ccbd-4870-99aa-ba1b5dc98083","Type":"ContainerDied","Data":"ab355bb287b35f902a1f2d710b86425e94344e4566c31d5accd58f25dba6aff5"}
Jul 16 21:11:14.570330 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:14.569996    2530 scope.go:117] "RemoveContainer" containerID="9040b029295c4d98cfbcbb0efcd3916cfcbdbb24951799d889d8bf69b0e98d80"
Jul 16 21:11:14.570442 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:14.570426    2530 scope.go:117] "RemoveContainer" containerID="ab355bb287b35f902a1f2d710b86425e94344e4566c31d5accd58f25dba6aff5"
Jul 16 21:11:14.570656 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:11:14.570635    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:11:15.546606 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:15.546560    2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:11:15.546802 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:15.546655    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:11:15.576269 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:15.576240    2530 scope.go:117] "RemoveContainer" containerID="ab355bb287b35f902a1f2d710b86425e94344e4566c31d5accd58f25dba6aff5"
Jul 16 21:11:15.576645 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:11:15.576414    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:11:18.968874 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.968830    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff"]
Jul 16 21:11:18.969343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969183    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a1de14b-e01c-44d8-a4a5-d07990aa9905" containerName="revoke-keys"
Jul 16 21:11:18.969343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969195    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a1de14b-e01c-44d8-a4a5-d07990aa9905" containerName="revoke-keys"
Jul 16 21:11:18.969343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969205    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a1de14b-e01c-44d8-a4a5-d07990aa9905" containerName="revoke-keys"
Jul 16 21:11:18.969343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969210    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a1de14b-e01c-44d8-a4a5-d07990aa9905" containerName="revoke-keys"
Jul 16 21:11:18.969343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969216    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aeebad2-8bf1-4452-b46f-3a408311d112" containerName="revoke-keys"
Jul 16 21:11:18.969343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969221    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aeebad2-8bf1-4452-b46f-3a408311d112" containerName="revoke-keys"
Jul 16 21:11:18.969343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969228    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff485212-ff1b-4df3-91fd-faf80e501f77" containerName="revoke-keys"
Jul 16 21:11:18.969343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969235    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff485212-ff1b-4df3-91fd-faf80e501f77" containerName="revoke-keys"
Jul 16 21:11:18.969343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969240    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aeebad2-8bf1-4452-b46f-3a408311d112" containerName="revoke-keys"
Jul 16 21:11:18.969343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969245    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aeebad2-8bf1-4452-b46f-3a408311d112" containerName="revoke-keys"
Jul 16 21:11:18.969343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969253    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85bbe7c3-c8c6-429d-b963-b0a0c771de16" containerName="revoke-keys"
Jul 16 21:11:18.969343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969258    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="85bbe7c3-c8c6-429d-b963-b0a0c771de16" containerName="revoke-keys"
Jul 16 21:11:18.969343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969265    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff485212-ff1b-4df3-91fd-faf80e501f77" containerName="revoke-keys"
Jul 16 21:11:18.969343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969270    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff485212-ff1b-4df3-91fd-faf80e501f77" containerName="revoke-keys"
Jul 16 21:11:18.969343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969279    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85bbe7c3-c8c6-429d-b963-b0a0c771de16" containerName="revoke-keys"
Jul 16 21:11:18.969343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969285    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="85bbe7c3-c8c6-429d-b963-b0a0c771de16" containerName="revoke-keys"
Jul 16 21:11:18.969343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969291    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aeebad2-8bf1-4452-b46f-3a408311d112" containerName="revoke-keys"
Jul 16 21:11:18.969343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969296    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aeebad2-8bf1-4452-b46f-3a408311d112" containerName="revoke-keys"
Jul 16 21:11:18.969963 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969358    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="85bbe7c3-c8c6-429d-b963-b0a0c771de16" containerName="revoke-keys"
Jul 16 21:11:18.969963 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969365    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a1de14b-e01c-44d8-a4a5-d07990aa9905" containerName="revoke-keys"
Jul 16 21:11:18.969963 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969371    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aeebad2-8bf1-4452-b46f-3a408311d112" containerName="revoke-keys"
Jul 16 21:11:18.969963 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969377    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aeebad2-8bf1-4452-b46f-3a408311d112" containerName="revoke-keys"
Jul 16 21:11:18.969963 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969385    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aeebad2-8bf1-4452-b46f-3a408311d112" containerName="revoke-keys"
Jul 16 21:11:18.969963 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969389    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff485212-ff1b-4df3-91fd-faf80e501f77" containerName="revoke-keys"
Jul 16 21:11:18.969963 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969395    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a1de14b-e01c-44d8-a4a5-d07990aa9905" containerName="revoke-keys"
Jul 16 21:11:18.969963 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969399    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff485212-ff1b-4df3-91fd-faf80e501f77" containerName="revoke-keys"
Jul 16 21:11:18.969963 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969404    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="85bbe7c3-c8c6-429d-b963-b0a0c771de16" containerName="revoke-keys"
Jul 16 21:11:18.969963 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969408    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a1de14b-e01c-44d8-a4a5-d07990aa9905" containerName="revoke-keys"
Jul 16 21:11:18.969963 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.969414    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff485212-ff1b-4df3-91fd-faf80e501f77" containerName="revoke-keys"
Jul 16 21:11:18.972384 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.972363    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff"
Jul 16 21:11:18.974749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.974721    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:11:18.980417 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:18.980388    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff"]
Jul 16 21:11:19.045004 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:19.044954    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff\" (UID: \"cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff"
Jul 16 21:11:19.146438 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:19.146402    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff\" (UID: \"cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff"
Jul 16 21:11:19.147025 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:19.147005    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff\" (UID: \"cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff"
Jul 16 21:11:19.284165 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:19.284069    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff"
Jul 16 21:11:19.426432 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:19.426404    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff"]
Jul 16 21:11:19.429075 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:11:19.429044    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc125cd8_f6ca_4e0b_b4d9_8926a3d9daa0.slice/crio-452067112b360ee6ebf0d06865635b6298351086cf229d71ef351c07068eebc1 WatchSource:0}: Error finding container 452067112b360ee6ebf0d06865635b6298351086cf229d71ef351c07068eebc1: Status 404 returned error can't find the container with id 452067112b360ee6ebf0d06865635b6298351086cf229d71ef351c07068eebc1
Jul 16 21:11:19.594060 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:19.593966    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff" event={"ID":"cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0","Type":"ContainerStarted","Data":"452067112b360ee6ebf0d06865635b6298351086cf229d71ef351c07068eebc1"}
Jul 16 21:11:20.600353 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:20.600316    2530 generic.go:356] "Generic (PLEG): container finished" podID="cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0" containerID="fac085b8c26dd69bc6a9068690bee5c6f5ee29dd85cabff94e6484a8991e6268" exitCode=7
Jul 16 21:11:20.600759 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:20.600409    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff" event={"ID":"cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0","Type":"ContainerDied","Data":"fac085b8c26dd69bc6a9068690bee5c6f5ee29dd85cabff94e6484a8991e6268"}
Jul 16 21:11:20.600759 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:20.600581    2530 scope.go:117] "RemoveContainer" containerID="fac085b8c26dd69bc6a9068690bee5c6f5ee29dd85cabff94e6484a8991e6268"
Jul 16 21:11:21.607071 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:21.606962    2530 generic.go:356] "Generic (PLEG): container finished" podID="cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0" containerID="ad25c2b0ceab30e04bb7b6fc25da7e2d7999f9198b19d232b87594636ed7cf62" exitCode=7
Jul 16 21:11:21.607071 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:21.607023    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff" event={"ID":"cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0","Type":"ContainerDied","Data":"ad25c2b0ceab30e04bb7b6fc25da7e2d7999f9198b19d232b87594636ed7cf62"}
Jul 16 21:11:21.607584 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:21.607084    2530 scope.go:117] "RemoveContainer" containerID="fac085b8c26dd69bc6a9068690bee5c6f5ee29dd85cabff94e6484a8991e6268"
Jul 16 21:11:21.607584 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:21.607324    2530 scope.go:117] "RemoveContainer" containerID="ad25c2b0ceab30e04bb7b6fc25da7e2d7999f9198b19d232b87594636ed7cf62"
Jul 16 21:11:21.607584 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:11:21.607551    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff_odh-ai-gateway-infra(cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff" podUID="cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0"
Jul 16 21:11:21.816885 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:21.816839    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx"]
Jul 16 21:11:21.817353 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:21.817331    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a1de14b-e01c-44d8-a4a5-d07990aa9905" containerName="revoke-keys"
Jul 16 21:11:21.817479 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:21.817354    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a1de14b-e01c-44d8-a4a5-d07990aa9905" containerName="revoke-keys"
Jul 16 21:11:21.817479 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:21.817373    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85bbe7c3-c8c6-429d-b963-b0a0c771de16" containerName="revoke-keys"
Jul 16 21:11:21.817479 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:21.817381    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="85bbe7c3-c8c6-429d-b963-b0a0c771de16" containerName="revoke-keys"
Jul 16 21:11:21.817479 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:21.817394    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff485212-ff1b-4df3-91fd-faf80e501f77" containerName="revoke-keys"
Jul 16 21:11:21.817479 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:21.817401    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff485212-ff1b-4df3-91fd-faf80e501f77" containerName="revoke-keys"
Jul 16 21:11:21.817733 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:21.817488    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="85bbe7c3-c8c6-429d-b963-b0a0c771de16" containerName="revoke-keys"
Jul 16 21:11:21.821712 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:21.821681    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx"
Jul 16 21:11:21.827950 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:21.827915    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx"]
Jul 16 21:11:21.868655 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:21.868547    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9e281f-ef8c-44b2-9e5a-e7d524624a51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx\" (UID: \"bc9e281f-ef8c-44b2-9e5a-e7d524624a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx"
Jul 16 21:11:21.969773 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:21.969735    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9e281f-ef8c-44b2-9e5a-e7d524624a51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx\" (UID: \"bc9e281f-ef8c-44b2-9e5a-e7d524624a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx"
Jul 16 21:11:21.970438 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:21.970418    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9e281f-ef8c-44b2-9e5a-e7d524624a51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx\" (UID: \"bc9e281f-ef8c-44b2-9e5a-e7d524624a51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx"
Jul 16 21:11:22.133816 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:22.133721    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx"
Jul 16 21:11:22.276080 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:22.276031    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx"]
Jul 16 21:11:22.279230 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:11:22.279197    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9e281f_ef8c_44b2_9e5a_e7d524624a51.slice/crio-a9149b369d85fc4e1f753410b00f62f05b0db561977b2b9ea27ee3e763f0c4b0 WatchSource:0}: Error finding container a9149b369d85fc4e1f753410b00f62f05b0db561977b2b9ea27ee3e763f0c4b0: Status 404 returned error can't find the container with id a9149b369d85fc4e1f753410b00f62f05b0db561977b2b9ea27ee3e763f0c4b0
Jul 16 21:11:22.613626 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:22.613599    2530 scope.go:117] "RemoveContainer" containerID="ad25c2b0ceab30e04bb7b6fc25da7e2d7999f9198b19d232b87594636ed7cf62"
Jul 16 21:11:22.614132 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:11:22.613862    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff_odh-ai-gateway-infra(cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff" podUID="cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0"
Jul 16 21:11:22.614956 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:22.614931    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx" event={"ID":"bc9e281f-ef8c-44b2-9e5a-e7d524624a51","Type":"ContainerStarted","Data":"58e731d5d4b9688cfffc3b4e24fd4e1bd211b72a9f82518a85a3eaa7d5677a17"}
Jul 16 21:11:22.615081 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:22.614977    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx" event={"ID":"bc9e281f-ef8c-44b2-9e5a-e7d524624a51","Type":"ContainerStarted","Data":"a9149b369d85fc4e1f753410b00f62f05b0db561977b2b9ea27ee3e763f0c4b0"}
Jul 16 21:11:22.644836 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:22.644776    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx" podStartSLOduration=1.644758644 podStartE2EDuration="1.644758644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:11:22.64385182 +0000 UTC m=+2320.665439588" watchObservedRunningTime="2026-07-16 21:11:22.644758644 +0000 UTC m=+2320.666346409"
Jul 16 21:11:23.621074 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:23.621035    2530 generic.go:356] "Generic (PLEG): container finished" podID="bc9e281f-ef8c-44b2-9e5a-e7d524624a51" containerID="58e731d5d4b9688cfffc3b4e24fd4e1bd211b72a9f82518a85a3eaa7d5677a17" exitCode=6
Jul 16 21:11:23.621535 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:23.621090    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx" event={"ID":"bc9e281f-ef8c-44b2-9e5a-e7d524624a51","Type":"ContainerDied","Data":"58e731d5d4b9688cfffc3b4e24fd4e1bd211b72a9f82518a85a3eaa7d5677a17"}
Jul 16 21:11:23.621535 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:23.621325    2530 scope.go:117] "RemoveContainer" containerID="58e731d5d4b9688cfffc3b4e24fd4e1bd211b72a9f82518a85a3eaa7d5677a17"
Jul 16 21:11:24.627474 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:24.627381    2530 generic.go:356] "Generic (PLEG): container finished" podID="bc9e281f-ef8c-44b2-9e5a-e7d524624a51" containerID="34ef88a79ee3df19af881e2d13f57dfdb0d1bba87570b97511d8d0345360d6c4" exitCode=6
Jul 16 21:11:24.627474 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:24.627428    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx" event={"ID":"bc9e281f-ef8c-44b2-9e5a-e7d524624a51","Type":"ContainerDied","Data":"34ef88a79ee3df19af881e2d13f57dfdb0d1bba87570b97511d8d0345360d6c4"}
Jul 16 21:11:24.627991 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:24.627495    2530 scope.go:117] "RemoveContainer" containerID="58e731d5d4b9688cfffc3b4e24fd4e1bd211b72a9f82518a85a3eaa7d5677a17"
Jul 16 21:11:24.627991 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:24.627681    2530 scope.go:117] "RemoveContainer" containerID="34ef88a79ee3df19af881e2d13f57dfdb0d1bba87570b97511d8d0345360d6c4"
Jul 16 21:11:24.627991 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:11:24.627924    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx_odh-ai-gateway-infra(bc9e281f-ef8c-44b2-9e5a-e7d524624a51)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx" podUID="bc9e281f-ef8c-44b2-9e5a-e7d524624a51"
Jul 16 21:11:25.634030 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:25.634003    2530 scope.go:117] "RemoveContainer" containerID="34ef88a79ee3df19af881e2d13f57dfdb0d1bba87570b97511d8d0345360d6c4"
Jul 16 21:11:25.634437 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:11:25.634208    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx_odh-ai-gateway-infra(bc9e281f-ef8c-44b2-9e5a-e7d524624a51)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx" podUID="bc9e281f-ef8c-44b2-9e5a-e7d524624a51"
Jul 16 21:11:29.502864 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:29.502830    2530 scope.go:117] "RemoveContainer" containerID="ab355bb287b35f902a1f2d710b86425e94344e4566c31d5accd58f25dba6aff5"
Jul 16 21:11:29.503502 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:11:29.503060    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:11:33.502058 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:33.501984    2530 scope.go:117] "RemoveContainer" containerID="ad25c2b0ceab30e04bb7b6fc25da7e2d7999f9198b19d232b87594636ed7cf62"
Jul 16 21:11:34.676268 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:34.676232    2530 generic.go:356] "Generic (PLEG): container finished" podID="cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0" containerID="18178fa67aad071e4bda2249323fa264f0a79248e794eb12b5cd23bd6baf746e" exitCode=7
Jul 16 21:11:34.676685 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:34.676309    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff" event={"ID":"cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0","Type":"ContainerDied","Data":"18178fa67aad071e4bda2249323fa264f0a79248e794eb12b5cd23bd6baf746e"}
Jul 16 21:11:34.676685 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:34.676378    2530 scope.go:117] "RemoveContainer" containerID="ad25c2b0ceab30e04bb7b6fc25da7e2d7999f9198b19d232b87594636ed7cf62"
Jul 16 21:11:34.676685 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:34.676656    2530 scope.go:117] "RemoveContainer" containerID="18178fa67aad071e4bda2249323fa264f0a79248e794eb12b5cd23bd6baf746e"
Jul 16 21:11:34.676900 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:11:34.676877    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff_odh-ai-gateway-infra(cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff" podUID="cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0"
Jul 16 21:11:35.711183 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:35.711144    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff"]
Jul 16 21:11:35.847531 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:35.847504    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff"
Jul 16 21:11:35.903170 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:35.903125    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0-maas-api-service-ca\") pod \"cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0\" (UID: \"cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0\") "
Jul 16 21:11:35.903552 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:35.903531    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0" (UID: "cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:11:36.004443 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:36.004348    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:11:36.502611 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:36.502576    2530 scope.go:117] "RemoveContainer" containerID="34ef88a79ee3df19af881e2d13f57dfdb0d1bba87570b97511d8d0345360d6c4"
Jul 16 21:11:36.689247 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:36.689205    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff" event={"ID":"cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0","Type":"ContainerDied","Data":"452067112b360ee6ebf0d06865635b6298351086cf229d71ef351c07068eebc1"}
Jul 16 21:11:36.689433 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:36.689266    2530 scope.go:117] "RemoveContainer" containerID="18178fa67aad071e4bda2249323fa264f0a79248e794eb12b5cd23bd6baf746e"
Jul 16 21:11:36.689433 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:36.689223    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff"
Jul 16 21:11:36.711916 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:36.711881    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff"]
Jul 16 21:11:36.717650 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:36.717619    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hc6ff"]
Jul 16 21:11:37.696742 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:37.696650    2530 generic.go:356] "Generic (PLEG): container finished" podID="bc9e281f-ef8c-44b2-9e5a-e7d524624a51" containerID="6a23ba4fc41c619f18aef3787ea0b47096e44c2420b54d4e0b77ada4e97c6996" exitCode=6
Jul 16 21:11:37.696742 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:37.696704    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx" event={"ID":"bc9e281f-ef8c-44b2-9e5a-e7d524624a51","Type":"ContainerDied","Data":"6a23ba4fc41c619f18aef3787ea0b47096e44c2420b54d4e0b77ada4e97c6996"}
Jul 16 21:11:37.696943 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:37.696756    2530 scope.go:117] "RemoveContainer" containerID="34ef88a79ee3df19af881e2d13f57dfdb0d1bba87570b97511d8d0345360d6c4"
Jul 16 21:11:37.697168 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:37.697155    2530 scope.go:117] "RemoveContainer" containerID="6a23ba4fc41c619f18aef3787ea0b47096e44c2420b54d4e0b77ada4e97c6996"
Jul 16 21:11:37.697434 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:11:37.697411    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx_odh-ai-gateway-infra(bc9e281f-ef8c-44b2-9e5a-e7d524624a51)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx" podUID="bc9e281f-ef8c-44b2-9e5a-e7d524624a51"
Jul 16 21:11:38.511045 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:38.509724    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0" path="/var/lib/kubelet/pods/cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0/volumes"
Jul 16 21:11:38.723864 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:38.723819    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx"]
Jul 16 21:11:38.866719 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:38.866685    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx"
Jul 16 21:11:38.930738 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:38.930681    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9e281f-ef8c-44b2-9e5a-e7d524624a51-maas-api-service-ca\") pod \"bc9e281f-ef8c-44b2-9e5a-e7d524624a51\" (UID: \"bc9e281f-ef8c-44b2-9e5a-e7d524624a51\") "
Jul 16 21:11:38.931153 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:38.931090    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9e281f-ef8c-44b2-9e5a-e7d524624a51-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc9e281f-ef8c-44b2-9e5a-e7d524624a51" (UID: "bc9e281f-ef8c-44b2-9e5a-e7d524624a51"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:11:39.032024 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:39.031933    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9e281f-ef8c-44b2-9e5a-e7d524624a51-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:11:39.711068 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:39.711038    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx"
Jul 16 21:11:39.711068 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:39.711057    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx" event={"ID":"bc9e281f-ef8c-44b2-9e5a-e7d524624a51","Type":"ContainerDied","Data":"a9149b369d85fc4e1f753410b00f62f05b0db561977b2b9ea27ee3e763f0c4b0"}
Jul 16 21:11:39.711665 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:39.711154    2530 scope.go:117] "RemoveContainer" containerID="6a23ba4fc41c619f18aef3787ea0b47096e44c2420b54d4e0b77ada4e97c6996"
Jul 16 21:11:39.732476 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:39.732442    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx"]
Jul 16 21:11:39.739680 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:39.739649    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fv7wx"]
Jul 16 21:11:40.507515 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:40.507476    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc9e281f-ef8c-44b2-9e5a-e7d524624a51" path="/var/lib/kubelet/pods/bc9e281f-ef8c-44b2-9e5a-e7d524624a51/volumes"
Jul 16 21:11:41.502896 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:41.502860    2530 scope.go:117] "RemoveContainer" containerID="ab355bb287b35f902a1f2d710b86425e94344e4566c31d5accd58f25dba6aff5"
Jul 16 21:11:41.503337 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:11:41.503043    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:11:52.504726 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:11:52.504695    2530 scope.go:117] "RemoveContainer" containerID="ab355bb287b35f902a1f2d710b86425e94344e4566c31d5accd58f25dba6aff5"
Jul 16 21:11:52.505262 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:11:52.504893    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:12:05.502253 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:05.502219    2530 scope.go:117] "RemoveContainer" containerID="ab355bb287b35f902a1f2d710b86425e94344e4566c31d5accd58f25dba6aff5"
Jul 16 21:12:06.838481 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:06.838442    2530 generic.go:356] "Generic (PLEG): container finished" podID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerID="65b0d947539cb0b888831c71173e464f5431d9304a1a99e2f557eb9066623688" exitCode=1
Jul 16 21:12:06.838917 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:06.838518    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" event={"ID":"b25e21cd-ccbd-4870-99aa-ba1b5dc98083","Type":"ContainerDied","Data":"65b0d947539cb0b888831c71173e464f5431d9304a1a99e2f557eb9066623688"}
Jul 16 21:12:06.838917 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:06.838590    2530 scope.go:117] "RemoveContainer" containerID="ab355bb287b35f902a1f2d710b86425e94344e4566c31d5accd58f25dba6aff5"
Jul 16 21:12:06.839040 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:06.838987    2530 scope.go:117] "RemoveContainer" containerID="65b0d947539cb0b888831c71173e464f5431d9304a1a99e2f557eb9066623688"
Jul 16 21:12:06.839240 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:12:06.839216    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:12:09.013931 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.013891    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg"]
Jul 16 21:12:09.014351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.014233    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc9e281f-ef8c-44b2-9e5a-e7d524624a51" containerName="revoke-keys"
Jul 16 21:12:09.014351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.014244    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc9e281f-ef8c-44b2-9e5a-e7d524624a51" containerName="revoke-keys"
Jul 16 21:12:09.014351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.014252    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0" containerName="revoke-keys"
Jul 16 21:12:09.014351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.014258    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0" containerName="revoke-keys"
Jul 16 21:12:09.014351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.014270    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc9e281f-ef8c-44b2-9e5a-e7d524624a51" containerName="revoke-keys"
Jul 16 21:12:09.014351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.014276    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc9e281f-ef8c-44b2-9e5a-e7d524624a51" containerName="revoke-keys"
Jul 16 21:12:09.014351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.014284    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc9e281f-ef8c-44b2-9e5a-e7d524624a51" containerName="revoke-keys"
Jul 16 21:12:09.014351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.014288    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc9e281f-ef8c-44b2-9e5a-e7d524624a51" containerName="revoke-keys"
Jul 16 21:12:09.014351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.014296    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0" containerName="revoke-keys"
Jul 16 21:12:09.014351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.014301    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0" containerName="revoke-keys"
Jul 16 21:12:09.014351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.014346    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0" containerName="revoke-keys"
Jul 16 21:12:09.014351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.014355    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc9e281f-ef8c-44b2-9e5a-e7d524624a51" containerName="revoke-keys"
Jul 16 21:12:09.014707 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.014361    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc9e281f-ef8c-44b2-9e5a-e7d524624a51" containerName="revoke-keys"
Jul 16 21:12:09.014707 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.014367    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc9e281f-ef8c-44b2-9e5a-e7d524624a51" containerName="revoke-keys"
Jul 16 21:12:09.014707 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.014373    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0" containerName="revoke-keys"
Jul 16 21:12:09.018514 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.018494    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg"
Jul 16 21:12:09.020887 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.020865    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:12:09.024259 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.024234    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg"]
Jul 16 21:12:09.095335 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.095289    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d850ac0e-1df6-4d2f-9854-bf555676cfb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg\" (UID: \"d850ac0e-1df6-4d2f-9854-bf555676cfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg"
Jul 16 21:12:09.196401 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.196357    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d850ac0e-1df6-4d2f-9854-bf555676cfb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg\" (UID: \"d850ac0e-1df6-4d2f-9854-bf555676cfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg"
Jul 16 21:12:09.197017 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.196997    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d850ac0e-1df6-4d2f-9854-bf555676cfb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg\" (UID: \"d850ac0e-1df6-4d2f-9854-bf555676cfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg"
Jul 16 21:12:09.329973 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.329865    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg"
Jul 16 21:12:09.467151 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.467121    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg"]
Jul 16 21:12:09.469521 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:12:09.469491    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd850ac0e_1df6_4d2f_9854_bf555676cfb3.slice/crio-66d87874a5b0cff3f765e3cba8b70d46da48c83358271c6f2920fbcadf5e15a1 WatchSource:0}: Error finding container 66d87874a5b0cff3f765e3cba8b70d46da48c83358271c6f2920fbcadf5e15a1: Status 404 returned error can't find the container with id 66d87874a5b0cff3f765e3cba8b70d46da48c83358271c6f2920fbcadf5e15a1
Jul 16 21:12:09.857507 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.857467    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg" event={"ID":"d850ac0e-1df6-4d2f-9854-bf555676cfb3","Type":"ContainerStarted","Data":"9e074d2a4d363fae0d3f067315b4f8d7646097a05ec6d0f6e1a04b26f91bc55e"}
Jul 16 21:12:09.857705 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.857523    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg" event={"ID":"d850ac0e-1df6-4d2f-9854-bf555676cfb3","Type":"ContainerStarted","Data":"66d87874a5b0cff3f765e3cba8b70d46da48c83358271c6f2920fbcadf5e15a1"}
Jul 16 21:12:09.878335 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:09.878287    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg" podStartSLOduration=1.878268463 podStartE2EDuration="1.878268463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:12:09.876308459 +0000 UTC m=+2367.897896248" watchObservedRunningTime="2026-07-16 21:12:09.878268463 +0000 UTC m=+2367.899856271"
Jul 16 21:12:10.545886 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:10.545831    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:12:10.546340 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:10.546307    2530 scope.go:117] "RemoveContainer" containerID="65b0d947539cb0b888831c71173e464f5431d9304a1a99e2f557eb9066623688"
Jul 16 21:12:10.546512 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:12:10.546489    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:12:10.863329 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:10.863236    2530 generic.go:356] "Generic (PLEG): container finished" podID="d850ac0e-1df6-4d2f-9854-bf555676cfb3" containerID="9e074d2a4d363fae0d3f067315b4f8d7646097a05ec6d0f6e1a04b26f91bc55e" exitCode=7
Jul 16 21:12:10.863480 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:10.863321    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg" event={"ID":"d850ac0e-1df6-4d2f-9854-bf555676cfb3","Type":"ContainerDied","Data":"9e074d2a4d363fae0d3f067315b4f8d7646097a05ec6d0f6e1a04b26f91bc55e"}
Jul 16 21:12:10.863534 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:10.863491    2530 scope.go:117] "RemoveContainer" containerID="9e074d2a4d363fae0d3f067315b4f8d7646097a05ec6d0f6e1a04b26f91bc55e"
Jul 16 21:12:11.861032 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:11.860940    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x"]
Jul 16 21:12:11.861445 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:11.861326    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0" containerName="revoke-keys"
Jul 16 21:12:11.861445 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:11.861338    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0" containerName="revoke-keys"
Jul 16 21:12:11.861445 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:11.861400    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc125cd8-f6ca-4e0b-b4d9-8926a3d9daa0" containerName="revoke-keys"
Jul 16 21:12:11.864258 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:11.864227    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x"
Jul 16 21:12:11.870717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:11.870259    2530 generic.go:356] "Generic (PLEG): container finished" podID="d850ac0e-1df6-4d2f-9854-bf555676cfb3" containerID="be173f2a35aafd0e685cf5dfa73a4493883cb773b3a65c01fc6e911538f184c4" exitCode=7
Jul 16 21:12:11.870717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:11.870360    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg" event={"ID":"d850ac0e-1df6-4d2f-9854-bf555676cfb3","Type":"ContainerDied","Data":"be173f2a35aafd0e685cf5dfa73a4493883cb773b3a65c01fc6e911538f184c4"}
Jul 16 21:12:11.871049 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:11.870748    2530 scope.go:117] "RemoveContainer" containerID="9e074d2a4d363fae0d3f067315b4f8d7646097a05ec6d0f6e1a04b26f91bc55e"
Jul 16 21:12:11.871049 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:11.871000    2530 scope.go:117] "RemoveContainer" containerID="be173f2a35aafd0e685cf5dfa73a4493883cb773b3a65c01fc6e911538f184c4"
Jul 16 21:12:11.871323 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:12:11.871283    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg_odh-ai-gateway-infra(d850ac0e-1df6-4d2f-9854-bf555676cfb3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg" podUID="d850ac0e-1df6-4d2f-9854-bf555676cfb3"
Jul 16 21:12:11.873289 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:11.873128    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x"]
Jul 16 21:12:11.916475 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:11.916431    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b14de5-b0c8-4721-badc-1e29ee24bc7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x\" (UID: \"04b14de5-b0c8-4721-badc-1e29ee24bc7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x"
Jul 16 21:12:12.017657 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:12.017611    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b14de5-b0c8-4721-badc-1e29ee24bc7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x\" (UID: \"04b14de5-b0c8-4721-badc-1e29ee24bc7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x"
Jul 16 21:12:12.018263 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:12.018242    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b14de5-b0c8-4721-badc-1e29ee24bc7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x\" (UID: \"04b14de5-b0c8-4721-badc-1e29ee24bc7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x"
Jul 16 21:12:12.177553 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:12.177451    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x"
Jul 16 21:12:12.317183 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:12.317136    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x"]
Jul 16 21:12:12.319405 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:12:12.319366    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b14de5_b0c8_4721_badc_1e29ee24bc7f.slice/crio-18866ed215d03ffcedcaa0dd8dd5bf73d9a15dda9feacd8f2c4a4ee8ccc62288 WatchSource:0}: Error finding container 18866ed215d03ffcedcaa0dd8dd5bf73d9a15dda9feacd8f2c4a4ee8ccc62288: Status 404 returned error can't find the container with id 18866ed215d03ffcedcaa0dd8dd5bf73d9a15dda9feacd8f2c4a4ee8ccc62288
Jul 16 21:12:12.876354 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:12.876268    2530 scope.go:117] "RemoveContainer" containerID="be173f2a35aafd0e685cf5dfa73a4493883cb773b3a65c01fc6e911538f184c4"
Jul 16 21:12:12.876809 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:12:12.876515    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg_odh-ai-gateway-infra(d850ac0e-1df6-4d2f-9854-bf555676cfb3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg" podUID="d850ac0e-1df6-4d2f-9854-bf555676cfb3"
Jul 16 21:12:12.877554 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:12.877520    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x" event={"ID":"04b14de5-b0c8-4721-badc-1e29ee24bc7f","Type":"ContainerStarted","Data":"01a5fcf9ff1cc1d5f235d622b617eddbc52bd3a2bd4eff08e56090115fdaf069"}
Jul 16 21:12:12.877682 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:12.877586    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x" event={"ID":"04b14de5-b0c8-4721-badc-1e29ee24bc7f","Type":"ContainerStarted","Data":"18866ed215d03ffcedcaa0dd8dd5bf73d9a15dda9feacd8f2c4a4ee8ccc62288"}
Jul 16 21:12:12.908497 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:12.908434    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x" podStartSLOduration=1.9084185630000001 podStartE2EDuration="1.908418563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:12:12.907236737 +0000 UTC m=+2370.928824504" watchObservedRunningTime="2026-07-16 21:12:12.908418563 +0000 UTC m=+2370.930006328"
Jul 16 21:12:13.884129 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:13.884067    2530 generic.go:356] "Generic (PLEG): container finished" podID="04b14de5-b0c8-4721-badc-1e29ee24bc7f" containerID="01a5fcf9ff1cc1d5f235d622b617eddbc52bd3a2bd4eff08e56090115fdaf069" exitCode=6
Jul 16 21:12:13.884514 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:13.884142    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x" event={"ID":"04b14de5-b0c8-4721-badc-1e29ee24bc7f","Type":"ContainerDied","Data":"01a5fcf9ff1cc1d5f235d622b617eddbc52bd3a2bd4eff08e56090115fdaf069"}
Jul 16 21:12:13.884514 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:13.884398    2530 scope.go:117] "RemoveContainer" containerID="01a5fcf9ff1cc1d5f235d622b617eddbc52bd3a2bd4eff08e56090115fdaf069"
Jul 16 21:12:14.890603 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:14.890518    2530 generic.go:356] "Generic (PLEG): container finished" podID="04b14de5-b0c8-4721-badc-1e29ee24bc7f" containerID="8bc903be035115a8a0a62ad284f0ff7054645b42386f92631339cff598233287" exitCode=6
Jul 16 21:12:14.890603 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:14.890575    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x" event={"ID":"04b14de5-b0c8-4721-badc-1e29ee24bc7f","Type":"ContainerDied","Data":"8bc903be035115a8a0a62ad284f0ff7054645b42386f92631339cff598233287"}
Jul 16 21:12:14.891081 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:14.890623    2530 scope.go:117] "RemoveContainer" containerID="01a5fcf9ff1cc1d5f235d622b617eddbc52bd3a2bd4eff08e56090115fdaf069"
Jul 16 21:12:14.891081 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:14.890825    2530 scope.go:117] "RemoveContainer" containerID="8bc903be035115a8a0a62ad284f0ff7054645b42386f92631339cff598233287"
Jul 16 21:12:14.891081 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:12:14.891014    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x_odh-ai-gateway-infra(04b14de5-b0c8-4721-badc-1e29ee24bc7f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x" podUID="04b14de5-b0c8-4721-badc-1e29ee24bc7f"
Jul 16 21:12:15.546169 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:15.546083    2530 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:12:15.546569 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:15.546554    2530 scope.go:117] "RemoveContainer" containerID="65b0d947539cb0b888831c71173e464f5431d9304a1a99e2f557eb9066623688"
Jul 16 21:12:15.546737 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:12:15.546717    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:12:15.896448 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:15.896359    2530 scope.go:117] "RemoveContainer" containerID="8bc903be035115a8a0a62ad284f0ff7054645b42386f92631339cff598233287"
Jul 16 21:12:15.896846 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:12:15.896563    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x_odh-ai-gateway-infra(04b14de5-b0c8-4721-badc-1e29ee24bc7f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x" podUID="04b14de5-b0c8-4721-badc-1e29ee24bc7f"
Jul 16 21:12:21.142446 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:21.142408    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs"]
Jul 16 21:12:21.145564 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:21.145541    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs"
Jul 16 21:12:21.153756 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:21.153727    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs"]
Jul 16 21:12:21.195565 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:21.195530    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122afb43-17f0-4ea7-9a37-6efeff6b41de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs\" (UID: \"122afb43-17f0-4ea7-9a37-6efeff6b41de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs"
Jul 16 21:12:21.296069 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:21.296022    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122afb43-17f0-4ea7-9a37-6efeff6b41de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs\" (UID: \"122afb43-17f0-4ea7-9a37-6efeff6b41de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs"
Jul 16 21:12:21.296684 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:21.296663    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122afb43-17f0-4ea7-9a37-6efeff6b41de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs\" (UID: \"122afb43-17f0-4ea7-9a37-6efeff6b41de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs"
Jul 16 21:12:21.456214 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:21.456176    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs"
Jul 16 21:12:21.593764 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:21.593689    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs"]
Jul 16 21:12:21.596535 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:12:21.596503    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122afb43_17f0_4ea7_9a37_6efeff6b41de.slice/crio-20b12349da55f898ce57cf88ed31cd16657815494cc74e955ff93f8ba3820e5e WatchSource:0}: Error finding container 20b12349da55f898ce57cf88ed31cd16657815494cc74e955ff93f8ba3820e5e: Status 404 returned error can't find the container with id 20b12349da55f898ce57cf88ed31cd16657815494cc74e955ff93f8ba3820e5e
Jul 16 21:12:21.922453 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:21.922421    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs" event={"ID":"122afb43-17f0-4ea7-9a37-6efeff6b41de","Type":"ContainerStarted","Data":"20b12349da55f898ce57cf88ed31cd16657815494cc74e955ff93f8ba3820e5e"}
Jul 16 21:12:22.928424 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:22.928386    2530 generic.go:356] "Generic (PLEG): container finished" podID="122afb43-17f0-4ea7-9a37-6efeff6b41de" containerID="f5e3f5e40ecbfc41e84852c1ed1ebbc12c74cf04e7095b2a6588a067a655b1f7" exitCode=6
Jul 16 21:12:22.928805 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:22.928449    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs" event={"ID":"122afb43-17f0-4ea7-9a37-6efeff6b41de","Type":"ContainerDied","Data":"f5e3f5e40ecbfc41e84852c1ed1ebbc12c74cf04e7095b2a6588a067a655b1f7"}
Jul 16 21:12:22.928805 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:22.928679    2530 scope.go:117] "RemoveContainer" containerID="f5e3f5e40ecbfc41e84852c1ed1ebbc12c74cf04e7095b2a6588a067a655b1f7"
Jul 16 21:12:23.502759 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:23.502720    2530 scope.go:117] "RemoveContainer" containerID="be173f2a35aafd0e685cf5dfa73a4493883cb773b3a65c01fc6e911538f184c4"
Jul 16 21:12:23.935340 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:23.935305    2530 generic.go:356] "Generic (PLEG): container finished" podID="122afb43-17f0-4ea7-9a37-6efeff6b41de" containerID="493d15fb41366a03365e39b4fcc2eed162df74305e9d5b80fd201eb375e23ffb" exitCode=6
Jul 16 21:12:23.935792 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:23.935389    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs" event={"ID":"122afb43-17f0-4ea7-9a37-6efeff6b41de","Type":"ContainerDied","Data":"493d15fb41366a03365e39b4fcc2eed162df74305e9d5b80fd201eb375e23ffb"}
Jul 16 21:12:23.935792 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:23.935453    2530 scope.go:117] "RemoveContainer" containerID="f5e3f5e40ecbfc41e84852c1ed1ebbc12c74cf04e7095b2a6588a067a655b1f7"
Jul 16 21:12:23.935792 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:23.935621    2530 scope.go:117] "RemoveContainer" containerID="493d15fb41366a03365e39b4fcc2eed162df74305e9d5b80fd201eb375e23ffb"
Jul 16 21:12:23.935947 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:12:23.935843    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs_odh-ai-gateway-infra(122afb43-17f0-4ea7-9a37-6efeff6b41de)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs" podUID="122afb43-17f0-4ea7-9a37-6efeff6b41de"
Jul 16 21:12:23.937376 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:23.937354    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg" event={"ID":"d850ac0e-1df6-4d2f-9854-bf555676cfb3","Type":"ContainerStarted","Data":"e66afc741d23b4b0f199962826a6a00b9403c9c64b38176b4ba5a714139e8cf2"}
Jul 16 21:12:24.943675 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:24.943638    2530 scope.go:117] "RemoveContainer" containerID="493d15fb41366a03365e39b4fcc2eed162df74305e9d5b80fd201eb375e23ffb"
Jul 16 21:12:24.944084 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:12:24.943873    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs_odh-ai-gateway-infra(122afb43-17f0-4ea7-9a37-6efeff6b41de)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs" podUID="122afb43-17f0-4ea7-9a37-6efeff6b41de"
Jul 16 21:12:24.945248 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:24.945224    2530 generic.go:356] "Generic (PLEG): container finished" podID="d850ac0e-1df6-4d2f-9854-bf555676cfb3" containerID="e66afc741d23b4b0f199962826a6a00b9403c9c64b38176b4ba5a714139e8cf2" exitCode=7
Jul 16 21:12:24.945380 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:24.945272    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg" event={"ID":"d850ac0e-1df6-4d2f-9854-bf555676cfb3","Type":"ContainerDied","Data":"e66afc741d23b4b0f199962826a6a00b9403c9c64b38176b4ba5a714139e8cf2"}
Jul 16 21:12:24.945380 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:24.945327    2530 scope.go:117] "RemoveContainer" containerID="be173f2a35aafd0e685cf5dfa73a4493883cb773b3a65c01fc6e911538f184c4"
Jul 16 21:12:24.945586 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:24.945570    2530 scope.go:117] "RemoveContainer" containerID="e66afc741d23b4b0f199962826a6a00b9403c9c64b38176b4ba5a714139e8cf2"
Jul 16 21:12:24.945765 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:12:24.945747    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg_odh-ai-gateway-infra(d850ac0e-1df6-4d2f-9854-bf555676cfb3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg" podUID="d850ac0e-1df6-4d2f-9854-bf555676cfb3"
Jul 16 21:12:24.973911 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:24.973886    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg"]
Jul 16 21:12:26.080736 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:26.080712    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg"
Jul 16 21:12:26.138210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:26.138186    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d850ac0e-1df6-4d2f-9854-bf555676cfb3-maas-api-service-ca\") pod \"d850ac0e-1df6-4d2f-9854-bf555676cfb3\" (UID: \"d850ac0e-1df6-4d2f-9854-bf555676cfb3\") "
Jul 16 21:12:26.138500 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:26.138479    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d850ac0e-1df6-4d2f-9854-bf555676cfb3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d850ac0e-1df6-4d2f-9854-bf555676cfb3" (UID: "d850ac0e-1df6-4d2f-9854-bf555676cfb3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:12:26.239002 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:26.238975    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d850ac0e-1df6-4d2f-9854-bf555676cfb3-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:12:26.960344 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:26.960313    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg" event={"ID":"d850ac0e-1df6-4d2f-9854-bf555676cfb3","Type":"ContainerDied","Data":"66d87874a5b0cff3f765e3cba8b70d46da48c83358271c6f2920fbcadf5e15a1"}
Jul 16 21:12:26.960344 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:26.960341    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg"
Jul 16 21:12:26.960584 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:26.960383    2530 scope.go:117] "RemoveContainer" containerID="e66afc741d23b4b0f199962826a6a00b9403c9c64b38176b4ba5a714139e8cf2"
Jul 16 21:12:26.978683 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:26.978658    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg"]
Jul 16 21:12:26.982612 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:26.982592    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-rvmkg"]
Jul 16 21:12:28.502648 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:28.502614    2530 scope.go:117] "RemoveContainer" containerID="8bc903be035115a8a0a62ad284f0ff7054645b42386f92631339cff598233287"
Jul 16 21:12:28.508465 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:28.508428    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d850ac0e-1df6-4d2f-9854-bf555676cfb3" path="/var/lib/kubelet/pods/d850ac0e-1df6-4d2f-9854-bf555676cfb3/volumes"
Jul 16 21:12:28.972941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:28.972912    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x" event={"ID":"04b14de5-b0c8-4721-badc-1e29ee24bc7f","Type":"ContainerStarted","Data":"69c613a2d3dca825092b1d437bddac0c739e8b0d1624f5a1641da3cd728441cb"}
Jul 16 21:12:29.979897 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:29.979866    2530 generic.go:356] "Generic (PLEG): container finished" podID="04b14de5-b0c8-4721-badc-1e29ee24bc7f" containerID="69c613a2d3dca825092b1d437bddac0c739e8b0d1624f5a1641da3cd728441cb" exitCode=6
Jul 16 21:12:29.980325 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:29.979915    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x" event={"ID":"04b14de5-b0c8-4721-badc-1e29ee24bc7f","Type":"ContainerDied","Data":"69c613a2d3dca825092b1d437bddac0c739e8b0d1624f5a1641da3cd728441cb"}
Jul 16 21:12:29.980325 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:29.979966    2530 scope.go:117] "RemoveContainer" containerID="8bc903be035115a8a0a62ad284f0ff7054645b42386f92631339cff598233287"
Jul 16 21:12:29.980325 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:29.980259    2530 scope.go:117] "RemoveContainer" containerID="69c613a2d3dca825092b1d437bddac0c739e8b0d1624f5a1641da3cd728441cb"
Jul 16 21:12:29.980472 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:12:29.980450    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x_odh-ai-gateway-infra(04b14de5-b0c8-4721-badc-1e29ee24bc7f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x" podUID="04b14de5-b0c8-4721-badc-1e29ee24bc7f"
Jul 16 21:12:30.005997 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:30.005978    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x"]
Jul 16 21:12:30.502506 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:30.502477    2530 scope.go:117] "RemoveContainer" containerID="65b0d947539cb0b888831c71173e464f5431d9304a1a99e2f557eb9066623688"
Jul 16 21:12:30.502677 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:12:30.502656    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:12:31.121829 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:31.121805    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x"
Jul 16 21:12:31.176356 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:31.176332    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b14de5-b0c8-4721-badc-1e29ee24bc7f-maas-api-service-ca\") pod \"04b14de5-b0c8-4721-badc-1e29ee24bc7f\" (UID: \"04b14de5-b0c8-4721-badc-1e29ee24bc7f\") "
Jul 16 21:12:31.176655 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:31.176636    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b14de5-b0c8-4721-badc-1e29ee24bc7f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04b14de5-b0c8-4721-badc-1e29ee24bc7f" (UID: "04b14de5-b0c8-4721-badc-1e29ee24bc7f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:12:31.277701 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:31.277644    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b14de5-b0c8-4721-badc-1e29ee24bc7f-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:12:31.991212 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:31.991183    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x"
Jul 16 21:12:31.991381 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:31.991182    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x" event={"ID":"04b14de5-b0c8-4721-badc-1e29ee24bc7f","Type":"ContainerDied","Data":"18866ed215d03ffcedcaa0dd8dd5bf73d9a15dda9feacd8f2c4a4ee8ccc62288"}
Jul 16 21:12:31.991381 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:31.991327    2530 scope.go:117] "RemoveContainer" containerID="69c613a2d3dca825092b1d437bddac0c739e8b0d1624f5a1641da3cd728441cb"
Jul 16 21:12:32.012156 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:32.012132    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x"]
Jul 16 21:12:32.018069 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:32.018049    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-6pm6x"]
Jul 16 21:12:32.506952 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:32.506918    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04b14de5-b0c8-4721-badc-1e29ee24bc7f" path="/var/lib/kubelet/pods/04b14de5-b0c8-4721-badc-1e29ee24bc7f/volumes"
Jul 16 21:12:35.502549 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:35.502520    2530 scope.go:117] "RemoveContainer" containerID="493d15fb41366a03365e39b4fcc2eed162df74305e9d5b80fd201eb375e23ffb"
Jul 16 21:12:36.010466 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:36.010427    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs" event={"ID":"122afb43-17f0-4ea7-9a37-6efeff6b41de","Type":"ContainerStarted","Data":"54cdd3251550cfc7c49bf60419bc9124c2f617a3ffa1f2ee98dd146cc95d3746"}
Jul 16 21:12:36.026036 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:36.025990    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs" podStartSLOduration=15.025977016 podStartE2EDuration="15.025977016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:12:36.024573847 +0000 UTC m=+2394.046161613" watchObservedRunningTime="2026-07-16 21:12:36.025977016 +0000 UTC m=+2394.047564820"
Jul 16 21:12:37.016646 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:37.016613    2530 generic.go:356] "Generic (PLEG): container finished" podID="122afb43-17f0-4ea7-9a37-6efeff6b41de" containerID="54cdd3251550cfc7c49bf60419bc9124c2f617a3ffa1f2ee98dd146cc95d3746" exitCode=6
Jul 16 21:12:37.017130 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:37.016688    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs" event={"ID":"122afb43-17f0-4ea7-9a37-6efeff6b41de","Type":"ContainerDied","Data":"54cdd3251550cfc7c49bf60419bc9124c2f617a3ffa1f2ee98dd146cc95d3746"}
Jul 16 21:12:37.017130 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:37.016748    2530 scope.go:117] "RemoveContainer" containerID="493d15fb41366a03365e39b4fcc2eed162df74305e9d5b80fd201eb375e23ffb"
Jul 16 21:12:37.017130 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:37.017119    2530 scope.go:117] "RemoveContainer" containerID="54cdd3251550cfc7c49bf60419bc9124c2f617a3ffa1f2ee98dd146cc95d3746"
Jul 16 21:12:37.017342 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:12:37.017323    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs_odh-ai-gateway-infra(122afb43-17f0-4ea7-9a37-6efeff6b41de)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs" podUID="122afb43-17f0-4ea7-9a37-6efeff6b41de"
Jul 16 21:12:37.043965 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:37.043941    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs"]
Jul 16 21:12:38.151555 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:38.151533    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs"
Jul 16 21:12:38.230964 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:38.230938    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122afb43-17f0-4ea7-9a37-6efeff6b41de-maas-api-service-ca\") pod \"122afb43-17f0-4ea7-9a37-6efeff6b41de\" (UID: \"122afb43-17f0-4ea7-9a37-6efeff6b41de\") "
Jul 16 21:12:38.231293 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:38.231272    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122afb43-17f0-4ea7-9a37-6efeff6b41de-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "122afb43-17f0-4ea7-9a37-6efeff6b41de" (UID: "122afb43-17f0-4ea7-9a37-6efeff6b41de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:12:38.331698 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:38.331641    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122afb43-17f0-4ea7-9a37-6efeff6b41de-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:12:39.027928 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:39.027895    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs"
Jul 16 21:12:39.027928 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:39.027914    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs" event={"ID":"122afb43-17f0-4ea7-9a37-6efeff6b41de","Type":"ContainerDied","Data":"20b12349da55f898ce57cf88ed31cd16657815494cc74e955ff93f8ba3820e5e"}
Jul 16 21:12:39.028235 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:39.027971    2530 scope.go:117] "RemoveContainer" containerID="54cdd3251550cfc7c49bf60419bc9124c2f617a3ffa1f2ee98dd146cc95d3746"
Jul 16 21:12:39.044544 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:39.044517    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs"]
Jul 16 21:12:39.052455 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:39.052436    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-4wlrs"]
Jul 16 21:12:40.507408 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:40.507370    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="122afb43-17f0-4ea7-9a37-6efeff6b41de" path="/var/lib/kubelet/pods/122afb43-17f0-4ea7-9a37-6efeff6b41de/volumes"
Jul 16 21:12:41.502331 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:41.502300    2530 scope.go:117] "RemoveContainer" containerID="65b0d947539cb0b888831c71173e464f5431d9304a1a99e2f557eb9066623688"
Jul 16 21:12:41.502523 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:12:41.502467    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:12:56.502034 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:12:56.501988    2530 scope.go:117] "RemoveContainer" containerID="65b0d947539cb0b888831c71173e464f5431d9304a1a99e2f557eb9066623688"
Jul 16 21:12:56.502525 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:12:56.502204    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:13:03.837364 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837277    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86"]
Jul 16 21:13:03.837717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837590    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d850ac0e-1df6-4d2f-9854-bf555676cfb3" containerName="revoke-keys"
Jul 16 21:13:03.837717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837602    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="d850ac0e-1df6-4d2f-9854-bf555676cfb3" containerName="revoke-keys"
Jul 16 21:13:03.837717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837613    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b14de5-b0c8-4721-badc-1e29ee24bc7f" containerName="revoke-keys"
Jul 16 21:13:03.837717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837618    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b14de5-b0c8-4721-badc-1e29ee24bc7f" containerName="revoke-keys"
Jul 16 21:13:03.837717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837629    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d850ac0e-1df6-4d2f-9854-bf555676cfb3" containerName="revoke-keys"
Jul 16 21:13:03.837717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837634    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="d850ac0e-1df6-4d2f-9854-bf555676cfb3" containerName="revoke-keys"
Jul 16 21:13:03.837717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837648    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="122afb43-17f0-4ea7-9a37-6efeff6b41de" containerName="revoke-keys"
Jul 16 21:13:03.837717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837653    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="122afb43-17f0-4ea7-9a37-6efeff6b41de" containerName="revoke-keys"
Jul 16 21:13:03.837717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837665    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b14de5-b0c8-4721-badc-1e29ee24bc7f" containerName="revoke-keys"
Jul 16 21:13:03.837717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837670    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b14de5-b0c8-4721-badc-1e29ee24bc7f" containerName="revoke-keys"
Jul 16 21:13:03.837717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837676    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="122afb43-17f0-4ea7-9a37-6efeff6b41de" containerName="revoke-keys"
Jul 16 21:13:03.837717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837681    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="122afb43-17f0-4ea7-9a37-6efeff6b41de" containerName="revoke-keys"
Jul 16 21:13:03.838088 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837729    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="d850ac0e-1df6-4d2f-9854-bf555676cfb3" containerName="revoke-keys"
Jul 16 21:13:03.838088 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837736    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="122afb43-17f0-4ea7-9a37-6efeff6b41de" containerName="revoke-keys"
Jul 16 21:13:03.838088 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837742    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="122afb43-17f0-4ea7-9a37-6efeff6b41de" containerName="revoke-keys"
Jul 16 21:13:03.838088 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837747    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="04b14de5-b0c8-4721-badc-1e29ee24bc7f" containerName="revoke-keys"
Jul 16 21:13:03.838088 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837752    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="d850ac0e-1df6-4d2f-9854-bf555676cfb3" containerName="revoke-keys"
Jul 16 21:13:03.838088 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.837759    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="04b14de5-b0c8-4721-badc-1e29ee24bc7f" containerName="revoke-keys"
Jul 16 21:13:03.840529 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.840511    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86"
Jul 16 21:13:03.843019 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.842997    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:13:03.848170 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.848150    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86"]
Jul 16 21:13:03.936095 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:03.936059    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac190c5f-ac04-4ad4-8890-aeb05de987eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86\" (UID: \"ac190c5f-ac04-4ad4-8890-aeb05de987eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86"
Jul 16 21:13:04.036461 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:04.036431    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac190c5f-ac04-4ad4-8890-aeb05de987eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86\" (UID: \"ac190c5f-ac04-4ad4-8890-aeb05de987eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86"
Jul 16 21:13:04.037019 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:04.037000    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac190c5f-ac04-4ad4-8890-aeb05de987eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86\" (UID: \"ac190c5f-ac04-4ad4-8890-aeb05de987eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86"
Jul 16 21:13:04.151721 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:04.151633    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86"
Jul 16 21:13:04.279364 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:04.279199    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86"]
Jul 16 21:13:04.282168 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:13:04.282143    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac190c5f_ac04_4ad4_8890_aeb05de987eb.slice/crio-434bc8cd61e326e13191113068c8bd869a820540d8bc38e6b3534a85a9c3eb0e WatchSource:0}: Error finding container 434bc8cd61e326e13191113068c8bd869a820540d8bc38e6b3534a85a9c3eb0e: Status 404 returned error can't find the container with id 434bc8cd61e326e13191113068c8bd869a820540d8bc38e6b3534a85a9c3eb0e
Jul 16 21:13:05.146130 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:05.146015    2530 generic.go:356] "Generic (PLEG): container finished" podID="ac190c5f-ac04-4ad4-8890-aeb05de987eb" containerID="d3a84767714f11657d0f62b5479fd59bb0a8ba572d5daaa35d2a77df3164d05b" exitCode=6
Jul 16 21:13:05.146130 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:05.146062    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86" event={"ID":"ac190c5f-ac04-4ad4-8890-aeb05de987eb","Type":"ContainerDied","Data":"d3a84767714f11657d0f62b5479fd59bb0a8ba572d5daaa35d2a77df3164d05b"}
Jul 16 21:13:05.146624 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:05.146150    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86" event={"ID":"ac190c5f-ac04-4ad4-8890-aeb05de987eb","Type":"ContainerStarted","Data":"434bc8cd61e326e13191113068c8bd869a820540d8bc38e6b3534a85a9c3eb0e"}
Jul 16 21:13:05.146624 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:05.146305    2530 scope.go:117] "RemoveContainer" containerID="d3a84767714f11657d0f62b5479fd59bb0a8ba572d5daaa35d2a77df3164d05b"
Jul 16 21:13:06.153496 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:06.153406    2530 generic.go:356] "Generic (PLEG): container finished" podID="ac190c5f-ac04-4ad4-8890-aeb05de987eb" containerID="b25161060102f5d65f86ae9940ea02f0cbe1213fba0d378ee5fb5627f8086135" exitCode=6
Jul 16 21:13:06.153496 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:06.153459    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86" event={"ID":"ac190c5f-ac04-4ad4-8890-aeb05de987eb","Type":"ContainerDied","Data":"b25161060102f5d65f86ae9940ea02f0cbe1213fba0d378ee5fb5627f8086135"}
Jul 16 21:13:06.153973 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:06.153514    2530 scope.go:117] "RemoveContainer" containerID="d3a84767714f11657d0f62b5479fd59bb0a8ba572d5daaa35d2a77df3164d05b"
Jul 16 21:13:06.153973 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:06.153687    2530 scope.go:117] "RemoveContainer" containerID="b25161060102f5d65f86ae9940ea02f0cbe1213fba0d378ee5fb5627f8086135"
Jul 16 21:13:06.153973 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:13:06.153891    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86_odh-ai-gateway-infra(ac190c5f-ac04-4ad4-8890-aeb05de987eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86" podUID="ac190c5f-ac04-4ad4-8890-aeb05de987eb"
Jul 16 21:13:07.159851 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:07.159822    2530 scope.go:117] "RemoveContainer" containerID="b25161060102f5d65f86ae9940ea02f0cbe1213fba0d378ee5fb5627f8086135"
Jul 16 21:13:07.160243 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:13:07.159983    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86_odh-ai-gateway-infra(ac190c5f-ac04-4ad4-8890-aeb05de987eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86" podUID="ac190c5f-ac04-4ad4-8890-aeb05de987eb"
Jul 16 21:13:08.502866 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:08.502829    2530 scope.go:117] "RemoveContainer" containerID="65b0d947539cb0b888831c71173e464f5431d9304a1a99e2f557eb9066623688"
Jul 16 21:13:08.503399 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:13:08.503038    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:13:11.188204 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.188168    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5"]
Jul 16 21:13:11.188917 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.188896    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="122afb43-17f0-4ea7-9a37-6efeff6b41de" containerName="revoke-keys"
Jul 16 21:13:11.189008 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.188919    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="122afb43-17f0-4ea7-9a37-6efeff6b41de" containerName="revoke-keys"
Jul 16 21:13:11.189008 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.188949    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b14de5-b0c8-4721-badc-1e29ee24bc7f" containerName="revoke-keys"
Jul 16 21:13:11.189008 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.188959    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b14de5-b0c8-4721-badc-1e29ee24bc7f" containerName="revoke-keys"
Jul 16 21:13:11.189008 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.188984    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d850ac0e-1df6-4d2f-9854-bf555676cfb3" containerName="revoke-keys"
Jul 16 21:13:11.189008 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.188994    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="d850ac0e-1df6-4d2f-9854-bf555676cfb3" containerName="revoke-keys"
Jul 16 21:13:11.189284 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.189182    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="122afb43-17f0-4ea7-9a37-6efeff6b41de" containerName="revoke-keys"
Jul 16 21:13:11.189284 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.189195    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="04b14de5-b0c8-4721-badc-1e29ee24bc7f" containerName="revoke-keys"
Jul 16 21:13:11.189284 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.189204    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="d850ac0e-1df6-4d2f-9854-bf555676cfb3" containerName="revoke-keys"
Jul 16 21:13:11.194056 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.194033    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5"
Jul 16 21:13:11.197056 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.197031    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5"]
Jul 16 21:13:11.287792 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.287764    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5987aea-a2c5-4d38-a881-f757afdc5841-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5\" (UID: \"f5987aea-a2c5-4d38-a881-f757afdc5841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5"
Jul 16 21:13:11.388257 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.388228    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5987aea-a2c5-4d38-a881-f757afdc5841-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5\" (UID: \"f5987aea-a2c5-4d38-a881-f757afdc5841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5"
Jul 16 21:13:11.388845 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.388827    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5987aea-a2c5-4d38-a881-f757afdc5841-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5\" (UID: \"f5987aea-a2c5-4d38-a881-f757afdc5841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5"
Jul 16 21:13:11.507156 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.507135    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5"
Jul 16 21:13:11.841552 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.841528    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5"]
Jul 16 21:13:11.843480 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:13:11.843457    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5987aea_a2c5_4d38_a881_f757afdc5841.slice/crio-ea5622258dc5f58877ae82d3ae072c955237e9864d011b35b263393593b90ed4 WatchSource:0}: Error finding container ea5622258dc5f58877ae82d3ae072c955237e9864d011b35b263393593b90ed4: Status 404 returned error can't find the container with id ea5622258dc5f58877ae82d3ae072c955237e9864d011b35b263393593b90ed4
Jul 16 21:13:11.913029 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.912999    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5"]
Jul 16 21:13:11.917331 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.917303    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5"
Jul 16 21:13:11.925548 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.925513    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5"]
Jul 16 21:13:11.992765 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:11.992739    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e96b48c-0880-4a11-b660-18f10741dcf6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5\" (UID: \"9e96b48c-0880-4a11-b660-18f10741dcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5"
Jul 16 21:13:12.094183 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:12.094120    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e96b48c-0880-4a11-b660-18f10741dcf6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5\" (UID: \"9e96b48c-0880-4a11-b660-18f10741dcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5"
Jul 16 21:13:12.094648 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:12.094631    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e96b48c-0880-4a11-b660-18f10741dcf6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5\" (UID: \"9e96b48c-0880-4a11-b660-18f10741dcf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5"
Jul 16 21:13:12.183626 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:12.183594    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5" event={"ID":"f5987aea-a2c5-4d38-a881-f757afdc5841","Type":"ContainerStarted","Data":"0fe8f33f1a33dfc7fee7165695ebf1a552c08ca25e18e7d02ceee5d45c6f557a"}
Jul 16 21:13:12.183738 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:12.183643    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5" event={"ID":"f5987aea-a2c5-4d38-a881-f757afdc5841","Type":"ContainerStarted","Data":"ea5622258dc5f58877ae82d3ae072c955237e9864d011b35b263393593b90ed4"}
Jul 16 21:13:12.200857 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:12.200807    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5" podStartSLOduration=1.200791093 podStartE2EDuration="1.200791093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:13:12.200418847 +0000 UTC m=+2430.222006608" watchObservedRunningTime="2026-07-16 21:13:12.200791093 +0000 UTC m=+2430.222378861"
Jul 16 21:13:12.227733 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:12.227713    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5"
Jul 16 21:13:12.354583 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:12.354560    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5"]
Jul 16 21:13:12.356424 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:13:12.356401    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e96b48c_0880_4a11_b660_18f10741dcf6.slice/crio-579458f068dc39613957db8314dbb3021a319c8a4f33b6f2bb72161dc19920ec WatchSource:0}: Error finding container 579458f068dc39613957db8314dbb3021a319c8a4f33b6f2bb72161dc19920ec: Status 404 returned error can't find the container with id 579458f068dc39613957db8314dbb3021a319c8a4f33b6f2bb72161dc19920ec
Jul 16 21:13:13.189804 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:13.189725    2530 generic.go:356] "Generic (PLEG): container finished" podID="f5987aea-a2c5-4d38-a881-f757afdc5841" containerID="0fe8f33f1a33dfc7fee7165695ebf1a552c08ca25e18e7d02ceee5d45c6f557a" exitCode=6
Jul 16 21:13:13.189940 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:13.189816    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5" event={"ID":"f5987aea-a2c5-4d38-a881-f757afdc5841","Type":"ContainerDied","Data":"0fe8f33f1a33dfc7fee7165695ebf1a552c08ca25e18e7d02ceee5d45c6f557a"}
Jul 16 21:13:13.190010 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:13.189968    2530 scope.go:117] "RemoveContainer" containerID="0fe8f33f1a33dfc7fee7165695ebf1a552c08ca25e18e7d02ceee5d45c6f557a"
Jul 16 21:13:13.191387 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:13.191363    2530 generic.go:356] "Generic (PLEG): container finished" podID="9e96b48c-0880-4a11-b660-18f10741dcf6" containerID="245bb5be184c255c0467b50801b67c9c75df2b9278f36ad36a8d7942258cddb0" exitCode=6
Jul 16 21:13:13.191526 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:13.191423    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5" event={"ID":"9e96b48c-0880-4a11-b660-18f10741dcf6","Type":"ContainerDied","Data":"245bb5be184c255c0467b50801b67c9c75df2b9278f36ad36a8d7942258cddb0"}
Jul 16 21:13:13.191526 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:13.191452    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5" event={"ID":"9e96b48c-0880-4a11-b660-18f10741dcf6","Type":"ContainerStarted","Data":"579458f068dc39613957db8314dbb3021a319c8a4f33b6f2bb72161dc19920ec"}
Jul 16 21:13:13.191652 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:13.191607    2530 scope.go:117] "RemoveContainer" containerID="245bb5be184c255c0467b50801b67c9c75df2b9278f36ad36a8d7942258cddb0"
Jul 16 21:13:14.198531 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:14.198446    2530 generic.go:356] "Generic (PLEG): container finished" podID="f5987aea-a2c5-4d38-a881-f757afdc5841" containerID="cf6b41cd5c3d5f26b7fcd64a0189122747739e7aaf68b8f9d4237a5796170e52" exitCode=6
Jul 16 21:13:14.198947 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:14.198527    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5" event={"ID":"f5987aea-a2c5-4d38-a881-f757afdc5841","Type":"ContainerDied","Data":"cf6b41cd5c3d5f26b7fcd64a0189122747739e7aaf68b8f9d4237a5796170e52"}
Jul 16 21:13:14.198947 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:14.198579    2530 scope.go:117] "RemoveContainer" containerID="0fe8f33f1a33dfc7fee7165695ebf1a552c08ca25e18e7d02ceee5d45c6f557a"
Jul 16 21:13:14.198947 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:14.198751    2530 scope.go:117] "RemoveContainer" containerID="cf6b41cd5c3d5f26b7fcd64a0189122747739e7aaf68b8f9d4237a5796170e52"
Jul 16 21:13:14.199121 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:13:14.198963    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5_odh-ai-gateway-infra(f5987aea-a2c5-4d38-a881-f757afdc5841)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5" podUID="f5987aea-a2c5-4d38-a881-f757afdc5841"
Jul 16 21:13:14.200473 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:14.200448    2530 generic.go:356] "Generic (PLEG): container finished" podID="9e96b48c-0880-4a11-b660-18f10741dcf6" containerID="ca40726089e85bc4be70a63bc764d53566eb438bb3da3051d32722dc841e16c9" exitCode=6
Jul 16 21:13:14.200567 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:14.200503    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5" event={"ID":"9e96b48c-0880-4a11-b660-18f10741dcf6","Type":"ContainerDied","Data":"ca40726089e85bc4be70a63bc764d53566eb438bb3da3051d32722dc841e16c9"}
Jul 16 21:13:14.200691 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:14.200678    2530 scope.go:117] "RemoveContainer" containerID="ca40726089e85bc4be70a63bc764d53566eb438bb3da3051d32722dc841e16c9"
Jul 16 21:13:14.200831 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:13:14.200816    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5_odh-ai-gateway-infra(9e96b48c-0880-4a11-b660-18f10741dcf6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5" podUID="9e96b48c-0880-4a11-b660-18f10741dcf6"
Jul 16 21:13:14.209213 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:14.209197    2530 scope.go:117] "RemoveContainer" containerID="245bb5be184c255c0467b50801b67c9c75df2b9278f36ad36a8d7942258cddb0"
Jul 16 21:13:15.206402 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:15.206362    2530 scope.go:117] "RemoveContainer" containerID="cf6b41cd5c3d5f26b7fcd64a0189122747739e7aaf68b8f9d4237a5796170e52"
Jul 16 21:13:15.206872 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:13:15.206591    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5_odh-ai-gateway-infra(f5987aea-a2c5-4d38-a881-f757afdc5841)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5" podUID="f5987aea-a2c5-4d38-a881-f757afdc5841"
Jul 16 21:13:15.207904 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:15.207885    2530 scope.go:117] "RemoveContainer" containerID="ca40726089e85bc4be70a63bc764d53566eb438bb3da3051d32722dc841e16c9"
Jul 16 21:13:15.208051 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:13:15.208031    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5_odh-ai-gateway-infra(9e96b48c-0880-4a11-b660-18f10741dcf6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5" podUID="9e96b48c-0880-4a11-b660-18f10741dcf6"
Jul 16 21:13:19.501936 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:19.501901    2530 scope.go:117] "RemoveContainer" containerID="65b0d947539cb0b888831c71173e464f5431d9304a1a99e2f557eb9066623688"
Jul 16 21:13:19.502389 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:13:19.502153    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-90b68ec7-7d88b68566-47twm_odh-ai-gateway-infra(b25e21cd-ccbd-4870-99aa-ba1b5dc98083)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083"
Jul 16 21:13:22.504433 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:22.504402    2530 scope.go:117] "RemoveContainer" containerID="b25161060102f5d65f86ae9940ea02f0cbe1213fba0d378ee5fb5627f8086135"
Jul 16 21:13:23.246983 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:23.246953    2530 generic.go:356] "Generic (PLEG): container finished" podID="ac190c5f-ac04-4ad4-8890-aeb05de987eb" containerID="3a60986ee74aaa643eb3eeabf4d726cd7354aa194a25124b5eeb92485e5f2c0e" exitCode=6
Jul 16 21:13:23.247177 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:23.247033    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86" event={"ID":"ac190c5f-ac04-4ad4-8890-aeb05de987eb","Type":"ContainerDied","Data":"3a60986ee74aaa643eb3eeabf4d726cd7354aa194a25124b5eeb92485e5f2c0e"}
Jul 16 21:13:23.247177 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:23.247113    2530 scope.go:117] "RemoveContainer" containerID="b25161060102f5d65f86ae9940ea02f0cbe1213fba0d378ee5fb5627f8086135"
Jul 16 21:13:23.247400 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:23.247379    2530 scope.go:117] "RemoveContainer" containerID="3a60986ee74aaa643eb3eeabf4d726cd7354aa194a25124b5eeb92485e5f2c0e"
Jul 16 21:13:23.247613 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:13:23.247595    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86_odh-ai-gateway-infra(ac190c5f-ac04-4ad4-8890-aeb05de987eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86" podUID="ac190c5f-ac04-4ad4-8890-aeb05de987eb"
Jul 16 21:13:24.272931 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:24.272894    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86"]
Jul 16 21:13:24.424366 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:24.424346    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86"
Jul 16 21:13:24.490563 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:24.490538    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac190c5f-ac04-4ad4-8890-aeb05de987eb-maas-api-service-ca\") pod \"ac190c5f-ac04-4ad4-8890-aeb05de987eb\" (UID: \"ac190c5f-ac04-4ad4-8890-aeb05de987eb\") "
Jul 16 21:13:24.490861 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:24.490841    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac190c5f-ac04-4ad4-8890-aeb05de987eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac190c5f-ac04-4ad4-8890-aeb05de987eb" (UID: "ac190c5f-ac04-4ad4-8890-aeb05de987eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:13:24.591828 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:24.591772    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac190c5f-ac04-4ad4-8890-aeb05de987eb-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:13:24.954584 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:24.954553    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"]
Jul 16 21:13:25.083124 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.083069    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:13:25.197512 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.197473    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn5c7\" (UniqueName: \"kubernetes.io/projected/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-kube-api-access-jn5c7\") pod \"b25e21cd-ccbd-4870-99aa-ba1b5dc98083\" (UID: \"b25e21cd-ccbd-4870-99aa-ba1b5dc98083\") "
Jul 16 21:13:25.197693 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.197560    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-maas-api-tls\") pod \"b25e21cd-ccbd-4870-99aa-ba1b5dc98083\" (UID: \"b25e21cd-ccbd-4870-99aa-ba1b5dc98083\") "
Jul 16 21:13:25.197693 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.197617    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-openshift-service-ca\") pod \"b25e21cd-ccbd-4870-99aa-ba1b5dc98083\" (UID: \"b25e21cd-ccbd-4870-99aa-ba1b5dc98083\") "
Jul 16 21:13:25.197988 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.197959    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b25e21cd-ccbd-4870-99aa-ba1b5dc98083" (UID: "b25e21cd-ccbd-4870-99aa-ba1b5dc98083"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:13:25.199774 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.199748    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-kube-api-access-jn5c7" (OuterVolumeSpecName: "kube-api-access-jn5c7") pod "b25e21cd-ccbd-4870-99aa-ba1b5dc98083" (UID: "b25e21cd-ccbd-4870-99aa-ba1b5dc98083"). InnerVolumeSpecName "kube-api-access-jn5c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:13:25.199862 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.199770    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b25e21cd-ccbd-4870-99aa-ba1b5dc98083" (UID: "b25e21cd-ccbd-4870-99aa-ba1b5dc98083"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:13:25.258760 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.258697    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"
Jul 16 21:13:25.258760 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.258719    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm" event={"ID":"b25e21cd-ccbd-4870-99aa-ba1b5dc98083","Type":"ContainerDied","Data":"293ecd5094b7d0f8d9714d9a8033e976cae1ad31292077e367e4ab00236b3c47"}
Jul 16 21:13:25.258923 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.258808    2530 scope.go:117] "RemoveContainer" containerID="65b0d947539cb0b888831c71173e464f5431d9304a1a99e2f557eb9066623688"
Jul 16 21:13:25.260284 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.260263    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86" event={"ID":"ac190c5f-ac04-4ad4-8890-aeb05de987eb","Type":"ContainerDied","Data":"434bc8cd61e326e13191113068c8bd869a820540d8bc38e6b3534a85a9c3eb0e"}
Jul 16 21:13:25.260374 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.260313    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86"
Jul 16 21:13:25.267532 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.267513    2530 scope.go:117] "RemoveContainer" containerID="3a60986ee74aaa643eb3eeabf4d726cd7354aa194a25124b5eeb92485e5f2c0e"
Jul 16 21:13:25.275790 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.275769    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86"]
Jul 16 21:13:25.284369 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.284349    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-b8r86"]
Jul 16 21:13:25.295757 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.295735    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"]
Jul 16 21:13:25.298272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.298244    2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-openshift-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:13:25.298272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.298266    2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn5c7\" (UniqueName: \"kubernetes.io/projected/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-kube-api-access-jn5c7\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:13:25.298390 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.298276    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25e21cd-ccbd-4870-99aa-ba1b5dc98083-maas-api-tls\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:13:25.300424 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:25.300408    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-90b68ec7-7d88b68566-47twm"]
Jul 16 21:13:26.502662 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:26.502631    2530 scope.go:117] "RemoveContainer" containerID="ca40726089e85bc4be70a63bc764d53566eb438bb3da3051d32722dc841e16c9"
Jul 16 21:13:26.508316 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:26.508287    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac190c5f-ac04-4ad4-8890-aeb05de987eb" path="/var/lib/kubelet/pods/ac190c5f-ac04-4ad4-8890-aeb05de987eb/volumes"
Jul 16 21:13:26.508682 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:26.508667    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" path="/var/lib/kubelet/pods/b25e21cd-ccbd-4870-99aa-ba1b5dc98083/volumes"
Jul 16 21:13:27.273390 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:27.273363    2530 generic.go:356] "Generic (PLEG): container finished" podID="9e96b48c-0880-4a11-b660-18f10741dcf6" containerID="96dad1ad7a17d4472d0e3e3c3c06cfb626d5cb75854d4b8e113f10444116e79a" exitCode=6
Jul 16 21:13:27.273500 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:27.273446    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5" event={"ID":"9e96b48c-0880-4a11-b660-18f10741dcf6","Type":"ContainerDied","Data":"96dad1ad7a17d4472d0e3e3c3c06cfb626d5cb75854d4b8e113f10444116e79a"}
Jul 16 21:13:27.273575 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:27.273522    2530 scope.go:117] "RemoveContainer" containerID="ca40726089e85bc4be70a63bc764d53566eb438bb3da3051d32722dc841e16c9"
Jul 16 21:13:27.273824 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:27.273804    2530 scope.go:117] "RemoveContainer" containerID="96dad1ad7a17d4472d0e3e3c3c06cfb626d5cb75854d4b8e113f10444116e79a"
Jul 16 21:13:27.274011 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:13:27.273987    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5_odh-ai-gateway-infra(9e96b48c-0880-4a11-b660-18f10741dcf6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5" podUID="9e96b48c-0880-4a11-b660-18f10741dcf6"
Jul 16 21:13:28.297280 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:28.297248    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5"]
Jul 16 21:13:28.449523 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:28.449504    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5"
Jul 16 21:13:28.534598 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:28.534574    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e96b48c-0880-4a11-b660-18f10741dcf6-maas-api-service-ca\") pod \"9e96b48c-0880-4a11-b660-18f10741dcf6\" (UID: \"9e96b48c-0880-4a11-b660-18f10741dcf6\") "
Jul 16 21:13:28.534897 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:28.534877    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e96b48c-0880-4a11-b660-18f10741dcf6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9e96b48c-0880-4a11-b660-18f10741dcf6" (UID: "9e96b48c-0880-4a11-b660-18f10741dcf6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:13:28.636030 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:28.635969    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e96b48c-0880-4a11-b660-18f10741dcf6-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:13:29.284564 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:29.284524    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5" event={"ID":"9e96b48c-0880-4a11-b660-18f10741dcf6","Type":"ContainerDied","Data":"579458f068dc39613957db8314dbb3021a319c8a4f33b6f2bb72161dc19920ec"}
Jul 16 21:13:29.284564 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:29.284555    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5"
Jul 16 21:13:29.284809 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:29.284606    2530 scope.go:117] "RemoveContainer" containerID="96dad1ad7a17d4472d0e3e3c3c06cfb626d5cb75854d4b8e113f10444116e79a"
Jul 16 21:13:29.309236 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:29.309207    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5"]
Jul 16 21:13:29.314549 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:29.314528    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-szct5"]
Jul 16 21:13:29.502826 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:29.502801    2530 scope.go:117] "RemoveContainer" containerID="cf6b41cd5c3d5f26b7fcd64a0189122747739e7aaf68b8f9d4237a5796170e52"
Jul 16 21:13:30.290751 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:30.290718    2530 generic.go:356] "Generic (PLEG): container finished" podID="f5987aea-a2c5-4d38-a881-f757afdc5841" containerID="1398d3a45fe8a6a1b7e02b2d08f4158a9df3eb8c4aa126a786a2a7da31c2e189" exitCode=6
Jul 16 21:13:30.290860 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:30.290755    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5" event={"ID":"f5987aea-a2c5-4d38-a881-f757afdc5841","Type":"ContainerDied","Data":"1398d3a45fe8a6a1b7e02b2d08f4158a9df3eb8c4aa126a786a2a7da31c2e189"}
Jul 16 21:13:30.290860 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:30.290827    2530 scope.go:117] "RemoveContainer" containerID="cf6b41cd5c3d5f26b7fcd64a0189122747739e7aaf68b8f9d4237a5796170e52"
Jul 16 21:13:30.291176 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:30.291156    2530 scope.go:117] "RemoveContainer" containerID="1398d3a45fe8a6a1b7e02b2d08f4158a9df3eb8c4aa126a786a2a7da31c2e189"
Jul 16 21:13:30.291363 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:13:30.291339    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5_odh-ai-gateway-infra(f5987aea-a2c5-4d38-a881-f757afdc5841)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5" podUID="f5987aea-a2c5-4d38-a881-f757afdc5841"
Jul 16 21:13:30.507992 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:30.507960    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e96b48c-0880-4a11-b660-18f10741dcf6" path="/var/lib/kubelet/pods/9e96b48c-0880-4a11-b660-18f10741dcf6/volumes"
Jul 16 21:13:31.321001 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:31.320967    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5"]
Jul 16 21:13:31.455997 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:31.455972    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5"
Jul 16 21:13:31.559332 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:31.559304    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5987aea-a2c5-4d38-a881-f757afdc5841-maas-api-service-ca\") pod \"f5987aea-a2c5-4d38-a881-f757afdc5841\" (UID: \"f5987aea-a2c5-4d38-a881-f757afdc5841\") "
Jul 16 21:13:31.559692 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:31.559665    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5987aea-a2c5-4d38-a881-f757afdc5841-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5987aea-a2c5-4d38-a881-f757afdc5841" (UID: "f5987aea-a2c5-4d38-a881-f757afdc5841"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:13:31.559853 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:31.559832    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5987aea-a2c5-4d38-a881-f757afdc5841-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:13:32.303234 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:32.303191    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5" event={"ID":"f5987aea-a2c5-4d38-a881-f757afdc5841","Type":"ContainerDied","Data":"ea5622258dc5f58877ae82d3ae072c955237e9864d011b35b263393593b90ed4"}
Jul 16 21:13:32.303234 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:32.303219    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5"
Jul 16 21:13:32.303496 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:32.303267    2530 scope.go:117] "RemoveContainer" containerID="1398d3a45fe8a6a1b7e02b2d08f4158a9df3eb8c4aa126a786a2a7da31c2e189"
Jul 16 21:13:32.324411 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:32.324374    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5"]
Jul 16 21:13:32.329902 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:32.329870    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-nzxk5"]
Jul 16 21:13:32.508040 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:13:32.508012    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5987aea-a2c5-4d38-a881-f757afdc5841" path="/var/lib/kubelet/pods/f5987aea-a2c5-4d38-a881-f757afdc5841/volumes"
Jul 16 21:14:01.967346 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967306    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx"]
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967622    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac190c5f-ac04-4ad4-8890-aeb05de987eb" containerName="revoke-keys"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967633    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac190c5f-ac04-4ad4-8890-aeb05de987eb" containerName="revoke-keys"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967641    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerName="maas-api"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967646    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerName="maas-api"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967653    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e96b48c-0880-4a11-b660-18f10741dcf6" containerName="revoke-keys"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967658    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e96b48c-0880-4a11-b660-18f10741dcf6" containerName="revoke-keys"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967666    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerName="maas-api"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967671    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerName="maas-api"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967681    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac190c5f-ac04-4ad4-8890-aeb05de987eb" containerName="revoke-keys"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967686    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac190c5f-ac04-4ad4-8890-aeb05de987eb" containerName="revoke-keys"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967693    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5987aea-a2c5-4d38-a881-f757afdc5841" containerName="revoke-keys"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967698    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5987aea-a2c5-4d38-a881-f757afdc5841" containerName="revoke-keys"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967705    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerName="maas-api"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967709    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerName="maas-api"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967716    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e96b48c-0880-4a11-b660-18f10741dcf6" containerName="revoke-keys"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967721    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e96b48c-0880-4a11-b660-18f10741dcf6" containerName="revoke-keys"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967726    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5987aea-a2c5-4d38-a881-f757afdc5841" containerName="revoke-keys"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967731    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5987aea-a2c5-4d38-a881-f757afdc5841" containerName="revoke-keys"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967737    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e96b48c-0880-4a11-b660-18f10741dcf6" containerName="revoke-keys"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967742    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e96b48c-0880-4a11-b660-18f10741dcf6" containerName="revoke-keys"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967755    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5987aea-a2c5-4d38-a881-f757afdc5841" containerName="revoke-keys"
Jul 16 21:14:01.967749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967760    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5987aea-a2c5-4d38-a881-f757afdc5841" containerName="revoke-keys"
Jul 16 21:14:01.968432 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967809    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerName="maas-api"
Jul 16 21:14:01.968432 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967815    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5987aea-a2c5-4d38-a881-f757afdc5841" containerName="revoke-keys"
Jul 16 21:14:01.968432 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967822    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e96b48c-0880-4a11-b660-18f10741dcf6" containerName="revoke-keys"
Jul 16 21:14:01.968432 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967828    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5987aea-a2c5-4d38-a881-f757afdc5841" containerName="revoke-keys"
Jul 16 21:14:01.968432 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967833    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerName="maas-api"
Jul 16 21:14:01.968432 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967838    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerName="maas-api"
Jul 16 21:14:01.968432 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967843    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e96b48c-0880-4a11-b660-18f10741dcf6" containerName="revoke-keys"
Jul 16 21:14:01.968432 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967849    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e96b48c-0880-4a11-b660-18f10741dcf6" containerName="revoke-keys"
Jul 16 21:14:01.968432 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967853    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac190c5f-ac04-4ad4-8890-aeb05de987eb" containerName="revoke-keys"
Jul 16 21:14:01.968432 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967858    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac190c5f-ac04-4ad4-8890-aeb05de987eb" containerName="revoke-keys"
Jul 16 21:14:01.968432 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.967864    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac190c5f-ac04-4ad4-8890-aeb05de987eb" containerName="revoke-keys"
Jul 16 21:14:01.970859 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.970843    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx"
Jul 16 21:14:01.973483 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.973463    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:14:01.978366 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:01.978341    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx"]
Jul 16 21:14:02.014389 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:02.014358    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0099a5bb-0c25-42fa-be9b-a4d36a893742-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx\" (UID: \"0099a5bb-0c25-42fa-be9b-a4d36a893742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx"
Jul 16 21:14:02.115650 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:02.115617    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0099a5bb-0c25-42fa-be9b-a4d36a893742-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx\" (UID: \"0099a5bb-0c25-42fa-be9b-a4d36a893742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx"
Jul 16 21:14:02.116314 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:02.116295    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0099a5bb-0c25-42fa-be9b-a4d36a893742-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx\" (UID: \"0099a5bb-0c25-42fa-be9b-a4d36a893742\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx"
Jul 16 21:14:02.282701 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:02.282612    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx"
Jul 16 21:14:02.417225 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:02.417186    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx"]
Jul 16 21:14:02.419038 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:14:02.419009    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0099a5bb_0c25_42fa_be9b_a4d36a893742.slice/crio-865771d674b712f4320f43c6880f210d6e038e8386158ba14793fa6d1654e162 WatchSource:0}: Error finding container 865771d674b712f4320f43c6880f210d6e038e8386158ba14793fa6d1654e162: Status 404 returned error can't find the container with id 865771d674b712f4320f43c6880f210d6e038e8386158ba14793fa6d1654e162
Jul 16 21:14:02.428765 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:02.428737    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx" event={"ID":"0099a5bb-0c25-42fa-be9b-a4d36a893742","Type":"ContainerStarted","Data":"865771d674b712f4320f43c6880f210d6e038e8386158ba14793fa6d1654e162"}
Jul 16 21:14:03.435614 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:03.435509    2530 generic.go:356] "Generic (PLEG): container finished" podID="0099a5bb-0c25-42fa-be9b-a4d36a893742" containerID="05c4312ee7b0e3cdd6208a0000a731583d546139585d649cf548d23ca8428ad0" exitCode=6
Jul 16 21:14:03.435614 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:03.435547    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx" event={"ID":"0099a5bb-0c25-42fa-be9b-a4d36a893742","Type":"ContainerDied","Data":"05c4312ee7b0e3cdd6208a0000a731583d546139585d649cf548d23ca8428ad0"}
Jul 16 21:14:03.436080 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:03.435802    2530 scope.go:117] "RemoveContainer" containerID="05c4312ee7b0e3cdd6208a0000a731583d546139585d649cf548d23ca8428ad0"
Jul 16 21:14:03.880186 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:03.880145    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5"]
Jul 16 21:14:03.880538 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:03.880521    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerName="maas-api"
Jul 16 21:14:03.880630 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:03.880541    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerName="maas-api"
Jul 16 21:14:03.880630 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:03.880586    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac190c5f-ac04-4ad4-8890-aeb05de987eb" containerName="revoke-keys"
Jul 16 21:14:03.880630 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:03.880596    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac190c5f-ac04-4ad4-8890-aeb05de987eb" containerName="revoke-keys"
Jul 16 21:14:03.880781 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:03.880680    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerName="maas-api"
Jul 16 21:14:03.880781 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:03.880694    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5987aea-a2c5-4d38-a881-f757afdc5841" containerName="revoke-keys"
Jul 16 21:14:03.883654 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:03.883631    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5"
Jul 16 21:14:03.890292 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:03.890263    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5"]
Jul 16 21:14:03.926982 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:03.926952    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c11ba8-188f-4e12-827a-afc60a1cf682-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5\" (UID: \"e5c11ba8-188f-4e12-827a-afc60a1cf682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5"
Jul 16 21:14:04.033149 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:04.028899    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c11ba8-188f-4e12-827a-afc60a1cf682-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5\" (UID: \"e5c11ba8-188f-4e12-827a-afc60a1cf682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5"
Jul 16 21:14:04.033149 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:04.029753    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c11ba8-188f-4e12-827a-afc60a1cf682-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5\" (UID: \"e5c11ba8-188f-4e12-827a-afc60a1cf682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5"
Jul 16 21:14:04.194967 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:04.194934    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5"
Jul 16 21:14:04.322780 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:04.322737    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5"]
Jul 16 21:14:04.324976 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:14:04.324947    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c11ba8_188f_4e12_827a_afc60a1cf682.slice/crio-d45d1fb64d5e1ba33bac6c5749f0d49c308d523d34d504ad7970e2114887805d WatchSource:0}: Error finding container d45d1fb64d5e1ba33bac6c5749f0d49c308d523d34d504ad7970e2114887805d: Status 404 returned error can't find the container with id d45d1fb64d5e1ba33bac6c5749f0d49c308d523d34d504ad7970e2114887805d
Jul 16 21:14:04.442195 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:04.442156    2530 generic.go:356] "Generic (PLEG): container finished" podID="0099a5bb-0c25-42fa-be9b-a4d36a893742" containerID="c0bd04afba981d47d815db1b9495ddeffd088399f44ac8034af8d7a3acd612b4" exitCode=6
Jul 16 21:14:04.442633 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:04.442239    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx" event={"ID":"0099a5bb-0c25-42fa-be9b-a4d36a893742","Type":"ContainerDied","Data":"c0bd04afba981d47d815db1b9495ddeffd088399f44ac8034af8d7a3acd612b4"}
Jul 16 21:14:04.442633 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:04.442311    2530 scope.go:117] "RemoveContainer" containerID="05c4312ee7b0e3cdd6208a0000a731583d546139585d649cf548d23ca8428ad0"
Jul 16 21:14:04.442633 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:04.442442    2530 scope.go:117] "RemoveContainer" containerID="c0bd04afba981d47d815db1b9495ddeffd088399f44ac8034af8d7a3acd612b4"
Jul 16 21:14:04.442821 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:14:04.442678    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx_odh-ai-gateway-infra(0099a5bb-0c25-42fa-be9b-a4d36a893742)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx" podUID="0099a5bb-0c25-42fa-be9b-a4d36a893742"
Jul 16 21:14:04.443652 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:04.443472    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5" event={"ID":"e5c11ba8-188f-4e12-827a-afc60a1cf682","Type":"ContainerStarted","Data":"d45d1fb64d5e1ba33bac6c5749f0d49c308d523d34d504ad7970e2114887805d"}
Jul 16 21:14:05.449347 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:05.449251    2530 scope.go:117] "RemoveContainer" containerID="c0bd04afba981d47d815db1b9495ddeffd088399f44ac8034af8d7a3acd612b4"
Jul 16 21:14:05.449753 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:14:05.449491    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx_odh-ai-gateway-infra(0099a5bb-0c25-42fa-be9b-a4d36a893742)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx" podUID="0099a5bb-0c25-42fa-be9b-a4d36a893742"
Jul 16 21:14:05.450512 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:05.450477    2530 generic.go:356] "Generic (PLEG): container finished" podID="e5c11ba8-188f-4e12-827a-afc60a1cf682" containerID="d1d40047b51171f41856c5d375766d0f98d3338d981bae03febb172f464f5734" exitCode=6
Jul 16 21:14:05.450574 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:05.450550    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5" event={"ID":"e5c11ba8-188f-4e12-827a-afc60a1cf682","Type":"ContainerDied","Data":"d1d40047b51171f41856c5d375766d0f98d3338d981bae03febb172f464f5734"}
Jul 16 21:14:05.450679 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:05.450666    2530 scope.go:117] "RemoveContainer" containerID="d1d40047b51171f41856c5d375766d0f98d3338d981bae03febb172f464f5734"
Jul 16 21:14:06.457288 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:06.457252    2530 generic.go:356] "Generic (PLEG): container finished" podID="e5c11ba8-188f-4e12-827a-afc60a1cf682" containerID="e037a83e53a5e887bdde9699e0da4ef285f6ceddeb1ebaf9e53c83f0391edf89" exitCode=6
Jul 16 21:14:06.457732 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:06.457336    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5" event={"ID":"e5c11ba8-188f-4e12-827a-afc60a1cf682","Type":"ContainerDied","Data":"e037a83e53a5e887bdde9699e0da4ef285f6ceddeb1ebaf9e53c83f0391edf89"}
Jul 16 21:14:06.457732 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:06.457397    2530 scope.go:117] "RemoveContainer" containerID="d1d40047b51171f41856c5d375766d0f98d3338d981bae03febb172f464f5734"
Jul 16 21:14:06.457732 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:06.457593    2530 scope.go:117] "RemoveContainer" containerID="e037a83e53a5e887bdde9699e0da4ef285f6ceddeb1ebaf9e53c83f0391edf89"
Jul 16 21:14:06.457848 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:14:06.457813    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5_odh-ai-gateway-infra(e5c11ba8-188f-4e12-827a-afc60a1cf682)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5" podUID="e5c11ba8-188f-4e12-827a-afc60a1cf682"
Jul 16 21:14:07.463210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:07.463185    2530 scope.go:117] "RemoveContainer" containerID="e037a83e53a5e887bdde9699e0da4ef285f6ceddeb1ebaf9e53c83f0391edf89"
Jul 16 21:14:07.463587 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:14:07.463391    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5_odh-ai-gateway-infra(e5c11ba8-188f-4e12-827a-afc60a1cf682)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5" podUID="e5c11ba8-188f-4e12-827a-afc60a1cf682"
Jul 16 21:14:18.507759 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:18.507732    2530 scope.go:117] "RemoveContainer" containerID="c0bd04afba981d47d815db1b9495ddeffd088399f44ac8034af8d7a3acd612b4"
Jul 16 21:14:19.512318 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:19.512288    2530 generic.go:356] "Generic (PLEG): container finished" podID="0099a5bb-0c25-42fa-be9b-a4d36a893742" containerID="2518300031d7096bccfcc09090c78908af9d5474ae30e3ad6b51290c6a3defcf" exitCode=6
Jul 16 21:14:19.512318 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:19.512323    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx" event={"ID":"0099a5bb-0c25-42fa-be9b-a4d36a893742","Type":"ContainerDied","Data":"2518300031d7096bccfcc09090c78908af9d5474ae30e3ad6b51290c6a3defcf"}
Jul 16 21:14:19.512805 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:19.512378    2530 scope.go:117] "RemoveContainer" containerID="c0bd04afba981d47d815db1b9495ddeffd088399f44ac8034af8d7a3acd612b4"
Jul 16 21:14:19.512805 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:19.512643    2530 scope.go:117] "RemoveContainer" containerID="2518300031d7096bccfcc09090c78908af9d5474ae30e3ad6b51290c6a3defcf"
Jul 16 21:14:19.512887 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:14:19.512846    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx_odh-ai-gateway-infra(0099a5bb-0c25-42fa-be9b-a4d36a893742)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx" podUID="0099a5bb-0c25-42fa-be9b-a4d36a893742"
Jul 16 21:14:20.536342 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:20.536310    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx"]
Jul 16 21:14:20.666217 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:20.666194    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx"
Jul 16 21:14:20.778078 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:20.778042    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0099a5bb-0c25-42fa-be9b-a4d36a893742-maas-api-service-ca\") pod \"0099a5bb-0c25-42fa-be9b-a4d36a893742\" (UID: \"0099a5bb-0c25-42fa-be9b-a4d36a893742\") "
Jul 16 21:14:20.778439 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:20.778407    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0099a5bb-0c25-42fa-be9b-a4d36a893742-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0099a5bb-0c25-42fa-be9b-a4d36a893742" (UID: "0099a5bb-0c25-42fa-be9b-a4d36a893742"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:14:20.879071 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:20.878992    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0099a5bb-0c25-42fa-be9b-a4d36a893742-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:14:21.502369 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:21.502337    2530 scope.go:117] "RemoveContainer" containerID="e037a83e53a5e887bdde9699e0da4ef285f6ceddeb1ebaf9e53c83f0391edf89"
Jul 16 21:14:21.523254 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:21.523226    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx"
Jul 16 21:14:21.523420 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:21.523251    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx" event={"ID":"0099a5bb-0c25-42fa-be9b-a4d36a893742","Type":"ContainerDied","Data":"865771d674b712f4320f43c6880f210d6e038e8386158ba14793fa6d1654e162"}
Jul 16 21:14:21.523420 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:21.523313    2530 scope.go:117] "RemoveContainer" containerID="2518300031d7096bccfcc09090c78908af9d5474ae30e3ad6b51290c6a3defcf"
Jul 16 21:14:21.555019 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:21.554994    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx"]
Jul 16 21:14:21.560627 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:21.560601    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n79wx"]
Jul 16 21:14:22.507746 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:22.507712    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0099a5bb-0c25-42fa-be9b-a4d36a893742" path="/var/lib/kubelet/pods/0099a5bb-0c25-42fa-be9b-a4d36a893742/volumes"
Jul 16 21:14:22.530037 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:22.530007    2530 generic.go:356] "Generic (PLEG): container finished" podID="e5c11ba8-188f-4e12-827a-afc60a1cf682" containerID="34564ec92e7292db327d67bc3154effdc3462af171f3363d6e4855f928b6937e" exitCode=6
Jul 16 21:14:22.530201 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:22.530078    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5" event={"ID":"e5c11ba8-188f-4e12-827a-afc60a1cf682","Type":"ContainerDied","Data":"34564ec92e7292db327d67bc3154effdc3462af171f3363d6e4855f928b6937e"}
Jul 16 21:14:22.530201 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:22.530156    2530 scope.go:117] "RemoveContainer" containerID="e037a83e53a5e887bdde9699e0da4ef285f6ceddeb1ebaf9e53c83f0391edf89"
Jul 16 21:14:22.530425 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:22.530407    2530 scope.go:117] "RemoveContainer" containerID="34564ec92e7292db327d67bc3154effdc3462af171f3363d6e4855f928b6937e"
Jul 16 21:14:22.530625 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:14:22.530606    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5_odh-ai-gateway-infra(e5c11ba8-188f-4e12-827a-afc60a1cf682)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5" podUID="e5c11ba8-188f-4e12-827a-afc60a1cf682"
Jul 16 21:14:23.556969 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:23.556929    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5"]
Jul 16 21:14:23.684618 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:23.684592    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5"
Jul 16 21:14:23.805643 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:23.805604    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c11ba8-188f-4e12-827a-afc60a1cf682-maas-api-service-ca\") pod \"e5c11ba8-188f-4e12-827a-afc60a1cf682\" (UID: \"e5c11ba8-188f-4e12-827a-afc60a1cf682\") "
Jul 16 21:14:23.806030 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:23.806002    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c11ba8-188f-4e12-827a-afc60a1cf682-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5c11ba8-188f-4e12-827a-afc60a1cf682" (UID: "e5c11ba8-188f-4e12-827a-afc60a1cf682"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:14:23.907117 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:23.907024    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c11ba8-188f-4e12-827a-afc60a1cf682-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:14:24.542765 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:24.542737    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5"
Jul 16 21:14:24.542943 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:24.542737    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5" event={"ID":"e5c11ba8-188f-4e12-827a-afc60a1cf682","Type":"ContainerDied","Data":"d45d1fb64d5e1ba33bac6c5749f0d49c308d523d34d504ad7970e2114887805d"}
Jul 16 21:14:24.542943 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:24.542865    2530 scope.go:117] "RemoveContainer" containerID="34564ec92e7292db327d67bc3154effdc3462af171f3363d6e4855f928b6937e"
Jul 16 21:14:24.559635 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:24.559610    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5"]
Jul 16 21:14:24.563783 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:24.563761    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-rvjz5"]
Jul 16 21:14:26.507841 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:26.507808    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5c11ba8-188f-4e12-827a-afc60a1cf682" path="/var/lib/kubelet/pods/e5c11ba8-188f-4e12-827a-afc60a1cf682/volumes"
Jul 16 21:14:49.146978 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.146942    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c"]
Jul 16 21:14:49.147407 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.147289    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0099a5bb-0c25-42fa-be9b-a4d36a893742" containerName="revoke-keys"
Jul 16 21:14:49.147407 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.147301    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="0099a5bb-0c25-42fa-be9b-a4d36a893742" containerName="revoke-keys"
Jul 16 21:14:49.147407 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.147321    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerName="maas-api"
Jul 16 21:14:49.147407 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.147327    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerName="maas-api"
Jul 16 21:14:49.147407 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.147335    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c11ba8-188f-4e12-827a-afc60a1cf682" containerName="revoke-keys"
Jul 16 21:14:49.147407 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.147341    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c11ba8-188f-4e12-827a-afc60a1cf682" containerName="revoke-keys"
Jul 16 21:14:49.147407 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.147349    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c11ba8-188f-4e12-827a-afc60a1cf682" containerName="revoke-keys"
Jul 16 21:14:49.147407 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.147354    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c11ba8-188f-4e12-827a-afc60a1cf682" containerName="revoke-keys"
Jul 16 21:14:49.147407 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.147364    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c11ba8-188f-4e12-827a-afc60a1cf682" containerName="revoke-keys"
Jul 16 21:14:49.147407 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.147369    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c11ba8-188f-4e12-827a-afc60a1cf682" containerName="revoke-keys"
Jul 16 21:14:49.147407 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.147375    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0099a5bb-0c25-42fa-be9b-a4d36a893742" containerName="revoke-keys"
Jul 16 21:14:49.147407 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.147381    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="0099a5bb-0c25-42fa-be9b-a4d36a893742" containerName="revoke-keys"
Jul 16 21:14:49.147816 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.147443    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="0099a5bb-0c25-42fa-be9b-a4d36a893742" containerName="revoke-keys"
Jul 16 21:14:49.147816 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.147450    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c11ba8-188f-4e12-827a-afc60a1cf682" containerName="revoke-keys"
Jul 16 21:14:49.147816 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.147456    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="0099a5bb-0c25-42fa-be9b-a4d36a893742" containerName="revoke-keys"
Jul 16 21:14:49.147816 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.147461    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b25e21cd-ccbd-4870-99aa-ba1b5dc98083" containerName="maas-api"
Jul 16 21:14:49.147816 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.147470    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c11ba8-188f-4e12-827a-afc60a1cf682" containerName="revoke-keys"
Jul 16 21:14:49.147816 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.147483    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c11ba8-188f-4e12-827a-afc60a1cf682" containerName="revoke-keys"
Jul 16 21:14:49.150361 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.150345    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c"
Jul 16 21:14:49.152603 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.152581    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:14:49.156426 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.156403    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c"]
Jul 16 21:14:49.219801 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.219766    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a12b0d-dff6-42dc-90ea-026cb8095aad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c\" (UID: \"f2a12b0d-dff6-42dc-90ea-026cb8095aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c"
Jul 16 21:14:49.320884 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.320842    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a12b0d-dff6-42dc-90ea-026cb8095aad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c\" (UID: \"f2a12b0d-dff6-42dc-90ea-026cb8095aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c"
Jul 16 21:14:49.321496 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.321477    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a12b0d-dff6-42dc-90ea-026cb8095aad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c\" (UID: \"f2a12b0d-dff6-42dc-90ea-026cb8095aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c"
Jul 16 21:14:49.462844 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.462814    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c"
Jul 16 21:14:49.590154 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.590130    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c"]
Jul 16 21:14:49.592976 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:14:49.592946    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a12b0d_dff6_42dc_90ea_026cb8095aad.slice/crio-41c013dc8ae99949f489bbd581920789f7a17ca2b43ead944db85b0ae61520d9 WatchSource:0}: Error finding container 41c013dc8ae99949f489bbd581920789f7a17ca2b43ead944db85b0ae61520d9: Status 404 returned error can't find the container with id 41c013dc8ae99949f489bbd581920789f7a17ca2b43ead944db85b0ae61520d9
Jul 16 21:14:49.649209 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:49.649179    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c" event={"ID":"f2a12b0d-dff6-42dc-90ea-026cb8095aad","Type":"ContainerStarted","Data":"41c013dc8ae99949f489bbd581920789f7a17ca2b43ead944db85b0ae61520d9"}
Jul 16 21:14:50.655125 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:50.655009    2530 generic.go:356] "Generic (PLEG): container finished" podID="f2a12b0d-dff6-42dc-90ea-026cb8095aad" containerID="45d3632b490e83722bed6e8a53d5aa0d637e8dfde213008ba1e81b333d557395" exitCode=6
Jul 16 21:14:50.655125 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:50.655064    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c" event={"ID":"f2a12b0d-dff6-42dc-90ea-026cb8095aad","Type":"ContainerDied","Data":"45d3632b490e83722bed6e8a53d5aa0d637e8dfde213008ba1e81b333d557395"}
Jul 16 21:14:50.655532 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:50.655287    2530 scope.go:117] "RemoveContainer" containerID="45d3632b490e83722bed6e8a53d5aa0d637e8dfde213008ba1e81b333d557395"
Jul 16 21:14:51.661325 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:51.661237    2530 generic.go:356] "Generic (PLEG): container finished" podID="f2a12b0d-dff6-42dc-90ea-026cb8095aad" containerID="0f138a38181de9c99d697b5a2fefde7307fd39b5b4344b1431ff193e885b5ab4" exitCode=6
Jul 16 21:14:51.661694 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:51.661318    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c" event={"ID":"f2a12b0d-dff6-42dc-90ea-026cb8095aad","Type":"ContainerDied","Data":"0f138a38181de9c99d697b5a2fefde7307fd39b5b4344b1431ff193e885b5ab4"}
Jul 16 21:14:51.661694 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:51.661419    2530 scope.go:117] "RemoveContainer" containerID="45d3632b490e83722bed6e8a53d5aa0d637e8dfde213008ba1e81b333d557395"
Jul 16 21:14:51.661694 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:51.661656    2530 scope.go:117] "RemoveContainer" containerID="0f138a38181de9c99d697b5a2fefde7307fd39b5b4344b1431ff193e885b5ab4"
Jul 16 21:14:51.661873 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:14:51.661856    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c_odh-ai-gateway-infra(f2a12b0d-dff6-42dc-90ea-026cb8095aad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c" podUID="f2a12b0d-dff6-42dc-90ea-026cb8095aad"
Jul 16 21:14:52.018914 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:52.018866    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b"]
Jul 16 21:14:52.019408 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:52.019386    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0099a5bb-0c25-42fa-be9b-a4d36a893742" containerName="revoke-keys"
Jul 16 21:14:52.019468 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:52.019413    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="0099a5bb-0c25-42fa-be9b-a4d36a893742" containerName="revoke-keys"
Jul 16 21:14:52.019528 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:52.019515    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="0099a5bb-0c25-42fa-be9b-a4d36a893742" containerName="revoke-keys"
Jul 16 21:14:52.023531 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:52.023506    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b"
Jul 16 21:14:52.031532 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:52.031489    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b"]
Jul 16 21:14:52.042461 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:52.042429    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f251ab-c2d4-4f59-a4ca-82d74af6f748-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b\" (UID: \"f3f251ab-c2d4-4f59-a4ca-82d74af6f748\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b"
Jul 16 21:14:52.143449 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:52.143404    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f251ab-c2d4-4f59-a4ca-82d74af6f748-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b\" (UID: \"f3f251ab-c2d4-4f59-a4ca-82d74af6f748\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b"
Jul 16 21:14:52.144034 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:52.144011    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f251ab-c2d4-4f59-a4ca-82d74af6f748-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b\" (UID: \"f3f251ab-c2d4-4f59-a4ca-82d74af6f748\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b"
Jul 16 21:14:52.338325 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:52.338236    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b"
Jul 16 21:14:52.472589 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:52.472566    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b"]
Jul 16 21:14:52.474472 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:14:52.474445    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f251ab_c2d4_4f59_a4ca_82d74af6f748.slice/crio-84620a13b5f9bb7c8db0c472bb1df235f01ae518c0c0b6758435cea7e19ddd0a WatchSource:0}: Error finding container 84620a13b5f9bb7c8db0c472bb1df235f01ae518c0c0b6758435cea7e19ddd0a: Status 404 returned error can't find the container with id 84620a13b5f9bb7c8db0c472bb1df235f01ae518c0c0b6758435cea7e19ddd0a
Jul 16 21:14:52.666832 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:52.666803    2530 scope.go:117] "RemoveContainer" containerID="0f138a38181de9c99d697b5a2fefde7307fd39b5b4344b1431ff193e885b5ab4"
Jul 16 21:14:52.667261 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:14:52.667047    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c_odh-ai-gateway-infra(f2a12b0d-dff6-42dc-90ea-026cb8095aad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c" podUID="f2a12b0d-dff6-42dc-90ea-026cb8095aad"
Jul 16 21:14:52.667905 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:52.667884    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b" event={"ID":"f3f251ab-c2d4-4f59-a4ca-82d74af6f748","Type":"ContainerStarted","Data":"84620a13b5f9bb7c8db0c472bb1df235f01ae518c0c0b6758435cea7e19ddd0a"}
Jul 16 21:14:53.673623 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:53.673585    2530 generic.go:356] "Generic (PLEG): container finished" podID="f3f251ab-c2d4-4f59-a4ca-82d74af6f748" containerID="787ccfd410996b25ef71667d45424380df5c5842a5c6bd2b30fb6685ce7543c2" exitCode=6
Jul 16 21:14:53.674043 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:53.673633    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b" event={"ID":"f3f251ab-c2d4-4f59-a4ca-82d74af6f748","Type":"ContainerDied","Data":"787ccfd410996b25ef71667d45424380df5c5842a5c6bd2b30fb6685ce7543c2"}
Jul 16 21:14:53.674043 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:53.673858    2530 scope.go:117] "RemoveContainer" containerID="787ccfd410996b25ef71667d45424380df5c5842a5c6bd2b30fb6685ce7543c2"
Jul 16 21:14:54.680768 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:54.680691    2530 generic.go:356] "Generic (PLEG): container finished" podID="f3f251ab-c2d4-4f59-a4ca-82d74af6f748" containerID="967c6a8d257668dc9027cd154d5f9895bf5fccc238162ea27a12b5c32aec0d52" exitCode=6
Jul 16 21:14:54.681166 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:54.680773    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b" event={"ID":"f3f251ab-c2d4-4f59-a4ca-82d74af6f748","Type":"ContainerDied","Data":"967c6a8d257668dc9027cd154d5f9895bf5fccc238162ea27a12b5c32aec0d52"}
Jul 16 21:14:54.681166 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:54.680836    2530 scope.go:117] "RemoveContainer" containerID="787ccfd410996b25ef71667d45424380df5c5842a5c6bd2b30fb6685ce7543c2"
Jul 16 21:14:54.681166 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:54.681014    2530 scope.go:117] "RemoveContainer" containerID="967c6a8d257668dc9027cd154d5f9895bf5fccc238162ea27a12b5c32aec0d52"
Jul 16 21:14:54.681332 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:14:54.681246    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b_odh-ai-gateway-infra(f3f251ab-c2d4-4f59-a4ca-82d74af6f748)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b" podUID="f3f251ab-c2d4-4f59-a4ca-82d74af6f748"
Jul 16 21:14:55.686459 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:14:55.686409    2530 scope.go:117] "RemoveContainer" containerID="967c6a8d257668dc9027cd154d5f9895bf5fccc238162ea27a12b5c32aec0d52"
Jul 16 21:14:55.686829 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:14:55.686615    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b_odh-ai-gateway-infra(f3f251ab-c2d4-4f59-a4ca-82d74af6f748)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b" podUID="f3f251ab-c2d4-4f59-a4ca-82d74af6f748"
Jul 16 21:15:03.502801 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:03.502768    2530 scope.go:117] "RemoveContainer" containerID="0f138a38181de9c99d697b5a2fefde7307fd39b5b4344b1431ff193e885b5ab4"
Jul 16 21:15:03.927703 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:03.927664    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5"]
Jul 16 21:15:03.930969 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:03.930952    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5"
Jul 16 21:15:03.938826 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:03.938801    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5"]
Jul 16 21:15:04.051883 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:04.051848    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/affd0ec8-6d96-4350-b987-56710d8a0cba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5\" (UID: \"affd0ec8-6d96-4350-b987-56710d8a0cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5"
Jul 16 21:15:04.152727 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:04.152634    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/affd0ec8-6d96-4350-b987-56710d8a0cba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5\" (UID: \"affd0ec8-6d96-4350-b987-56710d8a0cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5"
Jul 16 21:15:04.153288 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:04.153271    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/affd0ec8-6d96-4350-b987-56710d8a0cba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5\" (UID: \"affd0ec8-6d96-4350-b987-56710d8a0cba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5"
Jul 16 21:15:04.242465 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:04.242442    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5"
Jul 16 21:15:04.375168 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:04.375134    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5"]
Jul 16 21:15:04.378312 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:15:04.378285    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaffd0ec8_6d96_4350_b987_56710d8a0cba.slice/crio-9e1069970209617b57133ee35f30f51cad04ff334e19fb9be2d47446ce60795f WatchSource:0}: Error finding container 9e1069970209617b57133ee35f30f51cad04ff334e19fb9be2d47446ce60795f: Status 404 returned error can't find the container with id 9e1069970209617b57133ee35f30f51cad04ff334e19fb9be2d47446ce60795f
Jul 16 21:15:04.725837 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:04.725802    2530 generic.go:356] "Generic (PLEG): container finished" podID="f2a12b0d-dff6-42dc-90ea-026cb8095aad" containerID="7bcb66a604848bc4f55ea4bb34a752408c1f1edf98979f15c7fbeb625b879467" exitCode=6
Jul 16 21:15:04.726484 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:04.725872    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c" event={"ID":"f2a12b0d-dff6-42dc-90ea-026cb8095aad","Type":"ContainerDied","Data":"7bcb66a604848bc4f55ea4bb34a752408c1f1edf98979f15c7fbeb625b879467"}
Jul 16 21:15:04.726484 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:04.725938    2530 scope.go:117] "RemoveContainer" containerID="0f138a38181de9c99d697b5a2fefde7307fd39b5b4344b1431ff193e885b5ab4"
Jul 16 21:15:04.726484 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:04.726291    2530 scope.go:117] "RemoveContainer" containerID="7bcb66a604848bc4f55ea4bb34a752408c1f1edf98979f15c7fbeb625b879467"
Jul 16 21:15:04.726943 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:04.726521    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c_odh-ai-gateway-infra(f2a12b0d-dff6-42dc-90ea-026cb8095aad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c" podUID="f2a12b0d-dff6-42dc-90ea-026cb8095aad"
Jul 16 21:15:04.727779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:04.727755    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5" event={"ID":"affd0ec8-6d96-4350-b987-56710d8a0cba","Type":"ContainerStarted","Data":"9e1069970209617b57133ee35f30f51cad04ff334e19fb9be2d47446ce60795f"}
Jul 16 21:15:05.738503 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:05.738411    2530 generic.go:356] "Generic (PLEG): container finished" podID="affd0ec8-6d96-4350-b987-56710d8a0cba" containerID="8475c5bfc1072ce9a9e1b94951e6846086d9157083f805f2cc031801335429a9" exitCode=6
Jul 16 21:15:05.738503 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:05.738495    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5" event={"ID":"affd0ec8-6d96-4350-b987-56710d8a0cba","Type":"ContainerDied","Data":"8475c5bfc1072ce9a9e1b94951e6846086d9157083f805f2cc031801335429a9"}
Jul 16 21:15:05.739215 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:05.738687    2530 scope.go:117] "RemoveContainer" containerID="8475c5bfc1072ce9a9e1b94951e6846086d9157083f805f2cc031801335429a9"
Jul 16 21:15:05.751316 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:05.751252    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c"]
Jul 16 21:15:05.893031 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:05.893004    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c"
Jul 16 21:15:05.969376 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:05.969345    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a12b0d-dff6-42dc-90ea-026cb8095aad-maas-api-service-ca\") pod \"f2a12b0d-dff6-42dc-90ea-026cb8095aad\" (UID: \"f2a12b0d-dff6-42dc-90ea-026cb8095aad\") "
Jul 16 21:15:05.969681 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:05.969657    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a12b0d-dff6-42dc-90ea-026cb8095aad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f2a12b0d-dff6-42dc-90ea-026cb8095aad" (UID: "f2a12b0d-dff6-42dc-90ea-026cb8095aad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:15:06.070350 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:06.070322    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a12b0d-dff6-42dc-90ea-026cb8095aad-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:15:06.743894 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:06.743867    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c"
Jul 16 21:15:06.744410 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:06.743870    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c" event={"ID":"f2a12b0d-dff6-42dc-90ea-026cb8095aad","Type":"ContainerDied","Data":"41c013dc8ae99949f489bbd581920789f7a17ca2b43ead944db85b0ae61520d9"}
Jul 16 21:15:06.744410 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:06.744010    2530 scope.go:117] "RemoveContainer" containerID="7bcb66a604848bc4f55ea4bb34a752408c1f1edf98979f15c7fbeb625b879467"
Jul 16 21:15:06.752277 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:06.752247    2530 generic.go:356] "Generic (PLEG): container finished" podID="affd0ec8-6d96-4350-b987-56710d8a0cba" containerID="75788fe2c8c4f221a5a9555f7dc603536d6fd777de8507483b65727dc3f0f89c" exitCode=6
Jul 16 21:15:06.752421 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:06.752317    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5" event={"ID":"affd0ec8-6d96-4350-b987-56710d8a0cba","Type":"ContainerDied","Data":"75788fe2c8c4f221a5a9555f7dc603536d6fd777de8507483b65727dc3f0f89c"}
Jul 16 21:15:06.752562 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:06.752549    2530 scope.go:117] "RemoveContainer" containerID="75788fe2c8c4f221a5a9555f7dc603536d6fd777de8507483b65727dc3f0f89c"
Jul 16 21:15:06.752739 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:06.752722    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5_odh-ai-gateway-infra(affd0ec8-6d96-4350-b987-56710d8a0cba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5" podUID="affd0ec8-6d96-4350-b987-56710d8a0cba"
Jul 16 21:15:06.758376 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:06.758358    2530 scope.go:117] "RemoveContainer" containerID="8475c5bfc1072ce9a9e1b94951e6846086d9157083f805f2cc031801335429a9"
Jul 16 21:15:06.768094 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:06.768067    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c"]
Jul 16 21:15:06.770245 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:06.770224    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bqh2c"]
Jul 16 21:15:07.758967 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:07.758942    2530 scope.go:117] "RemoveContainer" containerID="75788fe2c8c4f221a5a9555f7dc603536d6fd777de8507483b65727dc3f0f89c"
Jul 16 21:15:07.759419 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:07.759151    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5_odh-ai-gateway-infra(affd0ec8-6d96-4350-b987-56710d8a0cba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5" podUID="affd0ec8-6d96-4350-b987-56710d8a0cba"
Jul 16 21:15:08.507869 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:08.507829    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2a12b0d-dff6-42dc-90ea-026cb8095aad" path="/var/lib/kubelet/pods/f2a12b0d-dff6-42dc-90ea-026cb8095aad/volumes"
Jul 16 21:15:09.501821 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:09.501793    2530 scope.go:117] "RemoveContainer" containerID="967c6a8d257668dc9027cd154d5f9895bf5fccc238162ea27a12b5c32aec0d52"
Jul 16 21:15:09.769351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:09.769319    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b" event={"ID":"f3f251ab-c2d4-4f59-a4ca-82d74af6f748","Type":"ContainerStarted","Data":"86c54a5ec32203ef92c1c357ba4e3e3a1ba552d89f90b0f3a816a0cc2f85b3cb"}
Jul 16 21:15:09.786437 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:09.786392    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b" podStartSLOduration=18.786374465 podStartE2EDuration="18.786374465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:15:09.784955985 +0000 UTC m=+2547.806543786" watchObservedRunningTime="2026-07-16 21:15:09.786374465 +0000 UTC m=+2547.807962251"
Jul 16 21:15:10.775944 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:10.775907    2530 generic.go:356] "Generic (PLEG): container finished" podID="f3f251ab-c2d4-4f59-a4ca-82d74af6f748" containerID="86c54a5ec32203ef92c1c357ba4e3e3a1ba552d89f90b0f3a816a0cc2f85b3cb" exitCode=6
Jul 16 21:15:10.776391 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:10.775980    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b" event={"ID":"f3f251ab-c2d4-4f59-a4ca-82d74af6f748","Type":"ContainerDied","Data":"86c54a5ec32203ef92c1c357ba4e3e3a1ba552d89f90b0f3a816a0cc2f85b3cb"}
Jul 16 21:15:10.776391 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:10.776040    2530 scope.go:117] "RemoveContainer" containerID="967c6a8d257668dc9027cd154d5f9895bf5fccc238162ea27a12b5c32aec0d52"
Jul 16 21:15:10.776391 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:10.776381    2530 scope.go:117] "RemoveContainer" containerID="86c54a5ec32203ef92c1c357ba4e3e3a1ba552d89f90b0f3a816a0cc2f85b3cb"
Jul 16 21:15:10.776565 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:10.776546    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b_odh-ai-gateway-infra(f3f251ab-c2d4-4f59-a4ca-82d74af6f748)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b" podUID="f3f251ab-c2d4-4f59-a4ca-82d74af6f748"
Jul 16 21:15:10.806817 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:10.806787    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b"]
Jul 16 21:15:11.425504 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.425470    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"]
Jul 16 21:15:11.425831 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.425820    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a12b0d-dff6-42dc-90ea-026cb8095aad" containerName="revoke-keys"
Jul 16 21:15:11.425879 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.425833    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a12b0d-dff6-42dc-90ea-026cb8095aad" containerName="revoke-keys"
Jul 16 21:15:11.425879 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.425851    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a12b0d-dff6-42dc-90ea-026cb8095aad" containerName="revoke-keys"
Jul 16 21:15:11.425879 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.425856    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a12b0d-dff6-42dc-90ea-026cb8095aad" containerName="revoke-keys"
Jul 16 21:15:11.425972 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.425905    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2a12b0d-dff6-42dc-90ea-026cb8095aad" containerName="revoke-keys"
Jul 16 21:15:11.425972 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.425916    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2a12b0d-dff6-42dc-90ea-026cb8095aad" containerName="revoke-keys"
Jul 16 21:15:11.429896 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.429875    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"
Jul 16 21:15:11.435207 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.435187    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2a7694fd-serving-cert\""
Jul 16 21:15:11.474392 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.474355    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"]
Jul 16 21:15:11.518625 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.518586    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-maas-api-tls\") pod \"maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn\" (UID: \"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"
Jul 16 21:15:11.518625 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.518628    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-openshift-service-ca\") pod \"maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn\" (UID: \"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"
Jul 16 21:15:11.518848 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.518687    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwmkk\" (UniqueName: \"kubernetes.io/projected/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-kube-api-access-dwmkk\") pod \"maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn\" (UID: \"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"
Jul 16 21:15:11.619679 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.619645    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-maas-api-tls\") pod \"maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn\" (UID: \"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"
Jul 16 21:15:11.619887 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.619691    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-openshift-service-ca\") pod \"maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn\" (UID: \"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"
Jul 16 21:15:11.619887 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.619780    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwmkk\" (UniqueName: \"kubernetes.io/projected/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-kube-api-access-dwmkk\") pod \"maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn\" (UID: \"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"
Jul 16 21:15:11.620199 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:11.620175    2530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-serving-cert: secret "maas-api-e2e-mt-2a7694fd-serving-cert" not found
Jul 16 21:15:11.620287 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:11.620254    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-maas-api-tls podName:f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e nodeName:}" failed. No retries permitted until 2026-07-16 21:15:12.120232203 +0000 UTC m=+2550.141819949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-maas-api-tls") pod "maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn" (UID: "f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e") : secret "maas-api-e2e-mt-2a7694fd-serving-cert" not found
Jul 16 21:15:11.620512 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.620494    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-openshift-service-ca\") pod \"maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn\" (UID: \"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"
Jul 16 21:15:11.637422 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.637390    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwmkk\" (UniqueName: \"kubernetes.io/projected/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-kube-api-access-dwmkk\") pod \"maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn\" (UID: \"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"
Jul 16 21:15:11.763018 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.762985    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv"]
Jul 16 21:15:11.763387 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.763374    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a12b0d-dff6-42dc-90ea-026cb8095aad" containerName="revoke-keys"
Jul 16 21:15:11.763447 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.763389    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a12b0d-dff6-42dc-90ea-026cb8095aad" containerName="revoke-keys"
Jul 16 21:15:11.763483 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.763450    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2a12b0d-dff6-42dc-90ea-026cb8095aad" containerName="revoke-keys"
Jul 16 21:15:11.766515 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.766499    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv"
Jul 16 21:15:11.780875 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.780845    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv"]
Jul 16 21:15:11.821046 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.821014    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18d1f88-2727-48e5-ae2a-990e6718a315-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv\" (UID: \"b18d1f88-2727-48e5-ae2a-990e6718a315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv"
Jul 16 21:15:11.920467 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.920444    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b"
Jul 16 21:15:11.921643 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.921620    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18d1f88-2727-48e5-ae2a-990e6718a315-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv\" (UID: \"b18d1f88-2727-48e5-ae2a-990e6718a315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv"
Jul 16 21:15:11.922244 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:11.922227    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18d1f88-2727-48e5-ae2a-990e6718a315-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv\" (UID: \"b18d1f88-2727-48e5-ae2a-990e6718a315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv"
Jul 16 21:15:12.023015 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:12.022935    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f251ab-c2d4-4f59-a4ca-82d74af6f748-maas-api-service-ca\") pod \"f3f251ab-c2d4-4f59-a4ca-82d74af6f748\" (UID: \"f3f251ab-c2d4-4f59-a4ca-82d74af6f748\") "
Jul 16 21:15:12.023387 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:12.023360    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f251ab-c2d4-4f59-a4ca-82d74af6f748-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f3f251ab-c2d4-4f59-a4ca-82d74af6f748" (UID: "f3f251ab-c2d4-4f59-a4ca-82d74af6f748"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:15:12.076391 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:12.076338    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv"
Jul 16 21:15:12.123866 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:12.123831    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-maas-api-tls\") pod \"maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn\" (UID: \"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"
Jul 16 21:15:12.124039 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:12.123969    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f251ab-c2d4-4f59-a4ca-82d74af6f748-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:15:12.124039 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:12.124000    2530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-serving-cert: secret "maas-api-e2e-mt-2a7694fd-serving-cert" not found
Jul 16 21:15:12.124196 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:12.124073    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-maas-api-tls podName:f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e nodeName:}" failed. No retries permitted until 2026-07-16 21:15:13.124054713 +0000 UTC m=+2551.145642460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-maas-api-tls") pod "maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn" (UID: "f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e") : secret "maas-api-e2e-mt-2a7694fd-serving-cert" not found
Jul 16 21:15:12.229502 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:12.229466    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv"]
Jul 16 21:15:12.232377 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:15:12.232351    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18d1f88_2727_48e5_ae2a_990e6718a315.slice/crio-2aceafbc8845cfea80c8854253ae1e8e1632b68b01da170aa4a5629b20e43216 WatchSource:0}: Error finding container 2aceafbc8845cfea80c8854253ae1e8e1632b68b01da170aa4a5629b20e43216: Status 404 returned error can't find the container with id 2aceafbc8845cfea80c8854253ae1e8e1632b68b01da170aa4a5629b20e43216
Jul 16 21:15:12.790943 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:12.790911    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv" event={"ID":"b18d1f88-2727-48e5-ae2a-990e6718a315","Type":"ContainerStarted","Data":"f8e5ba5ffa5583229e4fd482d6c585e5ae603f8d334941e4e0a099c18b1d5caa"}
Jul 16 21:15:12.791427 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:12.790983    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv" event={"ID":"b18d1f88-2727-48e5-ae2a-990e6718a315","Type":"ContainerStarted","Data":"2aceafbc8845cfea80c8854253ae1e8e1632b68b01da170aa4a5629b20e43216"}
Jul 16 21:15:12.792243 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:12.792212    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b"
Jul 16 21:15:12.792369 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:12.792219    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b" event={"ID":"f3f251ab-c2d4-4f59-a4ca-82d74af6f748","Type":"ContainerDied","Data":"84620a13b5f9bb7c8db0c472bb1df235f01ae518c0c0b6758435cea7e19ddd0a"}
Jul 16 21:15:12.792369 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:12.792357    2530 scope.go:117] "RemoveContainer" containerID="86c54a5ec32203ef92c1c357ba4e3e3a1ba552d89f90b0f3a816a0cc2f85b3cb"
Jul 16 21:15:12.814977 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:12.814924    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv" podStartSLOduration=1.8149089630000002 podStartE2EDuration="1.814908963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:15:12.813614729 +0000 UTC m=+2550.835202496" watchObservedRunningTime="2026-07-16 21:15:12.814908963 +0000 UTC m=+2550.836496728"
Jul 16 21:15:12.827736 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:12.827710    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b"]
Jul 16 21:15:12.835897 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:12.835876    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-p8m4b"]
Jul 16 21:15:13.134366 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:13.134275    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-maas-api-tls\") pod \"maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn\" (UID: \"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"
Jul 16 21:15:13.136786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:13.136769    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-maas-api-tls\") pod \"maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn\" (UID: \"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"
Jul 16 21:15:13.241452 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:13.241416    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"
Jul 16 21:15:13.378889 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:13.378748    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"]
Jul 16 21:15:13.381744 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:15:13.381718    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d35894_8daf_4a80_a7c0_0fd5a72d7f9e.slice/crio-60074186a5fccd8ef218dd43f66e38a77dcbc8d9836d28be99606f83572dac4d WatchSource:0}: Error finding container 60074186a5fccd8ef218dd43f66e38a77dcbc8d9836d28be99606f83572dac4d: Status 404 returned error can't find the container with id 60074186a5fccd8ef218dd43f66e38a77dcbc8d9836d28be99606f83572dac4d
Jul 16 21:15:13.798546 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:13.798503    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn" event={"ID":"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e","Type":"ContainerStarted","Data":"60074186a5fccd8ef218dd43f66e38a77dcbc8d9836d28be99606f83572dac4d"}
Jul 16 21:15:13.800016 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:13.799991    2530 generic.go:356] "Generic (PLEG): container finished" podID="b18d1f88-2727-48e5-ae2a-990e6718a315" containerID="f8e5ba5ffa5583229e4fd482d6c585e5ae603f8d334941e4e0a099c18b1d5caa" exitCode=6
Jul 16 21:15:13.800192 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:13.800037    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv" event={"ID":"b18d1f88-2727-48e5-ae2a-990e6718a315","Type":"ContainerDied","Data":"f8e5ba5ffa5583229e4fd482d6c585e5ae603f8d334941e4e0a099c18b1d5caa"}
Jul 16 21:15:13.800282 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:13.800263    2530 scope.go:117] "RemoveContainer" containerID="f8e5ba5ffa5583229e4fd482d6c585e5ae603f8d334941e4e0a099c18b1d5caa"
Jul 16 21:15:14.510134 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:14.509435    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3f251ab-c2d4-4f59-a4ca-82d74af6f748" path="/var/lib/kubelet/pods/f3f251ab-c2d4-4f59-a4ca-82d74af6f748/volumes"
Jul 16 21:15:14.806849 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:14.806745    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn" event={"ID":"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e","Type":"ContainerStarted","Data":"c74e2b742a1b7601d47165ce9fa03e05992dd23150970c3ef31f0aa2575831f2"}
Jul 16 21:15:14.807334 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:14.806888    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"
Jul 16 21:15:14.808622 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:14.808595    2530 generic.go:356] "Generic (PLEG): container finished" podID="b18d1f88-2727-48e5-ae2a-990e6718a315" containerID="a9a18e49323b0bbfef6e1a2c559eb90406ccac22840d78366ab5288131fb53b6" exitCode=6
Jul 16 21:15:14.808727 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:14.808669    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv" event={"ID":"b18d1f88-2727-48e5-ae2a-990e6718a315","Type":"ContainerDied","Data":"a9a18e49323b0bbfef6e1a2c559eb90406ccac22840d78366ab5288131fb53b6"}
Jul 16 21:15:14.808803 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:14.808733    2530 scope.go:117] "RemoveContainer" containerID="f8e5ba5ffa5583229e4fd482d6c585e5ae603f8d334941e4e0a099c18b1d5caa"
Jul 16 21:15:14.808858 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:14.808827    2530 scope.go:117] "RemoveContainer" containerID="a9a18e49323b0bbfef6e1a2c559eb90406ccac22840d78366ab5288131fb53b6"
Jul 16 21:15:14.809046 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:14.809022    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv_odh-ai-gateway-infra(b18d1f88-2727-48e5-ae2a-990e6718a315)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv" podUID="b18d1f88-2727-48e5-ae2a-990e6718a315"
Jul 16 21:15:14.822561 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:14.822403    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn" podStartSLOduration=3.241842466 podStartE2EDuration="3.822389495s" totalImagesPullingTime="580.547029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:15:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:15:14.821953841 +0000 UTC m=+2552.843541607" watchObservedRunningTime="2026-07-16 21:15:14.822389495 +0000 UTC m=+2552.843977260"
Jul 16 21:15:15.815372 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:15.815339    2530 scope.go:117] "RemoveContainer" containerID="a9a18e49323b0bbfef6e1a2c559eb90406ccac22840d78366ab5288131fb53b6"
Jul 16 21:15:15.815748 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:15.815556    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv_odh-ai-gateway-infra(b18d1f88-2727-48e5-ae2a-990e6718a315)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv" podUID="b18d1f88-2727-48e5-ae2a-990e6718a315"
Jul 16 21:15:18.502684 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:18.502645    2530 scope.go:117] "RemoveContainer" containerID="75788fe2c8c4f221a5a9555f7dc603536d6fd777de8507483b65727dc3f0f89c"
Jul 16 21:15:18.829330 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:18.829229    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5" event={"ID":"affd0ec8-6d96-4350-b987-56710d8a0cba","Type":"ContainerStarted","Data":"33870ce8de520cdcd172d79835bcbf01a701a968eea476a9035afef47af1d040"}
Jul 16 21:15:18.846405 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:18.846354    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5" podStartSLOduration=15.846338877 podStartE2EDuration="15.846338877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:15:18.845721855 +0000 UTC m=+2556.867309620" watchObservedRunningTime="2026-07-16 21:15:18.846338877 +0000 UTC m=+2556.867926655"
Jul 16 21:15:19.835194 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:19.835163    2530 generic.go:356] "Generic (PLEG): container finished" podID="affd0ec8-6d96-4350-b987-56710d8a0cba" containerID="33870ce8de520cdcd172d79835bcbf01a701a968eea476a9035afef47af1d040" exitCode=6
Jul 16 21:15:19.835666 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:19.835223    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5" event={"ID":"affd0ec8-6d96-4350-b987-56710d8a0cba","Type":"ContainerDied","Data":"33870ce8de520cdcd172d79835bcbf01a701a968eea476a9035afef47af1d040"}
Jul 16 21:15:19.835666 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:19.835270    2530 scope.go:117] "RemoveContainer" containerID="75788fe2c8c4f221a5a9555f7dc603536d6fd777de8507483b65727dc3f0f89c"
Jul 16 21:15:19.835666 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:19.835575    2530 scope.go:117] "RemoveContainer" containerID="33870ce8de520cdcd172d79835bcbf01a701a968eea476a9035afef47af1d040"
Jul 16 21:15:19.835871 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:19.835844    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5_odh-ai-gateway-infra(affd0ec8-6d96-4350-b987-56710d8a0cba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5" podUID="affd0ec8-6d96-4350-b987-56710d8a0cba"
Jul 16 21:15:19.857068 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:19.857046    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5"]
Jul 16 21:15:20.819999 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:20.819967    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"
Jul 16 21:15:20.976539 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:20.976512    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5"
Jul 16 21:15:21.107983 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:21.107896    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/affd0ec8-6d96-4350-b987-56710d8a0cba-maas-api-service-ca\") pod \"affd0ec8-6d96-4350-b987-56710d8a0cba\" (UID: \"affd0ec8-6d96-4350-b987-56710d8a0cba\") "
Jul 16 21:15:21.108299 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:21.108279    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/affd0ec8-6d96-4350-b987-56710d8a0cba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "affd0ec8-6d96-4350-b987-56710d8a0cba" (UID: "affd0ec8-6d96-4350-b987-56710d8a0cba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:15:21.209147 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:21.209089    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/affd0ec8-6d96-4350-b987-56710d8a0cba-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:15:21.847828 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:21.847793    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5"
Jul 16 21:15:21.848042 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:21.847828    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5" event={"ID":"affd0ec8-6d96-4350-b987-56710d8a0cba","Type":"ContainerDied","Data":"9e1069970209617b57133ee35f30f51cad04ff334e19fb9be2d47446ce60795f"}
Jul 16 21:15:21.848042 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:21.847889    2530 scope.go:117] "RemoveContainer" containerID="33870ce8de520cdcd172d79835bcbf01a701a968eea476a9035afef47af1d040"
Jul 16 21:15:21.868139 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:21.868091    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5"]
Jul 16 21:15:21.875053 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:21.875028    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-brmn5"]
Jul 16 21:15:22.509397 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:22.509361    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="affd0ec8-6d96-4350-b987-56710d8a0cba" path="/var/lib/kubelet/pods/affd0ec8-6d96-4350-b987-56710d8a0cba/volumes"
Jul 16 21:15:23.188591 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.188563    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a7694fd-g7glb"]
Jul 16 21:15:23.188887 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.188876    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="affd0ec8-6d96-4350-b987-56710d8a0cba" containerName="revoke-keys"
Jul 16 21:15:23.188942 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.188890    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="affd0ec8-6d96-4350-b987-56710d8a0cba" containerName="revoke-keys"
Jul 16 21:15:23.188942 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.188908    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3f251ab-c2d4-4f59-a4ca-82d74af6f748" containerName="revoke-keys"
Jul 16 21:15:23.188942 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.188915    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3f251ab-c2d4-4f59-a4ca-82d74af6f748" containerName="revoke-keys"
Jul 16 21:15:23.188942 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.188922    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3f251ab-c2d4-4f59-a4ca-82d74af6f748" containerName="revoke-keys"
Jul 16 21:15:23.188942 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.188926    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3f251ab-c2d4-4f59-a4ca-82d74af6f748" containerName="revoke-keys"
Jul 16 21:15:23.188942 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.188933    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="affd0ec8-6d96-4350-b987-56710d8a0cba" containerName="revoke-keys"
Jul 16 21:15:23.188942 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.188938    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="affd0ec8-6d96-4350-b987-56710d8a0cba" containerName="revoke-keys"
Jul 16 21:15:23.188942 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.188944    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3f251ab-c2d4-4f59-a4ca-82d74af6f748" containerName="revoke-keys"
Jul 16 21:15:23.189210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.188949    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3f251ab-c2d4-4f59-a4ca-82d74af6f748" containerName="revoke-keys"
Jul 16 21:15:23.189210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.188998    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3f251ab-c2d4-4f59-a4ca-82d74af6f748" containerName="revoke-keys"
Jul 16 21:15:23.189210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.189004    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3f251ab-c2d4-4f59-a4ca-82d74af6f748" containerName="revoke-keys"
Jul 16 21:15:23.189210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.189011    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="affd0ec8-6d96-4350-b987-56710d8a0cba" containerName="revoke-keys"
Jul 16 21:15:23.189210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.189018    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3f251ab-c2d4-4f59-a4ca-82d74af6f748" containerName="revoke-keys"
Jul 16 21:15:23.189210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.189023    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="affd0ec8-6d96-4350-b987-56710d8a0cba" containerName="revoke-keys"
Jul 16 21:15:23.193387 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.193369    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a7694fd-g7glb"
Jul 16 21:15:23.199638 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.199616    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a7694fd-g7glb"]
Jul 16 21:15:23.329201 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.329161    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/210510e4-59c4-4e4e-a5a4-2168c6c2e9eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2a7694fd-g7glb\" (UID: \"210510e4-59c4-4e4e-a5a4-2168c6c2e9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a7694fd-g7glb"
Jul 16 21:15:23.430628 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.430587    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/210510e4-59c4-4e4e-a5a4-2168c6c2e9eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2a7694fd-g7glb\" (UID: \"210510e4-59c4-4e4e-a5a4-2168c6c2e9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a7694fd-g7glb"
Jul 16 21:15:23.431276 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.431255    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/210510e4-59c4-4e4e-a5a4-2168c6c2e9eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2a7694fd-g7glb\" (UID: \"210510e4-59c4-4e4e-a5a4-2168c6c2e9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a7694fd-g7glb"
Jul 16 21:15:23.506289 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.506254    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a7694fd-g7glb"
Jul 16 21:15:23.847476 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.847452    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a7694fd-g7glb"]
Jul 16 21:15:23.850430 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:15:23.850406    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210510e4_59c4_4e4e_a5a4_2168c6c2e9eb.slice/crio-397ce0f0322ebc3014aab3f4ffeca730109988c2e7116bcb0f5747b70055e260 WatchSource:0}: Error finding container 397ce0f0322ebc3014aab3f4ffeca730109988c2e7116bcb0f5747b70055e260: Status 404 returned error can't find the container with id 397ce0f0322ebc3014aab3f4ffeca730109988c2e7116bcb0f5747b70055e260
Jul 16 21:15:23.860016 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:23.859987    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a7694fd-g7glb" event={"ID":"210510e4-59c4-4e4e-a5a4-2168c6c2e9eb","Type":"ContainerStarted","Data":"397ce0f0322ebc3014aab3f4ffeca730109988c2e7116bcb0f5747b70055e260"}
Jul 16 21:15:24.866775 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:24.866689    2530 generic.go:356] "Generic (PLEG): container finished" podID="210510e4-59c4-4e4e-a5a4-2168c6c2e9eb" containerID="2d434dbadbd3c4580571aa57ae95c554722ecc4992b3687bc18966c4d93b1b27" exitCode=0
Jul 16 21:15:24.866775 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:24.866733    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a7694fd-g7glb" event={"ID":"210510e4-59c4-4e4e-a5a4-2168c6c2e9eb","Type":"ContainerDied","Data":"2d434dbadbd3c4580571aa57ae95c554722ecc4992b3687bc18966c4d93b1b27"}
Jul 16 21:15:25.997213 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:25.997190    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a7694fd-g7glb"
Jul 16 21:15:26.154125 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:26.154008    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/210510e4-59c4-4e4e-a5a4-2168c6c2e9eb-maas-api-service-ca\") pod \"210510e4-59c4-4e4e-a5a4-2168c6c2e9eb\" (UID: \"210510e4-59c4-4e4e-a5a4-2168c6c2e9eb\") "
Jul 16 21:15:26.154426 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:26.154401    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/210510e4-59c4-4e4e-a5a4-2168c6c2e9eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "210510e4-59c4-4e4e-a5a4-2168c6c2e9eb" (UID: "210510e4-59c4-4e4e-a5a4-2168c6c2e9eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:15:26.255322 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:26.255282    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/210510e4-59c4-4e4e-a5a4-2168c6c2e9eb-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:15:26.876137 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:26.876082    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a7694fd-g7glb"
Jul 16 21:15:26.876306 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:26.876136    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a7694fd-g7glb" event={"ID":"210510e4-59c4-4e4e-a5a4-2168c6c2e9eb","Type":"ContainerDied","Data":"397ce0f0322ebc3014aab3f4ffeca730109988c2e7116bcb0f5747b70055e260"}
Jul 16 21:15:26.876306 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:26.876190    2530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="397ce0f0322ebc3014aab3f4ffeca730109988c2e7116bcb0f5747b70055e260"
Jul 16 21:15:27.502624 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:27.502594    2530 scope.go:117] "RemoveContainer" containerID="a9a18e49323b0bbfef6e1a2c559eb90406ccac22840d78366ab5288131fb53b6"
Jul 16 21:15:27.882651 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:27.882554    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv" event={"ID":"b18d1f88-2727-48e5-ae2a-990e6718a315","Type":"ContainerStarted","Data":"a6586e8a375651a18fcbec1eda3ccf826053f6657de6a2898347b246f13ed3bd"}
Jul 16 21:15:28.147195 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:28.147131    2530 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18d1f88_2727_48e5_ae2a_990e6718a315.slice/crio-a6586e8a375651a18fcbec1eda3ccf826053f6657de6a2898347b246f13ed3bd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18d1f88_2727_48e5_ae2a_990e6718a315.slice/crio-conmon-a6586e8a375651a18fcbec1eda3ccf826053f6657de6a2898347b246f13ed3bd.scope\": RecentStats: unable to find data in memory cache]"
Jul 16 21:15:28.888940 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:28.888904    2530 generic.go:356] "Generic (PLEG): container finished" podID="b18d1f88-2727-48e5-ae2a-990e6718a315" containerID="a6586e8a375651a18fcbec1eda3ccf826053f6657de6a2898347b246f13ed3bd" exitCode=6
Jul 16 21:15:28.889391 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:28.888965    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv" event={"ID":"b18d1f88-2727-48e5-ae2a-990e6718a315","Type":"ContainerDied","Data":"a6586e8a375651a18fcbec1eda3ccf826053f6657de6a2898347b246f13ed3bd"}
Jul 16 21:15:28.889391 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:28.889013    2530 scope.go:117] "RemoveContainer" containerID="a9a18e49323b0bbfef6e1a2c559eb90406ccac22840d78366ab5288131fb53b6"
Jul 16 21:15:28.889391 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:28.889274    2530 scope.go:117] "RemoveContainer" containerID="a6586e8a375651a18fcbec1eda3ccf826053f6657de6a2898347b246f13ed3bd"
Jul 16 21:15:28.889529 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:28.889504    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv_odh-ai-gateway-infra(b18d1f88-2727-48e5-ae2a-990e6718a315)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv" podUID="b18d1f88-2727-48e5-ae2a-990e6718a315"
Jul 16 21:15:28.916079 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:28.916045    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv"]
Jul 16 21:15:30.022164 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:30.022141    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv"
Jul 16 21:15:30.188717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:30.188624    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18d1f88-2727-48e5-ae2a-990e6718a315-maas-api-service-ca\") pod \"b18d1f88-2727-48e5-ae2a-990e6718a315\" (UID: \"b18d1f88-2727-48e5-ae2a-990e6718a315\") "
Jul 16 21:15:30.189038 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:30.189010    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18d1f88-2727-48e5-ae2a-990e6718a315-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b18d1f88-2727-48e5-ae2a-990e6718a315" (UID: "b18d1f88-2727-48e5-ae2a-990e6718a315"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:15:30.290067 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:30.290034    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18d1f88-2727-48e5-ae2a-990e6718a315-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:15:30.900244 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:30.900158    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv"
Jul 16 21:15:30.900244 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:30.900164    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv" event={"ID":"b18d1f88-2727-48e5-ae2a-990e6718a315","Type":"ContainerDied","Data":"2aceafbc8845cfea80c8854253ae1e8e1632b68b01da170aa4a5629b20e43216"}
Jul 16 21:15:30.900244 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:30.900228    2530 scope.go:117] "RemoveContainer" containerID="a6586e8a375651a18fcbec1eda3ccf826053f6657de6a2898347b246f13ed3bd"
Jul 16 21:15:30.916175 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:30.916148    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv"]
Jul 16 21:15:30.920840 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:30.920817    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-bbfkv"]
Jul 16 21:15:32.507703 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:32.507670    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b18d1f88-2727-48e5-ae2a-990e6718a315" path="/var/lib/kubelet/pods/b18d1f88-2727-48e5-ae2a-990e6718a315/volumes"
Jul 16 21:15:33.806393 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:33.806360    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"]
Jul 16 21:15:33.806870 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:33.806611    2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn" podUID="f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e" containerName="maas-api" containerID="cri-o://c74e2b742a1b7601d47165ce9fa03e05992dd23150970c3ef31f0aa2575831f2" gracePeriod=30
Jul 16 21:15:34.057089 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.057028    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"
Jul 16 21:15:34.125194 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.125159    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-openshift-service-ca\") pod \"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e\" (UID: \"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e\") "
Jul 16 21:15:34.125387 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.125245    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-maas-api-tls\") pod \"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e\" (UID: \"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e\") "
Jul 16 21:15:34.125387 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.125282    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwmkk\" (UniqueName: \"kubernetes.io/projected/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-kube-api-access-dwmkk\") pod \"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e\" (UID: \"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e\") "
Jul 16 21:15:34.125585 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.125557    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e" (UID: "f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:15:34.127523 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.127498    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e" (UID: "f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:15:34.127523 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.127512    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-kube-api-access-dwmkk" (OuterVolumeSpecName: "kube-api-access-dwmkk") pod "f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e" (UID: "f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e"). InnerVolumeSpecName "kube-api-access-dwmkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:15:34.226702 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.226663    2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-openshift-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:15:34.226702 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.226698    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-maas-api-tls\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:15:34.226702 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.226707    2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwmkk\" (UniqueName: \"kubernetes.io/projected/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e-kube-api-access-dwmkk\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:15:34.920952 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.920864    2530 generic.go:356] "Generic (PLEG): container finished" podID="f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e" containerID="c74e2b742a1b7601d47165ce9fa03e05992dd23150970c3ef31f0aa2575831f2" exitCode=0
Jul 16 21:15:34.920952 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.920913    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"
Jul 16 21:15:34.920952 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.920927    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn" event={"ID":"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e","Type":"ContainerDied","Data":"c74e2b742a1b7601d47165ce9fa03e05992dd23150970c3ef31f0aa2575831f2"}
Jul 16 21:15:34.921480 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.920978    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn" event={"ID":"f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e","Type":"ContainerDied","Data":"60074186a5fccd8ef218dd43f66e38a77dcbc8d9836d28be99606f83572dac4d"}
Jul 16 21:15:34.921480 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.921008    2530 scope.go:117] "RemoveContainer" containerID="c74e2b742a1b7601d47165ce9fa03e05992dd23150970c3ef31f0aa2575831f2"
Jul 16 21:15:34.929235 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.929215    2530 scope.go:117] "RemoveContainer" containerID="c74e2b742a1b7601d47165ce9fa03e05992dd23150970c3ef31f0aa2575831f2"
Jul 16 21:15:34.929510 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:34.929486    2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c74e2b742a1b7601d47165ce9fa03e05992dd23150970c3ef31f0aa2575831f2\": container with ID starting with c74e2b742a1b7601d47165ce9fa03e05992dd23150970c3ef31f0aa2575831f2 not found: ID does not exist" containerID="c74e2b742a1b7601d47165ce9fa03e05992dd23150970c3ef31f0aa2575831f2"
Jul 16 21:15:34.929605 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.929516    2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c74e2b742a1b7601d47165ce9fa03e05992dd23150970c3ef31f0aa2575831f2"} err="failed to get container status \"c74e2b742a1b7601d47165ce9fa03e05992dd23150970c3ef31f0aa2575831f2\": rpc error: code = NotFound desc = could not find container \"c74e2b742a1b7601d47165ce9fa03e05992dd23150970c3ef31f0aa2575831f2\": container with ID starting with c74e2b742a1b7601d47165ce9fa03e05992dd23150970c3ef31f0aa2575831f2 not found: ID does not exist"
Jul 16 21:15:34.954010 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.953976    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"]
Jul 16 21:15:34.955418 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:34.955400    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a7694fd-c86495cb7-njvbn"]
Jul 16 21:15:36.507291 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:36.507257    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e" path="/var/lib/kubelet/pods/f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e/volumes"
Jul 16 21:15:39.194336 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.194297    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk"]
Jul 16 21:15:39.194733 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.194613    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b18d1f88-2727-48e5-ae2a-990e6718a315" containerName="revoke-keys"
Jul 16 21:15:39.194733 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.194623    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b18d1f88-2727-48e5-ae2a-990e6718a315" containerName="revoke-keys"
Jul 16 21:15:39.194733 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.194639    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e" containerName="maas-api"
Jul 16 21:15:39.194733 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.194645    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e" containerName="maas-api"
Jul 16 21:15:39.194733 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.194653    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b18d1f88-2727-48e5-ae2a-990e6718a315" containerName="revoke-keys"
Jul 16 21:15:39.194733 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.194659    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b18d1f88-2727-48e5-ae2a-990e6718a315" containerName="revoke-keys"
Jul 16 21:15:39.194733 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.194673    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="affd0ec8-6d96-4350-b987-56710d8a0cba" containerName="revoke-keys"
Jul 16 21:15:39.194733 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.194678    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="affd0ec8-6d96-4350-b987-56710d8a0cba" containerName="revoke-keys"
Jul 16 21:15:39.194733 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.194685    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="210510e4-59c4-4e4e-a5a4-2168c6c2e9eb" containerName="revoke-keys"
Jul 16 21:15:39.194733 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.194690    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="210510e4-59c4-4e4e-a5a4-2168c6c2e9eb" containerName="revoke-keys"
Jul 16 21:15:39.194733 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.194697    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b18d1f88-2727-48e5-ae2a-990e6718a315" containerName="revoke-keys"
Jul 16 21:15:39.194733 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.194701    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b18d1f88-2727-48e5-ae2a-990e6718a315" containerName="revoke-keys"
Jul 16 21:15:39.195131 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.194749    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="affd0ec8-6d96-4350-b987-56710d8a0cba" containerName="revoke-keys"
Jul 16 21:15:39.195131 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.194758    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8d35894-8daf-4a80-a7c0-0fd5a72d7f9e" containerName="maas-api"
Jul 16 21:15:39.195131 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.194764    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b18d1f88-2727-48e5-ae2a-990e6718a315" containerName="revoke-keys"
Jul 16 21:15:39.195131 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.194771    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="210510e4-59c4-4e4e-a5a4-2168c6c2e9eb" containerName="revoke-keys"
Jul 16 21:15:39.195131 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.194776    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b18d1f88-2727-48e5-ae2a-990e6718a315" containerName="revoke-keys"
Jul 16 21:15:39.198860 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.198839    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk"
Jul 16 21:15:39.201975 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.201947    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:15:39.207734 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.207710    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk"]
Jul 16 21:15:39.269009 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.268974    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4884940f-5e1d-4e32-a068-941ec7abd7a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk\" (UID: \"4884940f-5e1d-4e32-a068-941ec7abd7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk"
Jul 16 21:15:39.370358 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.370319    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4884940f-5e1d-4e32-a068-941ec7abd7a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk\" (UID: \"4884940f-5e1d-4e32-a068-941ec7abd7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk"
Jul 16 21:15:39.370899 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.370882    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4884940f-5e1d-4e32-a068-941ec7abd7a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk\" (UID: \"4884940f-5e1d-4e32-a068-941ec7abd7a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk"
Jul 16 21:15:39.509524 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.509494    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk"
Jul 16 21:15:39.651910 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.651885    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk"]
Jul 16 21:15:39.654095 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:15:39.654064    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4884940f_5e1d_4e32_a068_941ec7abd7a2.slice/crio-ad01624af0b3f71c67b53e3d228fae9d255d65d30658fb8f2c959d194fb8b22c WatchSource:0}: Error finding container ad01624af0b3f71c67b53e3d228fae9d255d65d30658fb8f2c959d194fb8b22c: Status 404 returned error can't find the container with id ad01624af0b3f71c67b53e3d228fae9d255d65d30658fb8f2c959d194fb8b22c
Jul 16 21:15:39.944487 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.944452    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk" event={"ID":"4884940f-5e1d-4e32-a068-941ec7abd7a2","Type":"ContainerStarted","Data":"7111fa2757af0b939873b43c3d9f92394233c1d685e2722b3674d0b4ed7c77b3"}
Jul 16 21:15:39.944677 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.944507    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk" event={"ID":"4884940f-5e1d-4e32-a068-941ec7abd7a2","Type":"ContainerStarted","Data":"ad01624af0b3f71c67b53e3d228fae9d255d65d30658fb8f2c959d194fb8b22c"}
Jul 16 21:15:39.960939 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:39.960891    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk" podStartSLOduration=0.960874788 podStartE2EDuration="960.874788ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:15:39.959812903 +0000 UTC m=+2577.981400671" watchObservedRunningTime="2026-07-16 21:15:39.960874788 +0000 UTC m=+2577.982462618"
Jul 16 21:15:40.950761 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:40.950724    2530 generic.go:356] "Generic (PLEG): container finished" podID="4884940f-5e1d-4e32-a068-941ec7abd7a2" containerID="7111fa2757af0b939873b43c3d9f92394233c1d685e2722b3674d0b4ed7c77b3" exitCode=6
Jul 16 21:15:40.951184 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:40.950770    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk" event={"ID":"4884940f-5e1d-4e32-a068-941ec7abd7a2","Type":"ContainerDied","Data":"7111fa2757af0b939873b43c3d9f92394233c1d685e2722b3674d0b4ed7c77b3"}
Jul 16 21:15:40.951184 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:40.950992    2530 scope.go:117] "RemoveContainer" containerID="7111fa2757af0b939873b43c3d9f92394233c1d685e2722b3674d0b4ed7c77b3"
Jul 16 21:15:41.957437 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:41.957399    2530 generic.go:356] "Generic (PLEG): container finished" podID="4884940f-5e1d-4e32-a068-941ec7abd7a2" containerID="37fae8088300827fc96c633582e696ac0191e89880cd33dbeb710708ee18f5b0" exitCode=6
Jul 16 21:15:41.957867 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:41.957475    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk" event={"ID":"4884940f-5e1d-4e32-a068-941ec7abd7a2","Type":"ContainerDied","Data":"37fae8088300827fc96c633582e696ac0191e89880cd33dbeb710708ee18f5b0"}
Jul 16 21:15:41.957867 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:41.957529    2530 scope.go:117] "RemoveContainer" containerID="7111fa2757af0b939873b43c3d9f92394233c1d685e2722b3674d0b4ed7c77b3"
Jul 16 21:15:41.957867 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:41.957720    2530 scope.go:117] "RemoveContainer" containerID="37fae8088300827fc96c633582e696ac0191e89880cd33dbeb710708ee18f5b0"
Jul 16 21:15:41.958056 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:41.957938    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk_odh-ai-gateway-infra(4884940f-5e1d-4e32-a068-941ec7abd7a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk" podUID="4884940f-5e1d-4e32-a068-941ec7abd7a2"
Jul 16 21:15:42.963019 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:42.962989    2530 scope.go:117] "RemoveContainer" containerID="37fae8088300827fc96c633582e696ac0191e89880cd33dbeb710708ee18f5b0"
Jul 16 21:15:42.963462 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:42.963228    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk_odh-ai-gateway-infra(4884940f-5e1d-4e32-a068-941ec7abd7a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk" podUID="4884940f-5e1d-4e32-a068-941ec7abd7a2"
Jul 16 21:15:52.060435 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:52.060386    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f"]
Jul 16 21:15:52.060858 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:52.060845    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b18d1f88-2727-48e5-ae2a-990e6718a315" containerName="revoke-keys"
Jul 16 21:15:52.063749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:52.063731    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f"
Jul 16 21:15:52.072295 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:52.072264    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f"]
Jul 16 21:15:52.185019 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:52.184972    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce5aead-14b2-4e20-8c93-ca3d39b2d705-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f\" (UID: \"4ce5aead-14b2-4e20-8c93-ca3d39b2d705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f"
Jul 16 21:15:52.286543 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:52.286506    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce5aead-14b2-4e20-8c93-ca3d39b2d705-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f\" (UID: \"4ce5aead-14b2-4e20-8c93-ca3d39b2d705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f"
Jul 16 21:15:52.287169 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:52.287150    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce5aead-14b2-4e20-8c93-ca3d39b2d705-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f\" (UID: \"4ce5aead-14b2-4e20-8c93-ca3d39b2d705\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f"
Jul 16 21:15:52.375912 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:52.375826    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f"
Jul 16 21:15:52.715581 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:52.715553    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f"]
Jul 16 21:15:52.717754 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:15:52.717726    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce5aead_14b2_4e20_8c93_ca3d39b2d705.slice/crio-23a99d0a9318a0c0496097610f1362bb43cbaa511bcb0892d13a57348e563d20 WatchSource:0}: Error finding container 23a99d0a9318a0c0496097610f1362bb43cbaa511bcb0892d13a57348e563d20: Status 404 returned error can't find the container with id 23a99d0a9318a0c0496097610f1362bb43cbaa511bcb0892d13a57348e563d20
Jul 16 21:15:53.010300 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:53.010211    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f" event={"ID":"4ce5aead-14b2-4e20-8c93-ca3d39b2d705","Type":"ContainerStarted","Data":"5dcf4cf13a5ee521e17affc98e179d7c6f71a88c167f1efd3260c32f028a9f4e"}
Jul 16 21:15:53.010300 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:53.010270    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f" event={"ID":"4ce5aead-14b2-4e20-8c93-ca3d39b2d705","Type":"ContainerStarted","Data":"23a99d0a9318a0c0496097610f1362bb43cbaa511bcb0892d13a57348e563d20"}
Jul 16 21:15:53.025338 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:53.025248    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f" podStartSLOduration=1.025218987 podStartE2EDuration="1.025218987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:15:53.024941231 +0000 UTC m=+2591.046528997" watchObservedRunningTime="2026-07-16 21:15:53.025218987 +0000 UTC m=+2591.046806754"
Jul 16 21:15:54.016190 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:54.016150    2530 generic.go:356] "Generic (PLEG): container finished" podID="4ce5aead-14b2-4e20-8c93-ca3d39b2d705" containerID="5dcf4cf13a5ee521e17affc98e179d7c6f71a88c167f1efd3260c32f028a9f4e" exitCode=6
Jul 16 21:15:54.016673 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:54.016243    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f" event={"ID":"4ce5aead-14b2-4e20-8c93-ca3d39b2d705","Type":"ContainerDied","Data":"5dcf4cf13a5ee521e17affc98e179d7c6f71a88c167f1efd3260c32f028a9f4e"}
Jul 16 21:15:54.016673 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:54.016459    2530 scope.go:117] "RemoveContainer" containerID="5dcf4cf13a5ee521e17affc98e179d7c6f71a88c167f1efd3260c32f028a9f4e"
Jul 16 21:15:55.023017 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:55.022932    2530 generic.go:356] "Generic (PLEG): container finished" podID="4ce5aead-14b2-4e20-8c93-ca3d39b2d705" containerID="f6d9789cb7d12222d171e6e98703c2491f439c40e9126fde865ebc6be05c2ecc" exitCode=6
Jul 16 21:15:55.023017 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:55.022996    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f" event={"ID":"4ce5aead-14b2-4e20-8c93-ca3d39b2d705","Type":"ContainerDied","Data":"f6d9789cb7d12222d171e6e98703c2491f439c40e9126fde865ebc6be05c2ecc"}
Jul 16 21:15:55.023500 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:55.023040    2530 scope.go:117] "RemoveContainer" containerID="5dcf4cf13a5ee521e17affc98e179d7c6f71a88c167f1efd3260c32f028a9f4e"
Jul 16 21:15:55.023500 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:55.023258    2530 scope.go:117] "RemoveContainer" containerID="f6d9789cb7d12222d171e6e98703c2491f439c40e9126fde865ebc6be05c2ecc"
Jul 16 21:15:55.023500 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:55.023470    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f_odh-ai-gateway-infra(4ce5aead-14b2-4e20-8c93-ca3d39b2d705)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f" podUID="4ce5aead-14b2-4e20-8c93-ca3d39b2d705"
Jul 16 21:15:56.029425 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:56.029392    2530 scope.go:117] "RemoveContainer" containerID="f6d9789cb7d12222d171e6e98703c2491f439c40e9126fde865ebc6be05c2ecc"
Jul 16 21:15:56.029820 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:56.029573    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f_odh-ai-gateway-infra(4ce5aead-14b2-4e20-8c93-ca3d39b2d705)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f" podUID="4ce5aead-14b2-4e20-8c93-ca3d39b2d705"
Jul 16 21:15:56.502588 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:56.502557    2530 scope.go:117] "RemoveContainer" containerID="37fae8088300827fc96c633582e696ac0191e89880cd33dbeb710708ee18f5b0"
Jul 16 21:15:57.035322 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:57.035288    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk" event={"ID":"4884940f-5e1d-4e32-a068-941ec7abd7a2","Type":"ContainerStarted","Data":"d2a29b4c143f927c12b9b2b2a95ddf59908be5245629325c610a7dc642bd0a32"}
Jul 16 21:15:58.041433 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:58.041393    2530 generic.go:356] "Generic (PLEG): container finished" podID="4884940f-5e1d-4e32-a068-941ec7abd7a2" containerID="d2a29b4c143f927c12b9b2b2a95ddf59908be5245629325c610a7dc642bd0a32" exitCode=6
Jul 16 21:15:58.041897 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:58.041466    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk" event={"ID":"4884940f-5e1d-4e32-a068-941ec7abd7a2","Type":"ContainerDied","Data":"d2a29b4c143f927c12b9b2b2a95ddf59908be5245629325c610a7dc642bd0a32"}
Jul 16 21:15:58.041897 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:58.041547    2530 scope.go:117] "RemoveContainer" containerID="37fae8088300827fc96c633582e696ac0191e89880cd33dbeb710708ee18f5b0"
Jul 16 21:15:58.041897 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:58.041834    2530 scope.go:117] "RemoveContainer" containerID="d2a29b4c143f927c12b9b2b2a95ddf59908be5245629325c610a7dc642bd0a32"
Jul 16 21:15:58.042094 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:15:58.042073    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk_odh-ai-gateway-infra(4884940f-5e1d-4e32-a068-941ec7abd7a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk" podUID="4884940f-5e1d-4e32-a068-941ec7abd7a2"
Jul 16 21:15:58.062949 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:58.062921    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk"]
Jul 16 21:15:59.182666 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:59.182635    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk"
Jul 16 21:15:59.251201 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:59.251167    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4884940f-5e1d-4e32-a068-941ec7abd7a2-maas-api-service-ca\") pod \"4884940f-5e1d-4e32-a068-941ec7abd7a2\" (UID: \"4884940f-5e1d-4e32-a068-941ec7abd7a2\") "
Jul 16 21:15:59.251481 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:59.251459    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4884940f-5e1d-4e32-a068-941ec7abd7a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4884940f-5e1d-4e32-a068-941ec7abd7a2" (UID: "4884940f-5e1d-4e32-a068-941ec7abd7a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:15:59.351935 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:15:59.351845    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4884940f-5e1d-4e32-a068-941ec7abd7a2-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:16:00.053020 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:00.052980    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk" event={"ID":"4884940f-5e1d-4e32-a068-941ec7abd7a2","Type":"ContainerDied","Data":"ad01624af0b3f71c67b53e3d228fae9d255d65d30658fb8f2c959d194fb8b22c"}
Jul 16 21:16:00.053242 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:00.053043    2530 scope.go:117] "RemoveContainer" containerID="d2a29b4c143f927c12b9b2b2a95ddf59908be5245629325c610a7dc642bd0a32"
Jul 16 21:16:00.053242 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:00.052992    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk"
Jul 16 21:16:00.076184 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:00.076145    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk"]
Jul 16 21:16:00.082502 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:00.082474    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-v8msk"]
Jul 16 21:16:00.507020 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:00.506984    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4884940f-5e1d-4e32-a068-941ec7abd7a2" path="/var/lib/kubelet/pods/4884940f-5e1d-4e32-a068-941ec7abd7a2/volumes"
Jul 16 21:16:08.502523 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:08.502441    2530 scope.go:117] "RemoveContainer" containerID="f6d9789cb7d12222d171e6e98703c2491f439c40e9126fde865ebc6be05c2ecc"
Jul 16 21:16:09.096534 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:09.096436    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f" event={"ID":"4ce5aead-14b2-4e20-8c93-ca3d39b2d705","Type":"ContainerStarted","Data":"7f7130641c3b0597581974f1a979dd99fc337c422a1a9454fab04d7be082cec0"}
Jul 16 21:16:10.102529 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:10.102490    2530 generic.go:356] "Generic (PLEG): container finished" podID="4ce5aead-14b2-4e20-8c93-ca3d39b2d705" containerID="7f7130641c3b0597581974f1a979dd99fc337c422a1a9454fab04d7be082cec0" exitCode=6
Jul 16 21:16:10.102964 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:10.102558    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f" event={"ID":"4ce5aead-14b2-4e20-8c93-ca3d39b2d705","Type":"ContainerDied","Data":"7f7130641c3b0597581974f1a979dd99fc337c422a1a9454fab04d7be082cec0"}
Jul 16 21:16:10.102964 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:10.102641    2530 scope.go:117] "RemoveContainer" containerID="f6d9789cb7d12222d171e6e98703c2491f439c40e9126fde865ebc6be05c2ecc"
Jul 16 21:16:10.102964 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:10.102880    2530 scope.go:117] "RemoveContainer" containerID="7f7130641c3b0597581974f1a979dd99fc337c422a1a9454fab04d7be082cec0"
Jul 16 21:16:10.103117 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:16:10.103072    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f_odh-ai-gateway-infra(4ce5aead-14b2-4e20-8c93-ca3d39b2d705)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f" podUID="4ce5aead-14b2-4e20-8c93-ca3d39b2d705"
Jul 16 21:16:10.132049 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:10.132013    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f"]
Jul 16 21:16:11.244956 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:11.244932    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f"
Jul 16 21:16:11.360565 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:11.360473    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce5aead-14b2-4e20-8c93-ca3d39b2d705-maas-api-service-ca\") pod \"4ce5aead-14b2-4e20-8c93-ca3d39b2d705\" (UID: \"4ce5aead-14b2-4e20-8c93-ca3d39b2d705\") "
Jul 16 21:16:11.360846 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:11.360821    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ce5aead-14b2-4e20-8c93-ca3d39b2d705-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4ce5aead-14b2-4e20-8c93-ca3d39b2d705" (UID: "4ce5aead-14b2-4e20-8c93-ca3d39b2d705"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:16:11.461688 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:11.461649    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ce5aead-14b2-4e20-8c93-ca3d39b2d705-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:16:12.114166 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.114132    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f"
Jul 16 21:16:12.114365 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.114131    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f" event={"ID":"4ce5aead-14b2-4e20-8c93-ca3d39b2d705","Type":"ContainerDied","Data":"23a99d0a9318a0c0496097610f1362bb43cbaa511bcb0892d13a57348e563d20"}
Jul 16 21:16:12.114365 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.114264    2530 scope.go:117] "RemoveContainer" containerID="7f7130641c3b0597581974f1a979dd99fc337c422a1a9454fab04d7be082cec0"
Jul 16 21:16:12.135135 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.135089    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f"]
Jul 16 21:16:12.137150 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.137126    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-dcx6f"]
Jul 16 21:16:12.507655 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.507623    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ce5aead-14b2-4e20-8c93-ca3d39b2d705" path="/var/lib/kubelet/pods/4ce5aead-14b2-4e20-8c93-ca3d39b2d705/volumes"
Jul 16 21:16:12.710635 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.710604    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"]
Jul 16 21:16:12.711041 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.711019    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4884940f-5e1d-4e32-a068-941ec7abd7a2" containerName="revoke-keys"
Jul 16 21:16:12.711041 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.711041    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4884940f-5e1d-4e32-a068-941ec7abd7a2" containerName="revoke-keys"
Jul 16 21:16:12.711312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.711060    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce5aead-14b2-4e20-8c93-ca3d39b2d705" containerName="revoke-keys"
Jul 16 21:16:12.711312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.711069    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce5aead-14b2-4e20-8c93-ca3d39b2d705" containerName="revoke-keys"
Jul 16 21:16:12.711312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.711078    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce5aead-14b2-4e20-8c93-ca3d39b2d705" containerName="revoke-keys"
Jul 16 21:16:12.711312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.711086    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce5aead-14b2-4e20-8c93-ca3d39b2d705" containerName="revoke-keys"
Jul 16 21:16:12.711312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.711129    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4884940f-5e1d-4e32-a068-941ec7abd7a2" containerName="revoke-keys"
Jul 16 21:16:12.711312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.711138    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4884940f-5e1d-4e32-a068-941ec7abd7a2" containerName="revoke-keys"
Jul 16 21:16:12.711312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.711147    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4884940f-5e1d-4e32-a068-941ec7abd7a2" containerName="revoke-keys"
Jul 16 21:16:12.711312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.711154    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4884940f-5e1d-4e32-a068-941ec7abd7a2" containerName="revoke-keys"
Jul 16 21:16:12.711312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.711246    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4884940f-5e1d-4e32-a068-941ec7abd7a2" containerName="revoke-keys"
Jul 16 21:16:12.711312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.711258    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce5aead-14b2-4e20-8c93-ca3d39b2d705" containerName="revoke-keys"
Jul 16 21:16:12.711312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.711267    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4884940f-5e1d-4e32-a068-941ec7abd7a2" containerName="revoke-keys"
Jul 16 21:16:12.711312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.711278    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce5aead-14b2-4e20-8c93-ca3d39b2d705" containerName="revoke-keys"
Jul 16 21:16:12.715971 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.715950    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"
Jul 16 21:16:12.719073 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.719038    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e9d9b90e-serving-cert\""
Jul 16 21:16:12.727927 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.727903    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"]
Jul 16 21:16:12.875194 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.875083    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcfzz\" (UniqueName: \"kubernetes.io/projected/d1c96064-14f6-44d5-b355-b992c5c85794-kube-api-access-gcfzz\") pod \"maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52\" (UID: \"d1c96064-14f6-44d5-b355-b992c5c85794\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"
Jul 16 21:16:12.875353 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.875253    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c96064-14f6-44d5-b355-b992c5c85794-maas-api-tls\") pod \"maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52\" (UID: \"d1c96064-14f6-44d5-b355-b992c5c85794\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"
Jul 16 21:16:12.875353 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.875299    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c96064-14f6-44d5-b355-b992c5c85794-openshift-service-ca\") pod \"maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52\" (UID: \"d1c96064-14f6-44d5-b355-b992c5c85794\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"
Jul 16 21:16:12.975827 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.975788    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c96064-14f6-44d5-b355-b992c5c85794-maas-api-tls\") pod \"maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52\" (UID: \"d1c96064-14f6-44d5-b355-b992c5c85794\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"
Jul 16 21:16:12.975827 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.975831    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c96064-14f6-44d5-b355-b992c5c85794-openshift-service-ca\") pod \"maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52\" (UID: \"d1c96064-14f6-44d5-b355-b992c5c85794\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"
Jul 16 21:16:12.976074 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.975882    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfzz\" (UniqueName: \"kubernetes.io/projected/d1c96064-14f6-44d5-b355-b992c5c85794-kube-api-access-gcfzz\") pod \"maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52\" (UID: \"d1c96064-14f6-44d5-b355-b992c5c85794\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"
Jul 16 21:16:12.976074 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:16:12.975958    2530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-serving-cert: secret "maas-api-e2e-mt-e9d9b90e-serving-cert" not found
Jul 16 21:16:12.976074 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:16:12.976027    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c96064-14f6-44d5-b355-b992c5c85794-maas-api-tls podName:d1c96064-14f6-44d5-b355-b992c5c85794 nodeName:}" failed. No retries permitted until 2026-07-16 21:16:13.476009195 +0000 UTC m=+2611.497596942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1c96064-14f6-44d5-b355-b992c5c85794-maas-api-tls") pod "maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52" (UID: "d1c96064-14f6-44d5-b355-b992c5c85794") : secret "maas-api-e2e-mt-e9d9b90e-serving-cert" not found
Jul 16 21:16:12.976679 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.976658    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c96064-14f6-44d5-b355-b992c5c85794-openshift-service-ca\") pod \"maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52\" (UID: \"d1c96064-14f6-44d5-b355-b992c5c85794\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"
Jul 16 21:16:12.986538 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:12.986509    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcfzz\" (UniqueName: \"kubernetes.io/projected/d1c96064-14f6-44d5-b355-b992c5c85794-kube-api-access-gcfzz\") pod \"maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52\" (UID: \"d1c96064-14f6-44d5-b355-b992c5c85794\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"
Jul 16 21:16:13.481060 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:13.481020    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c96064-14f6-44d5-b355-b992c5c85794-maas-api-tls\") pod \"maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52\" (UID: \"d1c96064-14f6-44d5-b355-b992c5c85794\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"
Jul 16 21:16:13.483605 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:13.483580    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c96064-14f6-44d5-b355-b992c5c85794-maas-api-tls\") pod \"maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52\" (UID: \"d1c96064-14f6-44d5-b355-b992c5c85794\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"
Jul 16 21:16:13.628373 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:13.628327    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"
Jul 16 21:16:13.764779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:13.764754    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"]
Jul 16 21:16:13.767422 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:16:13.767396    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c96064_14f6_44d5_b355_b992c5c85794.slice/crio-d59a9b0a52ca72bb9763a3bca8f01ed70a335e8602ffc27ee0cf09226936d441 WatchSource:0}: Error finding container d59a9b0a52ca72bb9763a3bca8f01ed70a335e8602ffc27ee0cf09226936d441: Status 404 returned error can't find the container with id d59a9b0a52ca72bb9763a3bca8f01ed70a335e8602ffc27ee0cf09226936d441
Jul 16 21:16:13.768910 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:13.768889    2530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:16:14.126400 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:14.126315    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52" event={"ID":"d1c96064-14f6-44d5-b355-b992c5c85794","Type":"ContainerStarted","Data":"d59a9b0a52ca72bb9763a3bca8f01ed70a335e8602ffc27ee0cf09226936d441"}
Jul 16 21:16:15.132872 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:15.132836    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52" event={"ID":"d1c96064-14f6-44d5-b355-b992c5c85794","Type":"ContainerStarted","Data":"e58a0ba9a7f52ef94a719cc56596d9f4511f42e91856c473d326dbcaa72618ce"}
Jul 16 21:16:15.133274 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:15.132966    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"
Jul 16 21:16:15.150186 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:15.150133    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52" podStartSLOduration=1.989996145 podStartE2EDuration="3.150115797s" totalImagesPullingTime="1.160119652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:16:15.148639709 +0000 UTC m=+2613.170227477" watchObservedRunningTime="2026-07-16 21:16:15.150115797 +0000 UTC m=+2613.171703555"
Jul 16 21:16:21.144751 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:21.144716    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"
Jul 16 21:16:39.255178 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:39.255134    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2"]
Jul 16 21:16:39.255733 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:39.255628    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ce5aead-14b2-4e20-8c93-ca3d39b2d705" containerName="revoke-keys"
Jul 16 21:16:39.255733 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:39.255649    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ce5aead-14b2-4e20-8c93-ca3d39b2d705" containerName="revoke-keys"
Jul 16 21:16:39.255854 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:39.255736    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ce5aead-14b2-4e20-8c93-ca3d39b2d705" containerName="revoke-keys"
Jul 16 21:16:39.255854 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:39.255748    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4884940f-5e1d-4e32-a068-941ec7abd7a2" containerName="revoke-keys"
Jul 16 21:16:39.258998 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:39.258976    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2"
Jul 16 21:16:39.261903 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:39.261879    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:16:39.268794 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:39.268760    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2"]
Jul 16 21:16:39.297610 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:39.297572    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9713298-4d03-441c-8535-8831f39f0ca4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2\" (UID: \"a9713298-4d03-441c-8535-8831f39f0ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2"
Jul 16 21:16:39.399065 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:39.399020    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9713298-4d03-441c-8535-8831f39f0ca4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2\" (UID: \"a9713298-4d03-441c-8535-8831f39f0ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2"
Jul 16 21:16:39.399674 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:39.399652    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9713298-4d03-441c-8535-8831f39f0ca4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2\" (UID: \"a9713298-4d03-441c-8535-8831f39f0ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2"
Jul 16 21:16:39.570331 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:39.570228    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2"
Jul 16 21:16:39.728252 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:39.728211    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2"]
Jul 16 21:16:39.732041 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:16:39.732000    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9713298_4d03_441c_8535_8831f39f0ca4.slice/crio-a5f392993a4b25ceef407d602f8fe008af626e0eba4e20695593032064b3e12f WatchSource:0}: Error finding container a5f392993a4b25ceef407d602f8fe008af626e0eba4e20695593032064b3e12f: Status 404 returned error can't find the container with id a5f392993a4b25ceef407d602f8fe008af626e0eba4e20695593032064b3e12f
Jul 16 21:16:40.245279 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:40.245234    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2" event={"ID":"a9713298-4d03-441c-8535-8831f39f0ca4","Type":"ContainerStarted","Data":"d5504c62b4039a663cd2fc809be1a1198e8a95f6a636f4c54be9e856513bcf8c"}
Jul 16 21:16:40.245279 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:40.245288    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2" event={"ID":"a9713298-4d03-441c-8535-8831f39f0ca4","Type":"ContainerStarted","Data":"a5f392993a4b25ceef407d602f8fe008af626e0eba4e20695593032064b3e12f"}
Jul 16 21:16:40.264045 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:40.263989    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2" podStartSLOduration=1.2639714180000001 podStartE2EDuration="1.263971418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:16:40.263037452 +0000 UTC m=+2638.284625218" watchObservedRunningTime="2026-07-16 21:16:40.263971418 +0000 UTC m=+2638.285559183"
Jul 16 21:16:41.251258 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:41.251222    2530 generic.go:356] "Generic (PLEG): container finished" podID="a9713298-4d03-441c-8535-8831f39f0ca4" containerID="d5504c62b4039a663cd2fc809be1a1198e8a95f6a636f4c54be9e856513bcf8c" exitCode=6
Jul 16 21:16:41.251466 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:41.251305    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2" event={"ID":"a9713298-4d03-441c-8535-8831f39f0ca4","Type":"ContainerDied","Data":"d5504c62b4039a663cd2fc809be1a1198e8a95f6a636f4c54be9e856513bcf8c"}
Jul 16 21:16:41.251524 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:41.251497    2530 scope.go:117] "RemoveContainer" containerID="d5504c62b4039a663cd2fc809be1a1198e8a95f6a636f4c54be9e856513bcf8c"
Jul 16 21:16:42.258341 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:42.258306    2530 generic.go:356] "Generic (PLEG): container finished" podID="a9713298-4d03-441c-8535-8831f39f0ca4" containerID="13a854eaaa13018a7d66d2c62e8bfffd5f835a04688ea9038404d1ca81094cdb" exitCode=6
Jul 16 21:16:42.258859 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:42.258361    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2" event={"ID":"a9713298-4d03-441c-8535-8831f39f0ca4","Type":"ContainerDied","Data":"13a854eaaa13018a7d66d2c62e8bfffd5f835a04688ea9038404d1ca81094cdb"}
Jul 16 21:16:42.258859 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:42.258408    2530 scope.go:117] "RemoveContainer" containerID="d5504c62b4039a663cd2fc809be1a1198e8a95f6a636f4c54be9e856513bcf8c"
Jul 16 21:16:42.258859 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:42.258581    2530 scope.go:117] "RemoveContainer" containerID="13a854eaaa13018a7d66d2c62e8bfffd5f835a04688ea9038404d1ca81094cdb"
Jul 16 21:16:42.258859 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:16:42.258800    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2_odh-ai-gateway-infra(a9713298-4d03-441c-8535-8831f39f0ca4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2" podUID="a9713298-4d03-441c-8535-8831f39f0ca4"
Jul 16 21:16:43.264889 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:43.264860    2530 scope.go:117] "RemoveContainer" containerID="13a854eaaa13018a7d66d2c62e8bfffd5f835a04688ea9038404d1ca81094cdb"
Jul 16 21:16:43.265310 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:16:43.265056    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2_odh-ai-gateway-infra(a9713298-4d03-441c-8535-8831f39f0ca4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2" podUID="a9713298-4d03-441c-8535-8831f39f0ca4"
Jul 16 21:16:51.830535 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:51.830494    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8"]
Jul 16 21:16:51.833816 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:51.833798    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8"
Jul 16 21:16:51.841604 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:51.841579    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8"]
Jul 16 21:16:51.917694 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:51.917656    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf66233d-e555-4660-87bc-d0470f3a98db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8\" (UID: \"bf66233d-e555-4660-87bc-d0470f3a98db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8"
Jul 16 21:16:52.018244 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:52.018201    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf66233d-e555-4660-87bc-d0470f3a98db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8\" (UID: \"bf66233d-e555-4660-87bc-d0470f3a98db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8"
Jul 16 21:16:52.018843 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:52.018822    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf66233d-e555-4660-87bc-d0470f3a98db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8\" (UID: \"bf66233d-e555-4660-87bc-d0470f3a98db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8"
Jul 16 21:16:52.110148 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:52.110046    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j"]
Jul 16 21:16:52.113517 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:52.113495    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j"
Jul 16 21:16:52.120052 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:52.120022    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j"]
Jul 16 21:16:52.145419 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:52.145378    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8"
Jul 16 21:16:52.219692 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:52.219663    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe76fc5-58ae-4dde-b56b-836dcfe76401-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j\" (UID: \"abe76fc5-58ae-4dde-b56b-836dcfe76401\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j"
Jul 16 21:16:52.276563 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:52.276540    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8"]
Jul 16 21:16:52.279006 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:16:52.278976    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf66233d_e555_4660_87bc_d0470f3a98db.slice/crio-fb624b5727da87bd624cb9126c2249f44569239e5a3dbf3cc241496d3a511add WatchSource:0}: Error finding container fb624b5727da87bd624cb9126c2249f44569239e5a3dbf3cc241496d3a511add: Status 404 returned error can't find the container with id fb624b5727da87bd624cb9126c2249f44569239e5a3dbf3cc241496d3a511add
Jul 16 21:16:52.304381 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:52.304354    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8" event={"ID":"bf66233d-e555-4660-87bc-d0470f3a98db","Type":"ContainerStarted","Data":"fb624b5727da87bd624cb9126c2249f44569239e5a3dbf3cc241496d3a511add"}
Jul 16 21:16:52.321133 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:52.321090    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe76fc5-58ae-4dde-b56b-836dcfe76401-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j\" (UID: \"abe76fc5-58ae-4dde-b56b-836dcfe76401\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j"
Jul 16 21:16:52.321719 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:52.321696    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe76fc5-58ae-4dde-b56b-836dcfe76401-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j\" (UID: \"abe76fc5-58ae-4dde-b56b-836dcfe76401\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j"
Jul 16 21:16:52.425156 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:52.425072    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j"
Jul 16 21:16:52.643253 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:52.643222    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j"]
Jul 16 21:16:52.645374 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:16:52.645334    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe76fc5_58ae_4dde_b56b_836dcfe76401.slice/crio-db7fe28ec3cd37986f663fc96bbd83cbfb4578e18b5a527dc8ff400225829014 WatchSource:0}: Error finding container db7fe28ec3cd37986f663fc96bbd83cbfb4578e18b5a527dc8ff400225829014: Status 404 returned error can't find the container with id db7fe28ec3cd37986f663fc96bbd83cbfb4578e18b5a527dc8ff400225829014
Jul 16 21:16:53.310152 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:53.310054    2530 generic.go:356] "Generic (PLEG): container finished" podID="abe76fc5-58ae-4dde-b56b-836dcfe76401" containerID="ac82a55c99d644d54d1c865954c169ca99f682a2bc1a1ecc10e62de43c40643b" exitCode=6
Jul 16 21:16:53.310648 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:53.310145    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j" event={"ID":"abe76fc5-58ae-4dde-b56b-836dcfe76401","Type":"ContainerDied","Data":"ac82a55c99d644d54d1c865954c169ca99f682a2bc1a1ecc10e62de43c40643b"}
Jul 16 21:16:53.310648 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:53.310197    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j" event={"ID":"abe76fc5-58ae-4dde-b56b-836dcfe76401","Type":"ContainerStarted","Data":"db7fe28ec3cd37986f663fc96bbd83cbfb4578e18b5a527dc8ff400225829014"}
Jul 16 21:16:53.310648 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:53.310363    2530 scope.go:117] "RemoveContainer" containerID="ac82a55c99d644d54d1c865954c169ca99f682a2bc1a1ecc10e62de43c40643b"
Jul 16 21:16:53.311820 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:53.311801    2530 generic.go:356] "Generic (PLEG): container finished" podID="bf66233d-e555-4660-87bc-d0470f3a98db" containerID="13330741d7c048248d6fc31a895133a771a01ddd656a762729b0473cbeef1977" exitCode=6
Jul 16 21:16:53.311900 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:53.311837    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8" event={"ID":"bf66233d-e555-4660-87bc-d0470f3a98db","Type":"ContainerDied","Data":"13330741d7c048248d6fc31a895133a771a01ddd656a762729b0473cbeef1977"}
Jul 16 21:16:53.312028 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:53.312012    2530 scope.go:117] "RemoveContainer" containerID="13330741d7c048248d6fc31a895133a771a01ddd656a762729b0473cbeef1977"
Jul 16 21:16:54.012069 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:54.012033    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48"]
Jul 16 21:16:54.015391 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:54.015374    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48"
Jul 16 21:16:54.022867 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:54.022836    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48"]
Jul 16 21:16:54.138893 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:54.138852    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6bf82-64c0-40e3-bf36-09cc44f8f48f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-npw48\" (UID: \"d8c6bf82-64c0-40e3-bf36-09cc44f8f48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48"
Jul 16 21:16:54.240372 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:54.240337    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6bf82-64c0-40e3-bf36-09cc44f8f48f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-npw48\" (UID: \"d8c6bf82-64c0-40e3-bf36-09cc44f8f48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48"
Jul 16 21:16:54.240951 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:54.240923    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6bf82-64c0-40e3-bf36-09cc44f8f48f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-npw48\" (UID: \"d8c6bf82-64c0-40e3-bf36-09cc44f8f48f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48"
Jul 16 21:16:54.317577 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:54.317541    2530 generic.go:356] "Generic (PLEG): container finished" podID="abe76fc5-58ae-4dde-b56b-836dcfe76401" containerID="5c1b0bdb31bd82565ada7715b06ae47aae284bb9abf4ecfc9163cafa37b70ff3" exitCode=6
Jul 16 21:16:54.318080 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:54.317600    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j" event={"ID":"abe76fc5-58ae-4dde-b56b-836dcfe76401","Type":"ContainerDied","Data":"5c1b0bdb31bd82565ada7715b06ae47aae284bb9abf4ecfc9163cafa37b70ff3"}
Jul 16 21:16:54.318080 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:54.317674    2530 scope.go:117] "RemoveContainer" containerID="ac82a55c99d644d54d1c865954c169ca99f682a2bc1a1ecc10e62de43c40643b"
Jul 16 21:16:54.318080 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:54.317876    2530 scope.go:117] "RemoveContainer" containerID="5c1b0bdb31bd82565ada7715b06ae47aae284bb9abf4ecfc9163cafa37b70ff3"
Jul 16 21:16:54.318260 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:16:54.318138    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j_odh-ai-gateway-infra(abe76fc5-58ae-4dde-b56b-836dcfe76401)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j" podUID="abe76fc5-58ae-4dde-b56b-836dcfe76401"
Jul 16 21:16:54.319667 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:54.319644    2530 generic.go:356] "Generic (PLEG): container finished" podID="bf66233d-e555-4660-87bc-d0470f3a98db" containerID="300cde199404a02d02f87d507464ccc6e07b5825de3349c3d881d263f6960063" exitCode=6
Jul 16 21:16:54.319756 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:54.319691    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8" event={"ID":"bf66233d-e555-4660-87bc-d0470f3a98db","Type":"ContainerDied","Data":"300cde199404a02d02f87d507464ccc6e07b5825de3349c3d881d263f6960063"}
Jul 16 21:16:54.319890 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:54.319877    2530 scope.go:117] "RemoveContainer" containerID="300cde199404a02d02f87d507464ccc6e07b5825de3349c3d881d263f6960063"
Jul 16 21:16:54.320043 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:16:54.320028    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8_odh-ai-gateway-infra(bf66233d-e555-4660-87bc-d0470f3a98db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8" podUID="bf66233d-e555-4660-87bc-d0470f3a98db"
Jul 16 21:16:54.329178 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:54.329158    2530 scope.go:117] "RemoveContainer" containerID="13330741d7c048248d6fc31a895133a771a01ddd656a762729b0473cbeef1977"
Jul 16 21:16:54.369531 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:54.369500    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48"
Jul 16 21:16:54.508704 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:16:54.508674    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c6bf82_64c0_40e3_bf36_09cc44f8f48f.slice/crio-828ba0db8cb593454094e39063e991a0d53fc533ee2abf6853f19ec4bdc709cf WatchSource:0}: Error finding container 828ba0db8cb593454094e39063e991a0d53fc533ee2abf6853f19ec4bdc709cf: Status 404 returned error can't find the container with id 828ba0db8cb593454094e39063e991a0d53fc533ee2abf6853f19ec4bdc709cf
Jul 16 21:16:54.509592 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:54.509566    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48"]
Jul 16 21:16:55.326068 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:55.326037    2530 scope.go:117] "RemoveContainer" containerID="300cde199404a02d02f87d507464ccc6e07b5825de3349c3d881d263f6960063"
Jul 16 21:16:55.326592 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:16:55.326277    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8_odh-ai-gateway-infra(bf66233d-e555-4660-87bc-d0470f3a98db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8" podUID="bf66233d-e555-4660-87bc-d0470f3a98db"
Jul 16 21:16:55.327569 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:55.327550    2530 scope.go:117] "RemoveContainer" containerID="5c1b0bdb31bd82565ada7715b06ae47aae284bb9abf4ecfc9163cafa37b70ff3"
Jul 16 21:16:55.327758 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:16:55.327738    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j_odh-ai-gateway-infra(abe76fc5-58ae-4dde-b56b-836dcfe76401)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j" podUID="abe76fc5-58ae-4dde-b56b-836dcfe76401"
Jul 16 21:16:55.328822 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:55.328802    2530 generic.go:356] "Generic (PLEG): container finished" podID="d8c6bf82-64c0-40e3-bf36-09cc44f8f48f" containerID="6fcdccad482c614fe9a86c3fafc919f83166825d4352eadfd74bbb3c66d7be3c" exitCode=6
Jul 16 21:16:55.328929 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:55.328876    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48" event={"ID":"d8c6bf82-64c0-40e3-bf36-09cc44f8f48f","Type":"ContainerDied","Data":"6fcdccad482c614fe9a86c3fafc919f83166825d4352eadfd74bbb3c66d7be3c"}
Jul 16 21:16:55.328997 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:55.328956    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48" event={"ID":"d8c6bf82-64c0-40e3-bf36-09cc44f8f48f","Type":"ContainerStarted","Data":"828ba0db8cb593454094e39063e991a0d53fc533ee2abf6853f19ec4bdc709cf"}
Jul 16 21:16:55.329035 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:55.329001    2530 scope.go:117] "RemoveContainer" containerID="6fcdccad482c614fe9a86c3fafc919f83166825d4352eadfd74bbb3c66d7be3c"
Jul 16 21:16:55.502950 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:55.502913    2530 scope.go:117] "RemoveContainer" containerID="13a854eaaa13018a7d66d2c62e8bfffd5f835a04688ea9038404d1ca81094cdb"
Jul 16 21:16:55.574614 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:55.574580    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9"]
Jul 16 21:16:55.578133 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:55.578084    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9"
Jul 16 21:16:55.586187 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:55.586163    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9"]
Jul 16 21:16:55.654239 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:55.654195    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba61385c-c6f1-4a82-87d9-125ad45e918a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9\" (UID: \"ba61385c-c6f1-4a82-87d9-125ad45e918a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9"
Jul 16 21:16:55.755369 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:55.755266    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba61385c-c6f1-4a82-87d9-125ad45e918a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9\" (UID: \"ba61385c-c6f1-4a82-87d9-125ad45e918a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9"
Jul 16 21:16:55.755903 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:55.755881    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba61385c-c6f1-4a82-87d9-125ad45e918a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9\" (UID: \"ba61385c-c6f1-4a82-87d9-125ad45e918a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9"
Jul 16 21:16:55.889223 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:55.889182    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9"
Jul 16 21:16:56.028992 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:56.028963    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9"]
Jul 16 21:16:56.030942 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:16:56.030912    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba61385c_c6f1_4a82_87d9_125ad45e918a.slice/crio-4e9f44c310ee42814afd6e464dd723759e7f13f24f7a9f1c7fecac29a49750a4 WatchSource:0}: Error finding container 4e9f44c310ee42814afd6e464dd723759e7f13f24f7a9f1c7fecac29a49750a4: Status 404 returned error can't find the container with id 4e9f44c310ee42814afd6e464dd723759e7f13f24f7a9f1c7fecac29a49750a4
Jul 16 21:16:56.335260 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:56.335223    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9" event={"ID":"ba61385c-c6f1-4a82-87d9-125ad45e918a","Type":"ContainerStarted","Data":"3b60e5e663dbe6412df18a4a7d6abe8102faffb22f43fdfd14ef128371504503"}
Jul 16 21:16:56.335647 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:56.335294    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9" event={"ID":"ba61385c-c6f1-4a82-87d9-125ad45e918a","Type":"ContainerStarted","Data":"4e9f44c310ee42814afd6e464dd723759e7f13f24f7a9f1c7fecac29a49750a4"}
Jul 16 21:16:56.337016 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:56.336992    2530 generic.go:356] "Generic (PLEG): container finished" podID="d8c6bf82-64c0-40e3-bf36-09cc44f8f48f" containerID="32481f88714fbc27011bbd5b23be70853d9d9fb860ef8acf5868d48834ac5b41" exitCode=6
Jul 16 21:16:56.337162 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:56.337069    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48" event={"ID":"d8c6bf82-64c0-40e3-bf36-09cc44f8f48f","Type":"ContainerDied","Data":"32481f88714fbc27011bbd5b23be70853d9d9fb860ef8acf5868d48834ac5b41"}
Jul 16 21:16:56.337162 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:56.337142    2530 scope.go:117] "RemoveContainer" containerID="6fcdccad482c614fe9a86c3fafc919f83166825d4352eadfd74bbb3c66d7be3c"
Jul 16 21:16:56.337282 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:56.337265    2530 scope.go:117] "RemoveContainer" containerID="32481f88714fbc27011bbd5b23be70853d9d9fb860ef8acf5868d48834ac5b41"
Jul 16 21:16:56.337496 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:16:56.337467    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-npw48_odh-ai-gateway-infra(d8c6bf82-64c0-40e3-bf36-09cc44f8f48f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48" podUID="d8c6bf82-64c0-40e3-bf36-09cc44f8f48f"
Jul 16 21:16:56.339204 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:56.339185    2530 generic.go:356] "Generic (PLEG): container finished" podID="a9713298-4d03-441c-8535-8831f39f0ca4" containerID="03d0f4616f8fce599d00e9ad305a70a6d67af4d56f0da17b182a85826205ace9" exitCode=6
Jul 16 21:16:56.339291 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:56.339215    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2" event={"ID":"a9713298-4d03-441c-8535-8831f39f0ca4","Type":"ContainerDied","Data":"03d0f4616f8fce599d00e9ad305a70a6d67af4d56f0da17b182a85826205ace9"}
Jul 16 21:16:56.339460 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:56.339447    2530 scope.go:117] "RemoveContainer" containerID="03d0f4616f8fce599d00e9ad305a70a6d67af4d56f0da17b182a85826205ace9"
Jul 16 21:16:56.339631 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:16:56.339616    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2_odh-ai-gateway-infra(a9713298-4d03-441c-8535-8831f39f0ca4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2" podUID="a9713298-4d03-441c-8535-8831f39f0ca4"
Jul 16 21:16:56.348204 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:56.348176    2530 scope.go:117] "RemoveContainer" containerID="13a854eaaa13018a7d66d2c62e8bfffd5f835a04688ea9038404d1ca81094cdb"
Jul 16 21:16:56.353083 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:56.353032    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9" podStartSLOduration=1.353012631 podStartE2EDuration="1.353012631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:16:56.35103217 +0000 UTC m=+2654.372619950" watchObservedRunningTime="2026-07-16 21:16:56.353012631 +0000 UTC m=+2654.374600397"
Jul 16 21:16:57.345865 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:57.345827    2530 generic.go:356] "Generic (PLEG): container finished" podID="ba61385c-c6f1-4a82-87d9-125ad45e918a" containerID="3b60e5e663dbe6412df18a4a7d6abe8102faffb22f43fdfd14ef128371504503" exitCode=0
Jul 16 21:16:57.346394 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:57.345912    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9" event={"ID":"ba61385c-c6f1-4a82-87d9-125ad45e918a","Type":"ContainerDied","Data":"3b60e5e663dbe6412df18a4a7d6abe8102faffb22f43fdfd14ef128371504503"}
Jul 16 21:16:57.347673 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:57.347651    2530 scope.go:117] "RemoveContainer" containerID="32481f88714fbc27011bbd5b23be70853d9d9fb860ef8acf5868d48834ac5b41"
Jul 16 21:16:57.347873 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:16:57.347854    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-npw48_odh-ai-gateway-infra(d8c6bf82-64c0-40e3-bf36-09cc44f8f48f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48" podUID="d8c6bf82-64c0-40e3-bf36-09cc44f8f48f"
Jul 16 21:16:57.389431 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:57.389389    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2"]
Jul 16 21:16:57.533674 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:57.533642    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2"
Jul 16 21:16:57.672729 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:57.672634    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9713298-4d03-441c-8535-8831f39f0ca4-maas-api-service-ca\") pod \"a9713298-4d03-441c-8535-8831f39f0ca4\" (UID: \"a9713298-4d03-441c-8535-8831f39f0ca4\") "
Jul 16 21:16:57.673023 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:57.672982    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9713298-4d03-441c-8535-8831f39f0ca4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a9713298-4d03-441c-8535-8831f39f0ca4" (UID: "a9713298-4d03-441c-8535-8831f39f0ca4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:16:57.774205 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:57.774162    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9713298-4d03-441c-8535-8831f39f0ca4-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:16:58.355267 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:58.355229    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2"
Jul 16 21:16:58.355714 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:58.355268    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2" event={"ID":"a9713298-4d03-441c-8535-8831f39f0ca4","Type":"ContainerDied","Data":"a5f392993a4b25ceef407d602f8fe008af626e0eba4e20695593032064b3e12f"}
Jul 16 21:16:58.355714 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:58.355335    2530 scope.go:117] "RemoveContainer" containerID="03d0f4616f8fce599d00e9ad305a70a6d67af4d56f0da17b182a85826205ace9"
Jul 16 21:16:58.380696 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:58.380654    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2"]
Jul 16 21:16:58.383682 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:58.383649    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-c4pw2"]
Jul 16 21:16:58.501871 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:58.501834    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9"
Jul 16 21:16:58.508576 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:58.508537    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9713298-4d03-441c-8535-8831f39f0ca4" path="/var/lib/kubelet/pods/a9713298-4d03-441c-8535-8831f39f0ca4/volumes"
Jul 16 21:16:58.581397 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:58.581362    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba61385c-c6f1-4a82-87d9-125ad45e918a-maas-api-service-ca\") pod \"ba61385c-c6f1-4a82-87d9-125ad45e918a\" (UID: \"ba61385c-c6f1-4a82-87d9-125ad45e918a\") "
Jul 16 21:16:58.581732 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:58.581710    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba61385c-c6f1-4a82-87d9-125ad45e918a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba61385c-c6f1-4a82-87d9-125ad45e918a" (UID: "ba61385c-c6f1-4a82-87d9-125ad45e918a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:16:58.682881 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:58.682783    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba61385c-c6f1-4a82-87d9-125ad45e918a-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:16:59.365687 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:59.365652    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9" event={"ID":"ba61385c-c6f1-4a82-87d9-125ad45e918a","Type":"ContainerDied","Data":"4e9f44c310ee42814afd6e464dd723759e7f13f24f7a9f1c7fecac29a49750a4"}
Jul 16 21:16:59.366081 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:59.365691    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9"
Jul 16 21:16:59.366081 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:16:59.365701    2530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e9f44c310ee42814afd6e464dd723759e7f13f24f7a9f1c7fecac29a49750a4"
Jul 16 21:17:06.042888 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.042851    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"]
Jul 16 21:17:06.043435 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.043139    2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52" podUID="d1c96064-14f6-44d5-b355-b992c5c85794" containerName="maas-api" containerID="cri-o://e58a0ba9a7f52ef94a719cc56596d9f4511f42e91856c473d326dbcaa72618ce" gracePeriod=30
Jul 16 21:17:06.137909 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.137862    2530 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52" podUID="d1c96064-14f6-44d5-b355-b992c5c85794" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.75:8443/health\": dial tcp 10.133.0.75:8443: connect: connection refused"
Jul 16 21:17:06.295687 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.295616    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"
Jul 16 21:17:06.357515 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.357474    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcfzz\" (UniqueName: \"kubernetes.io/projected/d1c96064-14f6-44d5-b355-b992c5c85794-kube-api-access-gcfzz\") pod \"d1c96064-14f6-44d5-b355-b992c5c85794\" (UID: \"d1c96064-14f6-44d5-b355-b992c5c85794\") "
Jul 16 21:17:06.357515 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.357516    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c96064-14f6-44d5-b355-b992c5c85794-openshift-service-ca\") pod \"d1c96064-14f6-44d5-b355-b992c5c85794\" (UID: \"d1c96064-14f6-44d5-b355-b992c5c85794\") "
Jul 16 21:17:06.357755 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.357579    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c96064-14f6-44d5-b355-b992c5c85794-maas-api-tls\") pod \"d1c96064-14f6-44d5-b355-b992c5c85794\" (UID: \"d1c96064-14f6-44d5-b355-b992c5c85794\") "
Jul 16 21:17:06.357959 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.357937    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c96064-14f6-44d5-b355-b992c5c85794-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d1c96064-14f6-44d5-b355-b992c5c85794" (UID: "d1c96064-14f6-44d5-b355-b992c5c85794"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:17:06.359897 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.359872    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c96064-14f6-44d5-b355-b992c5c85794-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d1c96064-14f6-44d5-b355-b992c5c85794" (UID: "d1c96064-14f6-44d5-b355-b992c5c85794"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:17:06.360025 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.359931    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c96064-14f6-44d5-b355-b992c5c85794-kube-api-access-gcfzz" (OuterVolumeSpecName: "kube-api-access-gcfzz") pod "d1c96064-14f6-44d5-b355-b992c5c85794" (UID: "d1c96064-14f6-44d5-b355-b992c5c85794"). InnerVolumeSpecName "kube-api-access-gcfzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:17:06.399426 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.399381    2530 generic.go:356] "Generic (PLEG): container finished" podID="d1c96064-14f6-44d5-b355-b992c5c85794" containerID="e58a0ba9a7f52ef94a719cc56596d9f4511f42e91856c473d326dbcaa72618ce" exitCode=0
Jul 16 21:17:06.399620 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.399434    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"
Jul 16 21:17:06.399620 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.399455    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52" event={"ID":"d1c96064-14f6-44d5-b355-b992c5c85794","Type":"ContainerDied","Data":"e58a0ba9a7f52ef94a719cc56596d9f4511f42e91856c473d326dbcaa72618ce"}
Jul 16 21:17:06.399620 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.399519    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52" event={"ID":"d1c96064-14f6-44d5-b355-b992c5c85794","Type":"ContainerDied","Data":"d59a9b0a52ca72bb9763a3bca8f01ed70a335e8602ffc27ee0cf09226936d441"}
Jul 16 21:17:06.399620 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.399543    2530 scope.go:117] "RemoveContainer" containerID="e58a0ba9a7f52ef94a719cc56596d9f4511f42e91856c473d326dbcaa72618ce"
Jul 16 21:17:06.409395 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.409375    2530 scope.go:117] "RemoveContainer" containerID="e58a0ba9a7f52ef94a719cc56596d9f4511f42e91856c473d326dbcaa72618ce"
Jul 16 21:17:06.409718 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:17:06.409686    2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e58a0ba9a7f52ef94a719cc56596d9f4511f42e91856c473d326dbcaa72618ce\": container with ID starting with e58a0ba9a7f52ef94a719cc56596d9f4511f42e91856c473d326dbcaa72618ce not found: ID does not exist" containerID="e58a0ba9a7f52ef94a719cc56596d9f4511f42e91856c473d326dbcaa72618ce"
Jul 16 21:17:06.409821 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.409729    2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e58a0ba9a7f52ef94a719cc56596d9f4511f42e91856c473d326dbcaa72618ce"} err="failed to get container status \"e58a0ba9a7f52ef94a719cc56596d9f4511f42e91856c473d326dbcaa72618ce\": rpc error: code = NotFound desc = could not find container \"e58a0ba9a7f52ef94a719cc56596d9f4511f42e91856c473d326dbcaa72618ce\": container with ID starting with e58a0ba9a7f52ef94a719cc56596d9f4511f42e91856c473d326dbcaa72618ce not found: ID does not exist"
Jul 16 21:17:06.420708 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.420658    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"]
Jul 16 21:17:06.424881 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.424851    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e9d9b90e-5fc4dbddf5-q9q52"]
Jul 16 21:17:06.458415 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.458383    2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcfzz\" (UniqueName: \"kubernetes.io/projected/d1c96064-14f6-44d5-b355-b992c5c85794-kube-api-access-gcfzz\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:17:06.458415 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.458413    2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c96064-14f6-44d5-b355-b992c5c85794-openshift-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:17:06.458570 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.458424    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c96064-14f6-44d5-b355-b992c5c85794-maas-api-tls\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:17:06.502188 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.502163    2530 scope.go:117] "RemoveContainer" containerID="5c1b0bdb31bd82565ada7715b06ae47aae284bb9abf4ecfc9163cafa37b70ff3"
Jul 16 21:17:06.508225 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:06.508188    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1c96064-14f6-44d5-b355-b992c5c85794" path="/var/lib/kubelet/pods/d1c96064-14f6-44d5-b355-b992c5c85794/volumes"
Jul 16 21:17:07.406220 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:07.406184    2530 generic.go:356] "Generic (PLEG): container finished" podID="abe76fc5-58ae-4dde-b56b-836dcfe76401" containerID="97317331db1486e7a806558c7fe2b77a68de7455970e08da7901fee53d95c164" exitCode=6
Jul 16 21:17:07.406666 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:07.406258    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j" event={"ID":"abe76fc5-58ae-4dde-b56b-836dcfe76401","Type":"ContainerDied","Data":"97317331db1486e7a806558c7fe2b77a68de7455970e08da7901fee53d95c164"}
Jul 16 21:17:07.406666 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:07.406331    2530 scope.go:117] "RemoveContainer" containerID="5c1b0bdb31bd82565ada7715b06ae47aae284bb9abf4ecfc9163cafa37b70ff3"
Jul 16 21:17:07.406666 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:07.406655    2530 scope.go:117] "RemoveContainer" containerID="97317331db1486e7a806558c7fe2b77a68de7455970e08da7901fee53d95c164"
Jul 16 21:17:07.406917 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:17:07.406894    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j_odh-ai-gateway-infra(abe76fc5-58ae-4dde-b56b-836dcfe76401)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j" podUID="abe76fc5-58ae-4dde-b56b-836dcfe76401"
Jul 16 21:17:08.432708 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:08.432663    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j"]
Jul 16 21:17:08.503042 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:08.503008    2530 scope.go:117] "RemoveContainer" containerID="32481f88714fbc27011bbd5b23be70853d9d9fb860ef8acf5868d48834ac5b41"
Jul 16 21:17:08.503272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:08.503179    2530 scope.go:117] "RemoveContainer" containerID="300cde199404a02d02f87d507464ccc6e07b5825de3349c3d881d263f6960063"
Jul 16 21:17:08.580116 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:08.580072    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j"
Jul 16 21:17:08.676799 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:08.676760    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe76fc5-58ae-4dde-b56b-836dcfe76401-maas-api-service-ca\") pod \"abe76fc5-58ae-4dde-b56b-836dcfe76401\" (UID: \"abe76fc5-58ae-4dde-b56b-836dcfe76401\") "
Jul 16 21:17:08.682604 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:08.677626    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe76fc5-58ae-4dde-b56b-836dcfe76401-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "abe76fc5-58ae-4dde-b56b-836dcfe76401" (UID: "abe76fc5-58ae-4dde-b56b-836dcfe76401"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:17:08.777967 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:08.777927    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe76fc5-58ae-4dde-b56b-836dcfe76401-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:17:09.420349 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:09.420251    2530 generic.go:356] "Generic (PLEG): container finished" podID="d8c6bf82-64c0-40e3-bf36-09cc44f8f48f" containerID="d4b3db4a5ad05527e338d4e5dd792f609916c9578c093c6a2b4dddb8f414d0fd" exitCode=6
Jul 16 21:17:09.420349 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:09.420329    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48" event={"ID":"d8c6bf82-64c0-40e3-bf36-09cc44f8f48f","Type":"ContainerDied","Data":"d4b3db4a5ad05527e338d4e5dd792f609916c9578c093c6a2b4dddb8f414d0fd"}
Jul 16 21:17:09.420564 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:09.420418    2530 scope.go:117] "RemoveContainer" containerID="32481f88714fbc27011bbd5b23be70853d9d9fb860ef8acf5868d48834ac5b41"
Jul 16 21:17:09.420800 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:09.420780    2530 scope.go:117] "RemoveContainer" containerID="d4b3db4a5ad05527e338d4e5dd792f609916c9578c093c6a2b4dddb8f414d0fd"
Jul 16 21:17:09.421044 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:17:09.421017    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-npw48_odh-ai-gateway-infra(d8c6bf82-64c0-40e3-bf36-09cc44f8f48f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48" podUID="d8c6bf82-64c0-40e3-bf36-09cc44f8f48f"
Jul 16 21:17:09.422342 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:09.422320    2530 generic.go:356] "Generic (PLEG): container finished" podID="bf66233d-e555-4660-87bc-d0470f3a98db" containerID="63ba79a8e39131bcd02fccdfca4583da88867b8b6ea8aeedd4f87980e535e792" exitCode=6
Jul 16 21:17:09.422466 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:09.422395    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8" event={"ID":"bf66233d-e555-4660-87bc-d0470f3a98db","Type":"ContainerDied","Data":"63ba79a8e39131bcd02fccdfca4583da88867b8b6ea8aeedd4f87980e535e792"}
Jul 16 21:17:09.422706 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:09.422689    2530 scope.go:117] "RemoveContainer" containerID="63ba79a8e39131bcd02fccdfca4583da88867b8b6ea8aeedd4f87980e535e792"
Jul 16 21:17:09.422914 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:17:09.422880    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8_odh-ai-gateway-infra(bf66233d-e555-4660-87bc-d0470f3a98db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8" podUID="bf66233d-e555-4660-87bc-d0470f3a98db"
Jul 16 21:17:09.423674 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:09.423657    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j" event={"ID":"abe76fc5-58ae-4dde-b56b-836dcfe76401","Type":"ContainerDied","Data":"db7fe28ec3cd37986f663fc96bbd83cbfb4578e18b5a527dc8ff400225829014"}
Jul 16 21:17:09.423753 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:09.423687    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j"
Jul 16 21:17:09.432086 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:09.432065    2530 scope.go:117] "RemoveContainer" containerID="300cde199404a02d02f87d507464ccc6e07b5825de3349c3d881d263f6960063"
Jul 16 21:17:09.444470 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:09.444318    2530 scope.go:117] "RemoveContainer" containerID="97317331db1486e7a806558c7fe2b77a68de7455970e08da7901fee53d95c164"
Jul 16 21:17:09.460142 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:09.460084    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j"]
Jul 16 21:17:09.462478 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:09.462448    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-74r4j"]
Jul 16 21:17:10.445665 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:10.445627    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48"]
Jul 16 21:17:10.459983 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:10.459947    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8"]
Jul 16 21:17:10.510429 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:10.510389    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abe76fc5-58ae-4dde-b56b-836dcfe76401" path="/var/lib/kubelet/pods/abe76fc5-58ae-4dde-b56b-836dcfe76401/volumes"
Jul 16 21:17:10.611900 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:10.610590    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48"
Jul 16 21:17:10.647246 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:10.647214    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8"
Jul 16 21:17:10.694088 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:10.694055    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6bf82-64c0-40e3-bf36-09cc44f8f48f-maas-api-service-ca\") pod \"d8c6bf82-64c0-40e3-bf36-09cc44f8f48f\" (UID: \"d8c6bf82-64c0-40e3-bf36-09cc44f8f48f\") "
Jul 16 21:17:10.694266 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:10.694178    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf66233d-e555-4660-87bc-d0470f3a98db-maas-api-service-ca\") pod \"bf66233d-e555-4660-87bc-d0470f3a98db\" (UID: \"bf66233d-e555-4660-87bc-d0470f3a98db\") "
Jul 16 21:17:10.694431 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:10.694409    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c6bf82-64c0-40e3-bf36-09cc44f8f48f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d8c6bf82-64c0-40e3-bf36-09cc44f8f48f" (UID: "d8c6bf82-64c0-40e3-bf36-09cc44f8f48f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:17:10.694505 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:10.694487    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf66233d-e555-4660-87bc-d0470f3a98db-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bf66233d-e555-4660-87bc-d0470f3a98db" (UID: "bf66233d-e555-4660-87bc-d0470f3a98db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:17:10.795184 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:10.795151    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf66233d-e555-4660-87bc-d0470f3a98db-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:17:10.795184 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:10.795181    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c6bf82-64c0-40e3-bf36-09cc44f8f48f-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:17:11.438383 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:11.438355    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48"
Jul 16 21:17:11.438558 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:11.438355    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48" event={"ID":"d8c6bf82-64c0-40e3-bf36-09cc44f8f48f","Type":"ContainerDied","Data":"828ba0db8cb593454094e39063e991a0d53fc533ee2abf6853f19ec4bdc709cf"}
Jul 16 21:17:11.438558 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:11.438512    2530 scope.go:117] "RemoveContainer" containerID="d4b3db4a5ad05527e338d4e5dd792f609916c9578c093c6a2b4dddb8f414d0fd"
Jul 16 21:17:11.442239 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:11.442080    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8" event={"ID":"bf66233d-e555-4660-87bc-d0470f3a98db","Type":"ContainerDied","Data":"fb624b5727da87bd624cb9126c2249f44569239e5a3dbf3cc241496d3a511add"}
Jul 16 21:17:11.442516 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:11.442492    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8"
Jul 16 21:17:11.450345 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:11.450166    2530 scope.go:117] "RemoveContainer" containerID="63ba79a8e39131bcd02fccdfca4583da88867b8b6ea8aeedd4f87980e535e792"
Jul 16 21:17:11.464306 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:11.464277    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48"]
Jul 16 21:17:11.472499 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:11.472470    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-npw48"]
Jul 16 21:17:11.483131 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:11.483069    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8"]
Jul 16 21:17:11.489455 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:11.489422    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-jnnw8"]
Jul 16 21:17:12.508231 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:12.508194    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf66233d-e555-4660-87bc-d0470f3a98db" path="/var/lib/kubelet/pods/bf66233d-e555-4660-87bc-d0470f3a98db/volumes"
Jul 16 21:17:12.508622 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:12.508531    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8c6bf82-64c0-40e3-bf36-09cc44f8f48f" path="/var/lib/kubelet/pods/d8c6bf82-64c0-40e3-bf36-09cc44f8f48f/volumes"
Jul 16 21:17:29.310115 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310071    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z"]
Jul 16 21:17:29.310617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310497    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe76fc5-58ae-4dde-b56b-836dcfe76401" containerName="revoke-keys"
Jul 16 21:17:29.310617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310513    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe76fc5-58ae-4dde-b56b-836dcfe76401" containerName="revoke-keys"
Jul 16 21:17:29.310617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310529    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf66233d-e555-4660-87bc-d0470f3a98db" containerName="revoke-keys"
Jul 16 21:17:29.310617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310539    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf66233d-e555-4660-87bc-d0470f3a98db" containerName="revoke-keys"
Jul 16 21:17:29.310617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310551    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8c6bf82-64c0-40e3-bf36-09cc44f8f48f" containerName="revoke-keys"
Jul 16 21:17:29.310617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310560    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8c6bf82-64c0-40e3-bf36-09cc44f8f48f" containerName="revoke-keys"
Jul 16 21:17:29.310617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310570    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8c6bf82-64c0-40e3-bf36-09cc44f8f48f" containerName="revoke-keys"
Jul 16 21:17:29.310617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310576    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8c6bf82-64c0-40e3-bf36-09cc44f8f48f" containerName="revoke-keys"
Jul 16 21:17:29.310617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310582    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba61385c-c6f1-4a82-87d9-125ad45e918a" containerName="revoke-keys"
Jul 16 21:17:29.310617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310587    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba61385c-c6f1-4a82-87d9-125ad45e918a" containerName="revoke-keys"
Jul 16 21:17:29.310617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310596    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf66233d-e555-4660-87bc-d0470f3a98db" containerName="revoke-keys"
Jul 16 21:17:29.310617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310601    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf66233d-e555-4660-87bc-d0470f3a98db" containerName="revoke-keys"
Jul 16 21:17:29.310617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310608    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe76fc5-58ae-4dde-b56b-836dcfe76401" containerName="revoke-keys"
Jul 16 21:17:29.310617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310614    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe76fc5-58ae-4dde-b56b-836dcfe76401" containerName="revoke-keys"
Jul 16 21:17:29.310617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310621    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe76fc5-58ae-4dde-b56b-836dcfe76401" containerName="revoke-keys"
Jul 16 21:17:29.310617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310626    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe76fc5-58ae-4dde-b56b-836dcfe76401" containerName="revoke-keys"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310633    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9713298-4d03-441c-8535-8831f39f0ca4" containerName="revoke-keys"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310638    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9713298-4d03-441c-8535-8831f39f0ca4" containerName="revoke-keys"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310643    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9713298-4d03-441c-8535-8831f39f0ca4" containerName="revoke-keys"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310648    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9713298-4d03-441c-8535-8831f39f0ca4" containerName="revoke-keys"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310657    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8c6bf82-64c0-40e3-bf36-09cc44f8f48f" containerName="revoke-keys"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310662    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8c6bf82-64c0-40e3-bf36-09cc44f8f48f" containerName="revoke-keys"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310668    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9713298-4d03-441c-8535-8831f39f0ca4" containerName="revoke-keys"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310672    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9713298-4d03-441c-8535-8831f39f0ca4" containerName="revoke-keys"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310681    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1c96064-14f6-44d5-b355-b992c5c85794" containerName="maas-api"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310686    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1c96064-14f6-44d5-b355-b992c5c85794" containerName="maas-api"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310740    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8c6bf82-64c0-40e3-bf36-09cc44f8f48f" containerName="revoke-keys"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310747    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe76fc5-58ae-4dde-b56b-836dcfe76401" containerName="revoke-keys"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310752    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8c6bf82-64c0-40e3-bf36-09cc44f8f48f" containerName="revoke-keys"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310758    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9713298-4d03-441c-8535-8831f39f0ca4" containerName="revoke-keys"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310762    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf66233d-e555-4660-87bc-d0470f3a98db" containerName="revoke-keys"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310767    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf66233d-e555-4660-87bc-d0470f3a98db" containerName="revoke-keys"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310772    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf66233d-e555-4660-87bc-d0470f3a98db" containerName="revoke-keys"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310782    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1c96064-14f6-44d5-b355-b992c5c85794" containerName="maas-api"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310786    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9713298-4d03-441c-8535-8831f39f0ca4" containerName="revoke-keys"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310791    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe76fc5-58ae-4dde-b56b-836dcfe76401" containerName="revoke-keys"
Jul 16 21:17:29.311226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.310797    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba61385c-c6f1-4a82-87d9-125ad45e918a" containerName="revoke-keys"
Jul 16 21:17:29.314178 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.314154    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z"
Jul 16 21:17:29.316756 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.316730    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:17:29.334262 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.334212    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z"]
Jul 16 21:17:29.461224 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.461185    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47c9445-c770-4dd2-af56-4df63b61be5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z\" (UID: \"f47c9445-c770-4dd2-af56-4df63b61be5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z"
Jul 16 21:17:29.561913 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.561801    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47c9445-c770-4dd2-af56-4df63b61be5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z\" (UID: \"f47c9445-c770-4dd2-af56-4df63b61be5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z"
Jul 16 21:17:29.562530 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.562500    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47c9445-c770-4dd2-af56-4df63b61be5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z\" (UID: \"f47c9445-c770-4dd2-af56-4df63b61be5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z"
Jul 16 21:17:29.625778 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.625736    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z"
Jul 16 21:17:29.769250 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:29.769225    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z"]
Jul 16 21:17:29.771886 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:17:29.771852    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47c9445_c770_4dd2_af56_4df63b61be5c.slice/crio-965f035b811717c9f151f7bef0859c2989e7a856466c560051d4b8e86610842c WatchSource:0}: Error finding container 965f035b811717c9f151f7bef0859c2989e7a856466c560051d4b8e86610842c: Status 404 returned error can't find the container with id 965f035b811717c9f151f7bef0859c2989e7a856466c560051d4b8e86610842c
Jul 16 21:17:30.543481 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:30.543445    2530 generic.go:356] "Generic (PLEG): container finished" podID="f47c9445-c770-4dd2-af56-4df63b61be5c" containerID="42dba0f48c3234d2bd2cd22d3422eb4ee86f3a6e7467c6a66d4a9c62e1f3e5bf" exitCode=6
Jul 16 21:17:30.543902 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:30.543523    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z" event={"ID":"f47c9445-c770-4dd2-af56-4df63b61be5c","Type":"ContainerDied","Data":"42dba0f48c3234d2bd2cd22d3422eb4ee86f3a6e7467c6a66d4a9c62e1f3e5bf"}
Jul 16 21:17:30.543902 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:30.543590    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z" event={"ID":"f47c9445-c770-4dd2-af56-4df63b61be5c","Type":"ContainerStarted","Data":"965f035b811717c9f151f7bef0859c2989e7a856466c560051d4b8e86610842c"}
Jul 16 21:17:30.543902 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:30.543758    2530 scope.go:117] "RemoveContainer" containerID="42dba0f48c3234d2bd2cd22d3422eb4ee86f3a6e7467c6a66d4a9c62e1f3e5bf"
Jul 16 21:17:31.550211 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:31.550086    2530 generic.go:356] "Generic (PLEG): container finished" podID="f47c9445-c770-4dd2-af56-4df63b61be5c" containerID="6cf3e87de4be2b1ce5321d7c1620fb920ca5f8be9738f42129935339ff6cfc6d" exitCode=6
Jul 16 21:17:31.550211 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:31.550146    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z" event={"ID":"f47c9445-c770-4dd2-af56-4df63b61be5c","Type":"ContainerDied","Data":"6cf3e87de4be2b1ce5321d7c1620fb920ca5f8be9738f42129935339ff6cfc6d"}
Jul 16 21:17:31.550211 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:31.550213    2530 scope.go:117] "RemoveContainer" containerID="42dba0f48c3234d2bd2cd22d3422eb4ee86f3a6e7467c6a66d4a9c62e1f3e5bf"
Jul 16 21:17:31.550754 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:31.550443    2530 scope.go:117] "RemoveContainer" containerID="6cf3e87de4be2b1ce5321d7c1620fb920ca5f8be9738f42129935339ff6cfc6d"
Jul 16 21:17:31.550754 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:17:31.550655    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z_odh-ai-gateway-infra(f47c9445-c770-4dd2-af56-4df63b61be5c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z" podUID="f47c9445-c770-4dd2-af56-4df63b61be5c"
Jul 16 21:17:32.557295 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:32.557262    2530 scope.go:117] "RemoveContainer" containerID="6cf3e87de4be2b1ce5321d7c1620fb920ca5f8be9738f42129935339ff6cfc6d"
Jul 16 21:17:32.557797 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:17:32.557521    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z_odh-ai-gateway-infra(f47c9445-c770-4dd2-af56-4df63b61be5c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z" podUID="f47c9445-c770-4dd2-af56-4df63b61be5c"
Jul 16 21:17:42.154916 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:42.154825    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc"]
Jul 16 21:17:42.155302 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:42.155172    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf66233d-e555-4660-87bc-d0470f3a98db" containerName="revoke-keys"
Jul 16 21:17:42.155302 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:42.155184    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf66233d-e555-4660-87bc-d0470f3a98db" containerName="revoke-keys"
Jul 16 21:17:42.155302 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:42.155240    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe76fc5-58ae-4dde-b56b-836dcfe76401" containerName="revoke-keys"
Jul 16 21:17:42.155302 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:42.155247    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8c6bf82-64c0-40e3-bf36-09cc44f8f48f" containerName="revoke-keys"
Jul 16 21:17:42.155302 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:42.155253    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9713298-4d03-441c-8535-8831f39f0ca4" containerName="revoke-keys"
Jul 16 21:17:42.159328 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:42.159302    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc"
Jul 16 21:17:42.165268 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:42.165234    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc"]
Jul 16 21:17:42.277304 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:42.277265    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d193f689-91c1-4484-b5d8-dbbf3adae841-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc\" (UID: \"d193f689-91c1-4484-b5d8-dbbf3adae841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc"
Jul 16 21:17:42.377954 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:42.377911    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d193f689-91c1-4484-b5d8-dbbf3adae841-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc\" (UID: \"d193f689-91c1-4484-b5d8-dbbf3adae841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc"
Jul 16 21:17:42.378599 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:42.378574    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d193f689-91c1-4484-b5d8-dbbf3adae841-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc\" (UID: \"d193f689-91c1-4484-b5d8-dbbf3adae841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc"
Jul 16 21:17:42.471374 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:42.471337    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc"
Jul 16 21:17:42.623787 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:42.623755    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc"]
Jul 16 21:17:42.625551 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:17:42.625521    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd193f689_91c1_4484_b5d8_dbbf3adae841.slice/crio-275133bff7e242d762ff09debb6afa9ce9f5286c2eba1877cbbb185f891695e7 WatchSource:0}: Error finding container 275133bff7e242d762ff09debb6afa9ce9f5286c2eba1877cbbb185f891695e7: Status 404 returned error can't find the container with id 275133bff7e242d762ff09debb6afa9ce9f5286c2eba1877cbbb185f891695e7
Jul 16 21:17:43.607760 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:43.607659    2530 generic.go:356] "Generic (PLEG): container finished" podID="d193f689-91c1-4484-b5d8-dbbf3adae841" containerID="3fe5d4b26172dc73378c762c981b0c58b71ebe33f5647b6fb42284fa8b2bcc8a" exitCode=6
Jul 16 21:17:43.608175 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:43.607755    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc" event={"ID":"d193f689-91c1-4484-b5d8-dbbf3adae841","Type":"ContainerDied","Data":"3fe5d4b26172dc73378c762c981b0c58b71ebe33f5647b6fb42284fa8b2bcc8a"}
Jul 16 21:17:43.608175 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:43.607824    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc" event={"ID":"d193f689-91c1-4484-b5d8-dbbf3adae841","Type":"ContainerStarted","Data":"275133bff7e242d762ff09debb6afa9ce9f5286c2eba1877cbbb185f891695e7"}
Jul 16 21:17:43.608175 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:43.607893    2530 scope.go:117] "RemoveContainer" containerID="3fe5d4b26172dc73378c762c981b0c58b71ebe33f5647b6fb42284fa8b2bcc8a"
Jul 16 21:17:44.064409 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:44.064365    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2"]
Jul 16 21:17:44.067845 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:44.067815    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2"
Jul 16 21:17:44.082285 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:44.082248    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2"]
Jul 16 21:17:44.193952 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:44.193850    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f9cb38e-7537-41dc-85cf-7fa55dc65a24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2\" (UID: \"8f9cb38e-7537-41dc-85cf-7fa55dc65a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2"
Jul 16 21:17:44.295357 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:44.295299    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f9cb38e-7537-41dc-85cf-7fa55dc65a24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2\" (UID: \"8f9cb38e-7537-41dc-85cf-7fa55dc65a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2"
Jul 16 21:17:44.295956 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:44.295934    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f9cb38e-7537-41dc-85cf-7fa55dc65a24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2\" (UID: \"8f9cb38e-7537-41dc-85cf-7fa55dc65a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2"
Jul 16 21:17:44.378447 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:44.378413    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2"
Jul 16 21:17:44.520045 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:44.520005    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2"]
Jul 16 21:17:44.525006 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:17:44.524978    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9cb38e_7537_41dc_85cf_7fa55dc65a24.slice/crio-487b68cdf9d7e8d625f8cee0e3e0f2ce2997dd26cfa5543d9a2e03707737b7e5 WatchSource:0}: Error finding container 487b68cdf9d7e8d625f8cee0e3e0f2ce2997dd26cfa5543d9a2e03707737b7e5: Status 404 returned error can't find the container with id 487b68cdf9d7e8d625f8cee0e3e0f2ce2997dd26cfa5543d9a2e03707737b7e5
Jul 16 21:17:44.613966 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:44.613924    2530 generic.go:356] "Generic (PLEG): container finished" podID="d193f689-91c1-4484-b5d8-dbbf3adae841" containerID="a86c074e58033229e6c934a966ddd7bfe127a31e04ab3603436232aa136d39be" exitCode=6
Jul 16 21:17:44.614457 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:44.614009    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc" event={"ID":"d193f689-91c1-4484-b5d8-dbbf3adae841","Type":"ContainerDied","Data":"a86c074e58033229e6c934a966ddd7bfe127a31e04ab3603436232aa136d39be"}
Jul 16 21:17:44.614457 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:44.614068    2530 scope.go:117] "RemoveContainer" containerID="3fe5d4b26172dc73378c762c981b0c58b71ebe33f5647b6fb42284fa8b2bcc8a"
Jul 16 21:17:44.614457 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:44.614294    2530 scope.go:117] "RemoveContainer" containerID="a86c074e58033229e6c934a966ddd7bfe127a31e04ab3603436232aa136d39be"
Jul 16 21:17:44.614591 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:17:44.614544    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc_odh-ai-gateway-infra(d193f689-91c1-4484-b5d8-dbbf3adae841)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc" podUID="d193f689-91c1-4484-b5d8-dbbf3adae841"
Jul 16 21:17:44.615472 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:44.615350    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2" event={"ID":"8f9cb38e-7537-41dc-85cf-7fa55dc65a24","Type":"ContainerStarted","Data":"487b68cdf9d7e8d625f8cee0e3e0f2ce2997dd26cfa5543d9a2e03707737b7e5"}
Jul 16 21:17:45.622332 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:45.622223    2530 generic.go:356] "Generic (PLEG): container finished" podID="8f9cb38e-7537-41dc-85cf-7fa55dc65a24" containerID="9d75480cf161989f2a3e26e028753208455ee6b913391d979a918304b6064dd3" exitCode=6
Jul 16 21:17:45.622332 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:45.622312    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2" event={"ID":"8f9cb38e-7537-41dc-85cf-7fa55dc65a24","Type":"ContainerDied","Data":"9d75480cf161989f2a3e26e028753208455ee6b913391d979a918304b6064dd3"}
Jul 16 21:17:45.622847 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:45.622514    2530 scope.go:117] "RemoveContainer" containerID="9d75480cf161989f2a3e26e028753208455ee6b913391d979a918304b6064dd3"
Jul 16 21:17:45.624274 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:45.624258    2530 scope.go:117] "RemoveContainer" containerID="a86c074e58033229e6c934a966ddd7bfe127a31e04ab3603436232aa136d39be"
Jul 16 21:17:45.624445 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:17:45.624427    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc_odh-ai-gateway-infra(d193f689-91c1-4484-b5d8-dbbf3adae841)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc" podUID="d193f689-91c1-4484-b5d8-dbbf3adae841"
Jul 16 21:17:46.631585 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:46.631476    2530 generic.go:356] "Generic (PLEG): container finished" podID="8f9cb38e-7537-41dc-85cf-7fa55dc65a24" containerID="d6a5d5226a3824684d5dcb3942933b20bc53a29581a5b4dd528940f657624580" exitCode=6
Jul 16 21:17:46.631585 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:46.631568    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2" event={"ID":"8f9cb38e-7537-41dc-85cf-7fa55dc65a24","Type":"ContainerDied","Data":"d6a5d5226a3824684d5dcb3942933b20bc53a29581a5b4dd528940f657624580"}
Jul 16 21:17:46.632130 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:46.631626    2530 scope.go:117] "RemoveContainer" containerID="9d75480cf161989f2a3e26e028753208455ee6b913391d979a918304b6064dd3"
Jul 16 21:17:46.632130 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:46.631847    2530 scope.go:117] "RemoveContainer" containerID="d6a5d5226a3824684d5dcb3942933b20bc53a29581a5b4dd528940f657624580"
Jul 16 21:17:46.632238 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:17:46.632141    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2_odh-ai-gateway-infra(8f9cb38e-7537-41dc-85cf-7fa55dc65a24)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2" podUID="8f9cb38e-7537-41dc-85cf-7fa55dc65a24"
Jul 16 21:17:47.502373 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:47.502341    2530 scope.go:117] "RemoveContainer" containerID="6cf3e87de4be2b1ce5321d7c1620fb920ca5f8be9738f42129935339ff6cfc6d"
Jul 16 21:17:47.638027 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:47.638000    2530 scope.go:117] "RemoveContainer" containerID="d6a5d5226a3824684d5dcb3942933b20bc53a29581a5b4dd528940f657624580"
Jul 16 21:17:47.638460 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:17:47.638197    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2_odh-ai-gateway-infra(8f9cb38e-7537-41dc-85cf-7fa55dc65a24)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2" podUID="8f9cb38e-7537-41dc-85cf-7fa55dc65a24"
Jul 16 21:17:48.644837 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:48.644797    2530 generic.go:356] "Generic (PLEG): container finished" podID="f47c9445-c770-4dd2-af56-4df63b61be5c" containerID="26d90a52d344c6e1cbffefcf4a648a9d0d7a0a5a7bf813ece9ed64b48c9891ac" exitCode=6
Jul 16 21:17:48.645344 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:48.644859    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z" event={"ID":"f47c9445-c770-4dd2-af56-4df63b61be5c","Type":"ContainerDied","Data":"26d90a52d344c6e1cbffefcf4a648a9d0d7a0a5a7bf813ece9ed64b48c9891ac"}
Jul 16 21:17:48.645344 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:48.644914    2530 scope.go:117] "RemoveContainer" containerID="6cf3e87de4be2b1ce5321d7c1620fb920ca5f8be9738f42129935339ff6cfc6d"
Jul 16 21:17:48.645344 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:48.645251    2530 scope.go:117] "RemoveContainer" containerID="26d90a52d344c6e1cbffefcf4a648a9d0d7a0a5a7bf813ece9ed64b48c9891ac"
Jul 16 21:17:48.645511 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:17:48.645484    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z_odh-ai-gateway-infra(f47c9445-c770-4dd2-af56-4df63b61be5c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z" podUID="f47c9445-c770-4dd2-af56-4df63b61be5c"
Jul 16 21:17:49.673252 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:49.673215    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z"]
Jul 16 21:17:49.815464 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:49.815427    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z"
Jul 16 21:17:49.940238 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:49.940138    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47c9445-c770-4dd2-af56-4df63b61be5c-maas-api-service-ca\") pod \"f47c9445-c770-4dd2-af56-4df63b61be5c\" (UID: \"f47c9445-c770-4dd2-af56-4df63b61be5c\") "
Jul 16 21:17:49.940517 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:49.940491    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47c9445-c770-4dd2-af56-4df63b61be5c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f47c9445-c770-4dd2-af56-4df63b61be5c" (UID: "f47c9445-c770-4dd2-af56-4df63b61be5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:17:50.041513 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:50.041471    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f47c9445-c770-4dd2-af56-4df63b61be5c-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:17:50.659500 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:50.659456    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z" event={"ID":"f47c9445-c770-4dd2-af56-4df63b61be5c","Type":"ContainerDied","Data":"965f035b811717c9f151f7bef0859c2989e7a856466c560051d4b8e86610842c"}
Jul 16 21:17:50.659500 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:50.659482    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z"
Jul 16 21:17:50.659762 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:50.659536    2530 scope.go:117] "RemoveContainer" containerID="26d90a52d344c6e1cbffefcf4a648a9d0d7a0a5a7bf813ece9ed64b48c9891ac"
Jul 16 21:17:50.676305 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:50.676268    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z"]
Jul 16 21:17:50.686278 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:50.686246    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-sl65z"]
Jul 16 21:17:52.507996 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:52.507963    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f47c9445-c770-4dd2-af56-4df63b61be5c" path="/var/lib/kubelet/pods/f47c9445-c770-4dd2-af56-4df63b61be5c/volumes"
Jul 16 21:17:56.507661 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:56.507631    2530 scope.go:117] "RemoveContainer" containerID="a86c074e58033229e6c934a966ddd7bfe127a31e04ab3603436232aa136d39be"
Jul 16 21:17:57.693666 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:57.693572    2530 generic.go:356] "Generic (PLEG): container finished" podID="d193f689-91c1-4484-b5d8-dbbf3adae841" containerID="6dff947ced287e83fdbe94600ff789b00c5c10d62aee35aa076ef1855daa839a" exitCode=6
Jul 16 21:17:57.693666 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:57.693636    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc" event={"ID":"d193f689-91c1-4484-b5d8-dbbf3adae841","Type":"ContainerDied","Data":"6dff947ced287e83fdbe94600ff789b00c5c10d62aee35aa076ef1855daa839a"}
Jul 16 21:17:57.694251 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:57.693699    2530 scope.go:117] "RemoveContainer" containerID="a86c074e58033229e6c934a966ddd7bfe127a31e04ab3603436232aa136d39be"
Jul 16 21:17:57.694251 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:57.694056    2530 scope.go:117] "RemoveContainer" containerID="6dff947ced287e83fdbe94600ff789b00c5c10d62aee35aa076ef1855daa839a"
Jul 16 21:17:57.694357 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:17:57.694328    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc_odh-ai-gateway-infra(d193f689-91c1-4484-b5d8-dbbf3adae841)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc" podUID="d193f689-91c1-4484-b5d8-dbbf3adae841"
Jul 16 21:17:58.721790 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:58.721747    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc"]
Jul 16 21:17:58.862715 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:58.862688    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc"
Jul 16 21:17:58.916976 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:58.916933    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d193f689-91c1-4484-b5d8-dbbf3adae841-maas-api-service-ca\") pod \"d193f689-91c1-4484-b5d8-dbbf3adae841\" (UID: \"d193f689-91c1-4484-b5d8-dbbf3adae841\") "
Jul 16 21:17:58.917351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:58.917319    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d193f689-91c1-4484-b5d8-dbbf3adae841-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d193f689-91c1-4484-b5d8-dbbf3adae841" (UID: "d193f689-91c1-4484-b5d8-dbbf3adae841"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:17:59.018209 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:59.018121    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d193f689-91c1-4484-b5d8-dbbf3adae841-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:17:59.706007 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:59.705969    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc" event={"ID":"d193f689-91c1-4484-b5d8-dbbf3adae841","Type":"ContainerDied","Data":"275133bff7e242d762ff09debb6afa9ce9f5286c2eba1877cbbb185f891695e7"}
Jul 16 21:17:59.706246 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:59.706031    2530 scope.go:117] "RemoveContainer" containerID="6dff947ced287e83fdbe94600ff789b00c5c10d62aee35aa076ef1855daa839a"
Jul 16 21:17:59.706246 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:59.705978    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc"
Jul 16 21:17:59.727257 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:59.727223    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc"]
Jul 16 21:17:59.733647 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:17:59.733614    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-76qjc"]
Jul 16 21:18:00.507574 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:00.507538    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d193f689-91c1-4484-b5d8-dbbf3adae841" path="/var/lib/kubelet/pods/d193f689-91c1-4484-b5d8-dbbf3adae841/volumes"
Jul 16 21:18:02.504457 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:02.504411    2530 scope.go:117] "RemoveContainer" containerID="d6a5d5226a3824684d5dcb3942933b20bc53a29581a5b4dd528940f657624580"
Jul 16 21:18:03.727209 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:03.727171    2530 generic.go:356] "Generic (PLEG): container finished" podID="8f9cb38e-7537-41dc-85cf-7fa55dc65a24" containerID="9a74e43d31e735d06ccfba60efbe8291b2d54c2f7292e146982d97dba593f019" exitCode=6
Jul 16 21:18:03.727686 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:03.727224    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2" event={"ID":"8f9cb38e-7537-41dc-85cf-7fa55dc65a24","Type":"ContainerDied","Data":"9a74e43d31e735d06ccfba60efbe8291b2d54c2f7292e146982d97dba593f019"}
Jul 16 21:18:03.727686 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:03.727278    2530 scope.go:117] "RemoveContainer" containerID="d6a5d5226a3824684d5dcb3942933b20bc53a29581a5b4dd528940f657624580"
Jul 16 21:18:03.727686 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:03.727592    2530 scope.go:117] "RemoveContainer" containerID="9a74e43d31e735d06ccfba60efbe8291b2d54c2f7292e146982d97dba593f019"
Jul 16 21:18:03.727815 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:18:03.727785    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2_odh-ai-gateway-infra(8f9cb38e-7537-41dc-85cf-7fa55dc65a24)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2" podUID="8f9cb38e-7537-41dc-85cf-7fa55dc65a24"
Jul 16 21:18:04.765304 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:04.765253    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2"]
Jul 16 21:18:04.903588 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:04.903551    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2"
Jul 16 21:18:04.965581 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:04.965539    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f9cb38e-7537-41dc-85cf-7fa55dc65a24-maas-api-service-ca\") pod \"8f9cb38e-7537-41dc-85cf-7fa55dc65a24\" (UID: \"8f9cb38e-7537-41dc-85cf-7fa55dc65a24\") "
Jul 16 21:18:04.965909 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:04.965885    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9cb38e-7537-41dc-85cf-7fa55dc65a24-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f9cb38e-7537-41dc-85cf-7fa55dc65a24" (UID: "8f9cb38e-7537-41dc-85cf-7fa55dc65a24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:18:05.066923 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:05.066835    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f9cb38e-7537-41dc-85cf-7fa55dc65a24-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:18:05.739152 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:05.739120    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2"
Jul 16 21:18:05.739152 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:05.739132    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2" event={"ID":"8f9cb38e-7537-41dc-85cf-7fa55dc65a24","Type":"ContainerDied","Data":"487b68cdf9d7e8d625f8cee0e3e0f2ce2997dd26cfa5543d9a2e03707737b7e5"}
Jul 16 21:18:05.739399 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:05.739203    2530 scope.go:117] "RemoveContainer" containerID="9a74e43d31e735d06ccfba60efbe8291b2d54c2f7292e146982d97dba593f019"
Jul 16 21:18:05.766404 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:05.766359    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2"]
Jul 16 21:18:05.770542 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:05.770507    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-pbgn2"]
Jul 16 21:18:06.507374 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:06.507340    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f9cb38e-7537-41dc-85cf-7fa55dc65a24" path="/var/lib/kubelet/pods/8f9cb38e-7537-41dc-85cf-7fa55dc65a24/volumes"
Jul 16 21:18:29.355814 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.355763    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b"]
Jul 16 21:18:29.356256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356131    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9cb38e-7537-41dc-85cf-7fa55dc65a24" containerName="revoke-keys"
Jul 16 21:18:29.356256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356144    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9cb38e-7537-41dc-85cf-7fa55dc65a24" containerName="revoke-keys"
Jul 16 21:18:29.356256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356152    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47c9445-c770-4dd2-af56-4df63b61be5c" containerName="revoke-keys"
Jul 16 21:18:29.356256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356157    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47c9445-c770-4dd2-af56-4df63b61be5c" containerName="revoke-keys"
Jul 16 21:18:29.356256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356166    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9cb38e-7537-41dc-85cf-7fa55dc65a24" containerName="revoke-keys"
Jul 16 21:18:29.356256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356172    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9cb38e-7537-41dc-85cf-7fa55dc65a24" containerName="revoke-keys"
Jul 16 21:18:29.356256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356179    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d193f689-91c1-4484-b5d8-dbbf3adae841" containerName="revoke-keys"
Jul 16 21:18:29.356256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356189    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="d193f689-91c1-4484-b5d8-dbbf3adae841" containerName="revoke-keys"
Jul 16 21:18:29.356256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356199    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d193f689-91c1-4484-b5d8-dbbf3adae841" containerName="revoke-keys"
Jul 16 21:18:29.356256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356204    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="d193f689-91c1-4484-b5d8-dbbf3adae841" containerName="revoke-keys"
Jul 16 21:18:29.356256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356221    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47c9445-c770-4dd2-af56-4df63b61be5c" containerName="revoke-keys"
Jul 16 21:18:29.356256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356226    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47c9445-c770-4dd2-af56-4df63b61be5c" containerName="revoke-keys"
Jul 16 21:18:29.356678 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356279    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="d193f689-91c1-4484-b5d8-dbbf3adae841" containerName="revoke-keys"
Jul 16 21:18:29.356678 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356286    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f9cb38e-7537-41dc-85cf-7fa55dc65a24" containerName="revoke-keys"
Jul 16 21:18:29.356678 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356293    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f47c9445-c770-4dd2-af56-4df63b61be5c" containerName="revoke-keys"
Jul 16 21:18:29.356678 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356299    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="d193f689-91c1-4484-b5d8-dbbf3adae841" containerName="revoke-keys"
Jul 16 21:18:29.356678 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356306    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f9cb38e-7537-41dc-85cf-7fa55dc65a24" containerName="revoke-keys"
Jul 16 21:18:29.356678 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356311    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f9cb38e-7537-41dc-85cf-7fa55dc65a24" containerName="revoke-keys"
Jul 16 21:18:29.356678 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356318    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f47c9445-c770-4dd2-af56-4df63b61be5c" containerName="revoke-keys"
Jul 16 21:18:29.356678 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.356323    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f47c9445-c770-4dd2-af56-4df63b61be5c" containerName="revoke-keys"
Jul 16 21:18:29.359303 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.359286    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b"
Jul 16 21:18:29.361690 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.361664    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:18:29.369571 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.369536    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b"]
Jul 16 21:18:29.483023 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.482976    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7d88e23-8037-42cc-aad6-781fa05b1427-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b\" (UID: \"e7d88e23-8037-42cc-aad6-781fa05b1427\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b"
Jul 16 21:18:29.584007 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.583966    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7d88e23-8037-42cc-aad6-781fa05b1427-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b\" (UID: \"e7d88e23-8037-42cc-aad6-781fa05b1427\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b"
Jul 16 21:18:29.584601 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.584570    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7d88e23-8037-42cc-aad6-781fa05b1427-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b\" (UID: \"e7d88e23-8037-42cc-aad6-781fa05b1427\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b"
Jul 16 21:18:29.670550 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.670460    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b"
Jul 16 21:18:29.814055 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.813945    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b"]
Jul 16 21:18:29.816845 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:18:29.816816    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d88e23_8037_42cc_aad6_781fa05b1427.slice/crio-9a1f8c6417edae5489ec8162afe59da3fefff7ee28427aafc3d32ae853b64dbe WatchSource:0}: Error finding container 9a1f8c6417edae5489ec8162afe59da3fefff7ee28427aafc3d32ae853b64dbe: Status 404 returned error can't find the container with id 9a1f8c6417edae5489ec8162afe59da3fefff7ee28427aafc3d32ae853b64dbe
Jul 16 21:18:29.850431 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:29.850394    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b" event={"ID":"e7d88e23-8037-42cc-aad6-781fa05b1427","Type":"ContainerStarted","Data":"9a1f8c6417edae5489ec8162afe59da3fefff7ee28427aafc3d32ae853b64dbe"}
Jul 16 21:18:30.856270 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:30.856229    2530 generic.go:356] "Generic (PLEG): container finished" podID="e7d88e23-8037-42cc-aad6-781fa05b1427" containerID="d2f606efa8a48994753ac5bc40a45879b20bc1d05ffa838d9a56c853c931a563" exitCode=6
Jul 16 21:18:30.856761 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:30.856321    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b" event={"ID":"e7d88e23-8037-42cc-aad6-781fa05b1427","Type":"ContainerDied","Data":"d2f606efa8a48994753ac5bc40a45879b20bc1d05ffa838d9a56c853c931a563"}
Jul 16 21:18:30.856761 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:30.856576    2530 scope.go:117] "RemoveContainer" containerID="d2f606efa8a48994753ac5bc40a45879b20bc1d05ffa838d9a56c853c931a563"
Jul 16 21:18:31.862310 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:31.862277    2530 generic.go:356] "Generic (PLEG): container finished" podID="e7d88e23-8037-42cc-aad6-781fa05b1427" containerID="a86921ef0133c57a2e37cfe1d22c91e97d96a55eb3bfc65284b71d3a05948f31" exitCode=6
Jul 16 21:18:31.862810 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:31.862361    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b" event={"ID":"e7d88e23-8037-42cc-aad6-781fa05b1427","Type":"ContainerDied","Data":"a86921ef0133c57a2e37cfe1d22c91e97d96a55eb3bfc65284b71d3a05948f31"}
Jul 16 21:18:31.862810 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:31.862433    2530 scope.go:117] "RemoveContainer" containerID="d2f606efa8a48994753ac5bc40a45879b20bc1d05ffa838d9a56c853c931a563"
Jul 16 21:18:31.862810 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:31.862595    2530 scope.go:117] "RemoveContainer" containerID="a86921ef0133c57a2e37cfe1d22c91e97d96a55eb3bfc65284b71d3a05948f31"
Jul 16 21:18:31.862959 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:18:31.862820    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b_odh-ai-gateway-infra(e7d88e23-8037-42cc-aad6-781fa05b1427)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b" podUID="e7d88e23-8037-42cc-aad6-781fa05b1427"
Jul 16 21:18:31.917259 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:31.917223    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq"]
Jul 16 21:18:31.917580 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:31.917568    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47c9445-c770-4dd2-af56-4df63b61be5c" containerName="revoke-keys"
Jul 16 21:18:31.917627 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:31.917582    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47c9445-c770-4dd2-af56-4df63b61be5c" containerName="revoke-keys"
Jul 16 21:18:31.917627 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:31.917592    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9cb38e-7537-41dc-85cf-7fa55dc65a24" containerName="revoke-keys"
Jul 16 21:18:31.917627 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:31.917597    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9cb38e-7537-41dc-85cf-7fa55dc65a24" containerName="revoke-keys"
Jul 16 21:18:31.917627 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:31.917606    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d193f689-91c1-4484-b5d8-dbbf3adae841" containerName="revoke-keys"
Jul 16 21:18:31.917627 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:31.917611    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="d193f689-91c1-4484-b5d8-dbbf3adae841" containerName="revoke-keys"
Jul 16 21:18:31.917795 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:31.917666    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="d193f689-91c1-4484-b5d8-dbbf3adae841" containerName="revoke-keys"
Jul 16 21:18:31.921613 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:31.921594    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq"
Jul 16 21:18:31.930067 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:31.930035    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq"]
Jul 16 21:18:32.005602 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:32.005554    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f801ef-7be1-4495-b286-5d3a77af31b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq\" (UID: \"b2f801ef-7be1-4495-b286-5d3a77af31b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq"
Jul 16 21:18:32.106496 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:32.106458    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f801ef-7be1-4495-b286-5d3a77af31b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq\" (UID: \"b2f801ef-7be1-4495-b286-5d3a77af31b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq"
Jul 16 21:18:32.107139 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:32.107085    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f801ef-7be1-4495-b286-5d3a77af31b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq\" (UID: \"b2f801ef-7be1-4495-b286-5d3a77af31b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq"
Jul 16 21:18:32.233720 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:32.233676    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq"
Jul 16 21:18:32.371527 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:32.371500    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq"]
Jul 16 21:18:32.374010 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:18:32.373976    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f801ef_7be1_4495_b286_5d3a77af31b1.slice/crio-0202b8622dcb87e95f3d9d3ceadc6c21b4caca0a17abe7e87126119a3fe76622 WatchSource:0}: Error finding container 0202b8622dcb87e95f3d9d3ceadc6c21b4caca0a17abe7e87126119a3fe76622: Status 404 returned error can't find the container with id 0202b8622dcb87e95f3d9d3ceadc6c21b4caca0a17abe7e87126119a3fe76622
Jul 16 21:18:32.868178 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:32.868048    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq" event={"ID":"b2f801ef-7be1-4495-b286-5d3a77af31b1","Type":"ContainerStarted","Data":"ea33245b8ad1fb7d8d9de91bcf12c617a142675fd723479847477b608c87c6db"}
Jul 16 21:18:32.868178 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:32.868146    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq" event={"ID":"b2f801ef-7be1-4495-b286-5d3a77af31b1","Type":"ContainerStarted","Data":"0202b8622dcb87e95f3d9d3ceadc6c21b4caca0a17abe7e87126119a3fe76622"}
Jul 16 21:18:32.869884 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:32.869865    2530 scope.go:117] "RemoveContainer" containerID="a86921ef0133c57a2e37cfe1d22c91e97d96a55eb3bfc65284b71d3a05948f31"
Jul 16 21:18:32.870041 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:18:32.870024    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b_odh-ai-gateway-infra(e7d88e23-8037-42cc-aad6-781fa05b1427)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b" podUID="e7d88e23-8037-42cc-aad6-781fa05b1427"
Jul 16 21:18:32.884696 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:32.884645    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq" podStartSLOduration=1.884626943 podStartE2EDuration="1.884626943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:18:32.883594799 +0000 UTC m=+2750.905182568" watchObservedRunningTime="2026-07-16 21:18:32.884626943 +0000 UTC m=+2750.906214710"
Jul 16 21:18:33.875634 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:33.875593    2530 generic.go:356] "Generic (PLEG): container finished" podID="b2f801ef-7be1-4495-b286-5d3a77af31b1" containerID="ea33245b8ad1fb7d8d9de91bcf12c617a142675fd723479847477b608c87c6db" exitCode=6
Jul 16 21:18:33.876048 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:33.875648    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq" event={"ID":"b2f801ef-7be1-4495-b286-5d3a77af31b1","Type":"ContainerDied","Data":"ea33245b8ad1fb7d8d9de91bcf12c617a142675fd723479847477b608c87c6db"}
Jul 16 21:18:33.876048 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:33.875913    2530 scope.go:117] "RemoveContainer" containerID="ea33245b8ad1fb7d8d9de91bcf12c617a142675fd723479847477b608c87c6db"
Jul 16 21:18:34.882233 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:34.882195    2530 generic.go:356] "Generic (PLEG): container finished" podID="b2f801ef-7be1-4495-b286-5d3a77af31b1" containerID="0c7100b51b561358f5c138bc90ee458182b5285ee4b44508c11bb0f906204cc0" exitCode=6
Jul 16 21:18:34.882714 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:34.882271    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq" event={"ID":"b2f801ef-7be1-4495-b286-5d3a77af31b1","Type":"ContainerDied","Data":"0c7100b51b561358f5c138bc90ee458182b5285ee4b44508c11bb0f906204cc0"}
Jul 16 21:18:34.882714 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:34.882331    2530 scope.go:117] "RemoveContainer" containerID="ea33245b8ad1fb7d8d9de91bcf12c617a142675fd723479847477b608c87c6db"
Jul 16 21:18:34.882714 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:34.882536    2530 scope.go:117] "RemoveContainer" containerID="0c7100b51b561358f5c138bc90ee458182b5285ee4b44508c11bb0f906204cc0"
Jul 16 21:18:34.882864 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:18:34.882746    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq_odh-ai-gateway-infra(b2f801ef-7be1-4495-b286-5d3a77af31b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq" podUID="b2f801ef-7be1-4495-b286-5d3a77af31b1"
Jul 16 21:18:35.888176 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:35.888147    2530 scope.go:117] "RemoveContainer" containerID="0c7100b51b561358f5c138bc90ee458182b5285ee4b44508c11bb0f906204cc0"
Jul 16 21:18:35.888560 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:18:35.888344    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq_odh-ai-gateway-infra(b2f801ef-7be1-4495-b286-5d3a77af31b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq" podUID="b2f801ef-7be1-4495-b286-5d3a77af31b1"
Jul 16 21:18:44.117780 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:44.117740    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2"]
Jul 16 21:18:44.122899 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:44.122872    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2"
Jul 16 21:18:44.130406 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:44.129905    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2"]
Jul 16 21:18:44.215251 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:44.215207    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30638c0c-37ad-4576-a2ff-478fb62597c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2\" (UID: \"30638c0c-37ad-4576-a2ff-478fb62597c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2"
Jul 16 21:18:44.316698 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:44.316641    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30638c0c-37ad-4576-a2ff-478fb62597c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2\" (UID: \"30638c0c-37ad-4576-a2ff-478fb62597c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2"
Jul 16 21:18:44.317342 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:44.317321    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30638c0c-37ad-4576-a2ff-478fb62597c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2\" (UID: \"30638c0c-37ad-4576-a2ff-478fb62597c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2"
Jul 16 21:18:44.434990 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:44.434897    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2"
Jul 16 21:18:44.575400 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:44.575356    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2"]
Jul 16 21:18:44.579006 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:18:44.578977    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30638c0c_37ad_4576_a2ff_478fb62597c4.slice/crio-ab38d69039f45cdf806fff76ff2da3557a6abc85a241957d640b5e8bea94a263 WatchSource:0}: Error finding container ab38d69039f45cdf806fff76ff2da3557a6abc85a241957d640b5e8bea94a263: Status 404 returned error can't find the container with id ab38d69039f45cdf806fff76ff2da3557a6abc85a241957d640b5e8bea94a263
Jul 16 21:18:44.931884 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:44.931848    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2" event={"ID":"30638c0c-37ad-4576-a2ff-478fb62597c4","Type":"ContainerStarted","Data":"5f55b81343833dba672f53d57370710e3af24289551de07960520f8984e04c01"}
Jul 16 21:18:44.932078 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:44.931904    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2" event={"ID":"30638c0c-37ad-4576-a2ff-478fb62597c4","Type":"ContainerStarted","Data":"ab38d69039f45cdf806fff76ff2da3557a6abc85a241957d640b5e8bea94a263"}
Jul 16 21:18:44.948121 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:44.948045    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2" podStartSLOduration=0.948026864 podStartE2EDuration="948.026864ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:18:44.946764862 +0000 UTC m=+2762.968352629" watchObservedRunningTime="2026-07-16 21:18:44.948026864 +0000 UTC m=+2762.969614633"
Jul 16 21:18:45.938435 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:45.938394    2530 generic.go:356] "Generic (PLEG): container finished" podID="30638c0c-37ad-4576-a2ff-478fb62597c4" containerID="5f55b81343833dba672f53d57370710e3af24289551de07960520f8984e04c01" exitCode=6
Jul 16 21:18:45.938948 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:45.938484    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2" event={"ID":"30638c0c-37ad-4576-a2ff-478fb62597c4","Type":"ContainerDied","Data":"5f55b81343833dba672f53d57370710e3af24289551de07960520f8984e04c01"}
Jul 16 21:18:45.938948 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:45.938732    2530 scope.go:117] "RemoveContainer" containerID="5f55b81343833dba672f53d57370710e3af24289551de07960520f8984e04c01"
Jul 16 21:18:46.501973 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:46.501879    2530 scope.go:117] "RemoveContainer" containerID="0c7100b51b561358f5c138bc90ee458182b5285ee4b44508c11bb0f906204cc0"
Jul 16 21:18:46.945926 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:46.945892    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq" event={"ID":"b2f801ef-7be1-4495-b286-5d3a77af31b1","Type":"ContainerStarted","Data":"8612bea013aaa17f8e99693c76a94ba5e718e0be4d4811fa6db62122bf1a6e50"}
Jul 16 21:18:46.947877 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:46.947851    2530 generic.go:356] "Generic (PLEG): container finished" podID="30638c0c-37ad-4576-a2ff-478fb62597c4" containerID="ffb33979edb636904e93e83c0d12ad87fe5558d8244c1bb2111df80ba72063a9" exitCode=6
Jul 16 21:18:46.948006 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:46.947921    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2" event={"ID":"30638c0c-37ad-4576-a2ff-478fb62597c4","Type":"ContainerDied","Data":"ffb33979edb636904e93e83c0d12ad87fe5558d8244c1bb2111df80ba72063a9"}
Jul 16 21:18:46.948006 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:46.947968    2530 scope.go:117] "RemoveContainer" containerID="5f55b81343833dba672f53d57370710e3af24289551de07960520f8984e04c01"
Jul 16 21:18:46.948204 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:46.948182    2530 scope.go:117] "RemoveContainer" containerID="ffb33979edb636904e93e83c0d12ad87fe5558d8244c1bb2111df80ba72063a9"
Jul 16 21:18:46.948445 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:18:46.948421    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2_odh-ai-gateway-infra(30638c0c-37ad-4576-a2ff-478fb62597c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2" podUID="30638c0c-37ad-4576-a2ff-478fb62597c4"
Jul 16 21:18:47.502202 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:47.502087    2530 scope.go:117] "RemoveContainer" containerID="a86921ef0133c57a2e37cfe1d22c91e97d96a55eb3bfc65284b71d3a05948f31"
Jul 16 21:18:47.954345 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:47.954310    2530 generic.go:356] "Generic (PLEG): container finished" podID="b2f801ef-7be1-4495-b286-5d3a77af31b1" containerID="8612bea013aaa17f8e99693c76a94ba5e718e0be4d4811fa6db62122bf1a6e50" exitCode=6
Jul 16 21:18:47.954779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:47.954386    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq" event={"ID":"b2f801ef-7be1-4495-b286-5d3a77af31b1","Type":"ContainerDied","Data":"8612bea013aaa17f8e99693c76a94ba5e718e0be4d4811fa6db62122bf1a6e50"}
Jul 16 21:18:47.954779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:47.954456    2530 scope.go:117] "RemoveContainer" containerID="0c7100b51b561358f5c138bc90ee458182b5285ee4b44508c11bb0f906204cc0"
Jul 16 21:18:47.954889 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:47.954821    2530 scope.go:117] "RemoveContainer" containerID="8612bea013aaa17f8e99693c76a94ba5e718e0be4d4811fa6db62122bf1a6e50"
Jul 16 21:18:47.955071 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:18:47.955044    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq_odh-ai-gateway-infra(b2f801ef-7be1-4495-b286-5d3a77af31b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq" podUID="b2f801ef-7be1-4495-b286-5d3a77af31b1"
Jul 16 21:18:47.956226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:47.956207    2530 scope.go:117] "RemoveContainer" containerID="ffb33979edb636904e93e83c0d12ad87fe5558d8244c1bb2111df80ba72063a9"
Jul 16 21:18:47.956444 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:18:47.956415    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2_odh-ai-gateway-infra(30638c0c-37ad-4576-a2ff-478fb62597c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2" podUID="30638c0c-37ad-4576-a2ff-478fb62597c4"
Jul 16 21:18:47.957945 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:47.957926    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b" event={"ID":"e7d88e23-8037-42cc-aad6-781fa05b1427","Type":"ContainerStarted","Data":"7ee23b325c94fae52f9bf9ad13ef26bd4d0c02b213a9cf3c38e5743e1913f6e4"}
Jul 16 21:18:47.975374 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:47.975339    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq"]
Jul 16 21:18:48.013381 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:48.010474    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b" podStartSLOduration=19.010453002 podStartE2EDuration="19.010453002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:18:48.007742966 +0000 UTC m=+2766.029330732" watchObservedRunningTime="2026-07-16 21:18:48.010453002 +0000 UTC m=+2766.032040769"
Jul 16 21:18:48.965812 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:48.965775    2530 generic.go:356] "Generic (PLEG): container finished" podID="e7d88e23-8037-42cc-aad6-781fa05b1427" containerID="7ee23b325c94fae52f9bf9ad13ef26bd4d0c02b213a9cf3c38e5743e1913f6e4" exitCode=6
Jul 16 21:18:48.966161 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:48.965828    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b" event={"ID":"e7d88e23-8037-42cc-aad6-781fa05b1427","Type":"ContainerDied","Data":"7ee23b325c94fae52f9bf9ad13ef26bd4d0c02b213a9cf3c38e5743e1913f6e4"}
Jul 16 21:18:48.966161 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:48.965895    2530 scope.go:117] "RemoveContainer" containerID="a86921ef0133c57a2e37cfe1d22c91e97d96a55eb3bfc65284b71d3a05948f31"
Jul 16 21:18:48.966264 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:48.966248    2530 scope.go:117] "RemoveContainer" containerID="7ee23b325c94fae52f9bf9ad13ef26bd4d0c02b213a9cf3c38e5743e1913f6e4"
Jul 16 21:18:48.966464 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:18:48.966444    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b_odh-ai-gateway-infra(e7d88e23-8037-42cc-aad6-781fa05b1427)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b" podUID="e7d88e23-8037-42cc-aad6-781fa05b1427"
Jul 16 21:18:49.017396 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:49.017359    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b"]
Jul 16 21:18:49.118979 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:49.118947    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq"
Jul 16 21:18:49.260021 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:49.259927    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f801ef-7be1-4495-b286-5d3a77af31b1-maas-api-service-ca\") pod \"b2f801ef-7be1-4495-b286-5d3a77af31b1\" (UID: \"b2f801ef-7be1-4495-b286-5d3a77af31b1\") "
Jul 16 21:18:49.260313 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:49.260292    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f801ef-7be1-4495-b286-5d3a77af31b1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b2f801ef-7be1-4495-b286-5d3a77af31b1" (UID: "b2f801ef-7be1-4495-b286-5d3a77af31b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:18:49.361607 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:49.361567    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f801ef-7be1-4495-b286-5d3a77af31b1-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:18:49.972858 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:49.972829    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq"
Jul 16 21:18:49.973162 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:49.972822    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq" event={"ID":"b2f801ef-7be1-4495-b286-5d3a77af31b1","Type":"ContainerDied","Data":"0202b8622dcb87e95f3d9d3ceadc6c21b4caca0a17abe7e87126119a3fe76622"}
Jul 16 21:18:49.973162 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:49.972965    2530 scope.go:117] "RemoveContainer" containerID="8612bea013aaa17f8e99693c76a94ba5e718e0be4d4811fa6db62122bf1a6e50"
Jul 16 21:18:50.001251 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:50.001211    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq"]
Jul 16 21:18:50.010536 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:50.010491    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gdlbq"]
Jul 16 21:18:50.117873 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:50.117840    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b"
Jul 16 21:18:50.167486 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:50.167450    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7d88e23-8037-42cc-aad6-781fa05b1427-maas-api-service-ca\") pod \"e7d88e23-8037-42cc-aad6-781fa05b1427\" (UID: \"e7d88e23-8037-42cc-aad6-781fa05b1427\") "
Jul 16 21:18:50.167856 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:50.167837    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d88e23-8037-42cc-aad6-781fa05b1427-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e7d88e23-8037-42cc-aad6-781fa05b1427" (UID: "e7d88e23-8037-42cc-aad6-781fa05b1427"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:18:50.268143 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:50.268036    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7d88e23-8037-42cc-aad6-781fa05b1427-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:18:50.507662 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:50.507617    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2f801ef-7be1-4495-b286-5d3a77af31b1" path="/var/lib/kubelet/pods/b2f801ef-7be1-4495-b286-5d3a77af31b1/volumes"
Jul 16 21:18:50.979272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:50.979232    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b" event={"ID":"e7d88e23-8037-42cc-aad6-781fa05b1427","Type":"ContainerDied","Data":"9a1f8c6417edae5489ec8162afe59da3fefff7ee28427aafc3d32ae853b64dbe"}
Jul 16 21:18:50.979678 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:50.979297    2530 scope.go:117] "RemoveContainer" containerID="7ee23b325c94fae52f9bf9ad13ef26bd4d0c02b213a9cf3c38e5743e1913f6e4"
Jul 16 21:18:50.979678 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:50.979249    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b"
Jul 16 21:18:50.995431 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:50.995394    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b"]
Jul 16 21:18:51.003386 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:51.003353    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-m6s8b"]
Jul 16 21:18:52.507866 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:52.507824    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7d88e23-8037-42cc-aad6-781fa05b1427" path="/var/lib/kubelet/pods/e7d88e23-8037-42cc-aad6-781fa05b1427/volumes"
Jul 16 21:18:59.502040 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:18:59.502006    2530 scope.go:117] "RemoveContainer" containerID="ffb33979edb636904e93e83c0d12ad87fe5558d8244c1bb2111df80ba72063a9"
Jul 16 21:19:00.021330 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:00.021297    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2" event={"ID":"30638c0c-37ad-4576-a2ff-478fb62597c4","Type":"ContainerStarted","Data":"b249d4f3cda7c89e3a3bd26de6293a2aac6a1f03858a1dc665b2a114a638bd91"}
Jul 16 21:19:01.027703 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:01.027667    2530 generic.go:356] "Generic (PLEG): container finished" podID="30638c0c-37ad-4576-a2ff-478fb62597c4" containerID="b249d4f3cda7c89e3a3bd26de6293a2aac6a1f03858a1dc665b2a114a638bd91" exitCode=6
Jul 16 21:19:01.028455 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:01.027740    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2" event={"ID":"30638c0c-37ad-4576-a2ff-478fb62597c4","Type":"ContainerDied","Data":"b249d4f3cda7c89e3a3bd26de6293a2aac6a1f03858a1dc665b2a114a638bd91"}
Jul 16 21:19:01.028455 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:01.027800    2530 scope.go:117] "RemoveContainer" containerID="ffb33979edb636904e93e83c0d12ad87fe5558d8244c1bb2111df80ba72063a9"
Jul 16 21:19:01.028455 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:01.028057    2530 scope.go:117] "RemoveContainer" containerID="b249d4f3cda7c89e3a3bd26de6293a2aac6a1f03858a1dc665b2a114a638bd91"
Jul 16 21:19:01.028455 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:19:01.028277    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2_odh-ai-gateway-infra(30638c0c-37ad-4576-a2ff-478fb62597c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2" podUID="30638c0c-37ad-4576-a2ff-478fb62597c4"
Jul 16 21:19:01.058181 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:01.058147    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2"]
Jul 16 21:19:02.171294 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:02.171265    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2"
Jul 16 21:19:02.267287 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:02.267245    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30638c0c-37ad-4576-a2ff-478fb62597c4-maas-api-service-ca\") pod \"30638c0c-37ad-4576-a2ff-478fb62597c4\" (UID: \"30638c0c-37ad-4576-a2ff-478fb62597c4\") "
Jul 16 21:19:02.267606 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:02.267583    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30638c0c-37ad-4576-a2ff-478fb62597c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "30638c0c-37ad-4576-a2ff-478fb62597c4" (UID: "30638c0c-37ad-4576-a2ff-478fb62597c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:19:02.368148 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:02.368028    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30638c0c-37ad-4576-a2ff-478fb62597c4-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:19:03.045504 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:03.045475    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2"
Jul 16 21:19:03.045683 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:03.045510    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2" event={"ID":"30638c0c-37ad-4576-a2ff-478fb62597c4","Type":"ContainerDied","Data":"ab38d69039f45cdf806fff76ff2da3557a6abc85a241957d640b5e8bea94a263"}
Jul 16 21:19:03.045683 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:03.045571    2530 scope.go:117] "RemoveContainer" containerID="b249d4f3cda7c89e3a3bd26de6293a2aac6a1f03858a1dc665b2a114a638bd91"
Jul 16 21:19:03.064527 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:03.064500    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2"]
Jul 16 21:19:03.070090 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:03.070066    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-htfw2"]
Jul 16 21:19:04.507267 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:04.507231    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30638c0c-37ad-4576-a2ff-478fb62597c4" path="/var/lib/kubelet/pods/30638c0c-37ad-4576-a2ff-478fb62597c4/volumes"
Jul 16 21:19:21.803125 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803077    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"]
Jul 16 21:19:21.803580 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803548    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f801ef-7be1-4495-b286-5d3a77af31b1" containerName="revoke-keys"
Jul 16 21:19:21.803580 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803566    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f801ef-7be1-4495-b286-5d3a77af31b1" containerName="revoke-keys"
Jul 16 21:19:21.803710 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803588    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d88e23-8037-42cc-aad6-781fa05b1427" containerName="revoke-keys"
Jul 16 21:19:21.803710 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803597    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d88e23-8037-42cc-aad6-781fa05b1427" containerName="revoke-keys"
Jul 16 21:19:21.803710 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803613    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f801ef-7be1-4495-b286-5d3a77af31b1" containerName="revoke-keys"
Jul 16 21:19:21.803710 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803621    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f801ef-7be1-4495-b286-5d3a77af31b1" containerName="revoke-keys"
Jul 16 21:19:21.803710 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803635    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30638c0c-37ad-4576-a2ff-478fb62597c4" containerName="revoke-keys"
Jul 16 21:19:21.803710 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803644    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="30638c0c-37ad-4576-a2ff-478fb62597c4" containerName="revoke-keys"
Jul 16 21:19:21.803710 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803653    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30638c0c-37ad-4576-a2ff-478fb62597c4" containerName="revoke-keys"
Jul 16 21:19:21.803710 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803662    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="30638c0c-37ad-4576-a2ff-478fb62597c4" containerName="revoke-keys"
Jul 16 21:19:21.803710 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803678    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30638c0c-37ad-4576-a2ff-478fb62597c4" containerName="revoke-keys"
Jul 16 21:19:21.803710 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803686    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="30638c0c-37ad-4576-a2ff-478fb62597c4" containerName="revoke-keys"
Jul 16 21:19:21.803710 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803696    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f801ef-7be1-4495-b286-5d3a77af31b1" containerName="revoke-keys"
Jul 16 21:19:21.803710 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803704    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f801ef-7be1-4495-b286-5d3a77af31b1" containerName="revoke-keys"
Jul 16 21:19:21.803710 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803713    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d88e23-8037-42cc-aad6-781fa05b1427" containerName="revoke-keys"
Jul 16 21:19:21.804354 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803722    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d88e23-8037-42cc-aad6-781fa05b1427" containerName="revoke-keys"
Jul 16 21:19:21.804354 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803827    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f801ef-7be1-4495-b286-5d3a77af31b1" containerName="revoke-keys"
Jul 16 21:19:21.804354 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803841    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d88e23-8037-42cc-aad6-781fa05b1427" containerName="revoke-keys"
Jul 16 21:19:21.804354 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803851    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d88e23-8037-42cc-aad6-781fa05b1427" containerName="revoke-keys"
Jul 16 21:19:21.804354 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803860    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f801ef-7be1-4495-b286-5d3a77af31b1" containerName="revoke-keys"
Jul 16 21:19:21.804354 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803867    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f801ef-7be1-4495-b286-5d3a77af31b1" containerName="revoke-keys"
Jul 16 21:19:21.804354 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803875    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="30638c0c-37ad-4576-a2ff-478fb62597c4" containerName="revoke-keys"
Jul 16 21:19:21.804354 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803884    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d88e23-8037-42cc-aad6-781fa05b1427" containerName="revoke-keys"
Jul 16 21:19:21.804354 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.803892    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="30638c0c-37ad-4576-a2ff-478fb62597c4" containerName="revoke-keys"
Jul 16 21:19:21.807025 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.807003    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"
Jul 16 21:19:21.809575 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.809550    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c69b60-serving-cert\""
Jul 16 21:19:21.816998 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.816963    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"]
Jul 16 21:19:21.931124 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.931060    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5792\" (UniqueName: \"kubernetes.io/projected/38836870-ea92-4c5d-96ac-29ca53c88d3c-kube-api-access-s5792\") pod \"maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7\" (UID: \"38836870-ea92-4c5d-96ac-29ca53c88d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"
Jul 16 21:19:21.931326 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.931188    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38836870-ea92-4c5d-96ac-29ca53c88d3c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7\" (UID: \"38836870-ea92-4c5d-96ac-29ca53c88d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"
Jul 16 21:19:21.931326 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:21.931232    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38836870-ea92-4c5d-96ac-29ca53c88d3c-maas-api-tls\") pod \"maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7\" (UID: \"38836870-ea92-4c5d-96ac-29ca53c88d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"
Jul 16 21:19:22.032081 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:22.032046    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38836870-ea92-4c5d-96ac-29ca53c88d3c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7\" (UID: \"38836870-ea92-4c5d-96ac-29ca53c88d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"
Jul 16 21:19:22.032081 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:22.032084    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38836870-ea92-4c5d-96ac-29ca53c88d3c-maas-api-tls\") pod \"maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7\" (UID: \"38836870-ea92-4c5d-96ac-29ca53c88d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"
Jul 16 21:19:22.032303 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:22.032169    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5792\" (UniqueName: \"kubernetes.io/projected/38836870-ea92-4c5d-96ac-29ca53c88d3c-kube-api-access-s5792\") pod \"maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7\" (UID: \"38836870-ea92-4c5d-96ac-29ca53c88d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"
Jul 16 21:19:22.032303 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:19:22.032270    2530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-serving-cert: secret "maas-api-e2e-inf-a-c69b60-serving-cert" not found
Jul 16 21:19:22.032369 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:19:22.032345    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38836870-ea92-4c5d-96ac-29ca53c88d3c-maas-api-tls podName:38836870-ea92-4c5d-96ac-29ca53c88d3c nodeName:}" failed. No retries permitted until 2026-07-16 21:19:22.53232671 +0000 UTC m=+2800.553914454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/38836870-ea92-4c5d-96ac-29ca53c88d3c-maas-api-tls") pod "maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7" (UID: "38836870-ea92-4c5d-96ac-29ca53c88d3c") : secret "maas-api-e2e-inf-a-c69b60-serving-cert" not found
Jul 16 21:19:22.032813 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:22.032790    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38836870-ea92-4c5d-96ac-29ca53c88d3c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7\" (UID: \"38836870-ea92-4c5d-96ac-29ca53c88d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"
Jul 16 21:19:22.044223 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:22.044194    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5792\" (UniqueName: \"kubernetes.io/projected/38836870-ea92-4c5d-96ac-29ca53c88d3c-kube-api-access-s5792\") pod \"maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7\" (UID: \"38836870-ea92-4c5d-96ac-29ca53c88d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"
Jul 16 21:19:22.536704 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:22.536667    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38836870-ea92-4c5d-96ac-29ca53c88d3c-maas-api-tls\") pod \"maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7\" (UID: \"38836870-ea92-4c5d-96ac-29ca53c88d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"
Jul 16 21:19:22.539207 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:22.539177    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38836870-ea92-4c5d-96ac-29ca53c88d3c-maas-api-tls\") pod \"maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7\" (UID: \"38836870-ea92-4c5d-96ac-29ca53c88d3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"
Jul 16 21:19:22.719031 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:22.718993    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"
Jul 16 21:19:22.858996 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:22.858974    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"]
Jul 16 21:19:22.861593 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:19:22.861555    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38836870_ea92_4c5d_96ac_29ca53c88d3c.slice/crio-33884dab88b4bab5c0fcbdefaf6817cc42b987ed1a06b12c62df743046c8d2a0 WatchSource:0}: Error finding container 33884dab88b4bab5c0fcbdefaf6817cc42b987ed1a06b12c62df743046c8d2a0: Status 404 returned error can't find the container with id 33884dab88b4bab5c0fcbdefaf6817cc42b987ed1a06b12c62df743046c8d2a0
Jul 16 21:19:23.137351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:23.137274    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7" event={"ID":"38836870-ea92-4c5d-96ac-29ca53c88d3c","Type":"ContainerStarted","Data":"33884dab88b4bab5c0fcbdefaf6817cc42b987ed1a06b12c62df743046c8d2a0"}
Jul 16 21:19:24.143160 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:24.143118    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7" event={"ID":"38836870-ea92-4c5d-96ac-29ca53c88d3c","Type":"ContainerStarted","Data":"19d79218572b9ad30352a3934d9bde66aabba0092e442746e1ce6b696552cd97"}
Jul 16 21:19:24.143528 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:24.143195    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"
Jul 16 21:19:24.172665 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:24.172610    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7" podStartSLOduration=2.590085259 podStartE2EDuration="3.17259368s" totalImagesPullingTime="582.508421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:19:24.17119052 +0000 UTC m=+2802.192778288" watchObservedRunningTime="2026-07-16 21:19:24.17259368 +0000 UTC m=+2802.194181445"
Jul 16 21:19:29.398825 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.398789    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6"]
Jul 16 21:19:29.399236 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.399138    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d88e23-8037-42cc-aad6-781fa05b1427" containerName="revoke-keys"
Jul 16 21:19:29.399236 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.399150    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d88e23-8037-42cc-aad6-781fa05b1427" containerName="revoke-keys"
Jul 16 21:19:29.399236 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.399206    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="30638c0c-37ad-4576-a2ff-478fb62597c4" containerName="revoke-keys"
Jul 16 21:19:29.402232 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.402215    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6"
Jul 16 21:19:29.404502 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.404479    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:19:29.410197 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.410164    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6"]
Jul 16 21:19:29.501750 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.501720    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c97846-4513-40af-82b9-9a7a127d5841-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6\" (UID: \"f7c97846-4513-40af-82b9-9a7a127d5841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6"
Jul 16 21:19:29.602861 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.602826    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c97846-4513-40af-82b9-9a7a127d5841-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6\" (UID: \"f7c97846-4513-40af-82b9-9a7a127d5841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6"
Jul 16 21:19:29.603432 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.603415    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c97846-4513-40af-82b9-9a7a127d5841-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6\" (UID: \"f7c97846-4513-40af-82b9-9a7a127d5841\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6"
Jul 16 21:19:29.713866 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.713834    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6"
Jul 16 21:19:29.797749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.797707    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"]
Jul 16 21:19:29.802533 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.802510    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"
Jul 16 21:19:29.805903 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.805881    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-72cda6-serving-cert\""
Jul 16 21:19:29.818983 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.818040    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"]
Jul 16 21:19:29.864476 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.864381    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6"]
Jul 16 21:19:29.867403 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:19:29.867375    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c97846_4513_40af_82b9_9a7a127d5841.slice/crio-c78130b0ecf24df40be1a2c1baf21acba72f1c2ca9c7dc9af9592291f4a76595 WatchSource:0}: Error finding container c78130b0ecf24df40be1a2c1baf21acba72f1c2ca9c7dc9af9592291f4a76595: Status 404 returned error can't find the container with id c78130b0ecf24df40be1a2c1baf21acba72f1c2ca9c7dc9af9592291f4a76595
Jul 16 21:19:29.906174 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.906145    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjt6t\" (UniqueName: \"kubernetes.io/projected/b163dc4b-6403-4c32-8bba-380690b93ad1-kube-api-access-qjt6t\") pod \"maas-api-e2e-inf-b-72cda6-64c8d78866-smljh\" (UID: \"b163dc4b-6403-4c32-8bba-380690b93ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"
Jul 16 21:19:29.906311 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.906237    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b163dc4b-6403-4c32-8bba-380690b93ad1-maas-api-tls\") pod \"maas-api-e2e-inf-b-72cda6-64c8d78866-smljh\" (UID: \"b163dc4b-6403-4c32-8bba-380690b93ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"
Jul 16 21:19:29.906311 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:29.906275    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b163dc4b-6403-4c32-8bba-380690b93ad1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-72cda6-64c8d78866-smljh\" (UID: \"b163dc4b-6403-4c32-8bba-380690b93ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"
Jul 16 21:19:30.006707 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:30.006630    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjt6t\" (UniqueName: \"kubernetes.io/projected/b163dc4b-6403-4c32-8bba-380690b93ad1-kube-api-access-qjt6t\") pod \"maas-api-e2e-inf-b-72cda6-64c8d78866-smljh\" (UID: \"b163dc4b-6403-4c32-8bba-380690b93ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"
Jul 16 21:19:30.006707 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:30.006692    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b163dc4b-6403-4c32-8bba-380690b93ad1-maas-api-tls\") pod \"maas-api-e2e-inf-b-72cda6-64c8d78866-smljh\" (UID: \"b163dc4b-6403-4c32-8bba-380690b93ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"
Jul 16 21:19:30.006937 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:30.006717    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b163dc4b-6403-4c32-8bba-380690b93ad1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-72cda6-64c8d78866-smljh\" (UID: \"b163dc4b-6403-4c32-8bba-380690b93ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"
Jul 16 21:19:30.007319 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:30.007300    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b163dc4b-6403-4c32-8bba-380690b93ad1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-72cda6-64c8d78866-smljh\" (UID: \"b163dc4b-6403-4c32-8bba-380690b93ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"
Jul 16 21:19:30.009272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:30.009251    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b163dc4b-6403-4c32-8bba-380690b93ad1-maas-api-tls\") pod \"maas-api-e2e-inf-b-72cda6-64c8d78866-smljh\" (UID: \"b163dc4b-6403-4c32-8bba-380690b93ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"
Jul 16 21:19:30.020967 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:30.020942    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjt6t\" (UniqueName: \"kubernetes.io/projected/b163dc4b-6403-4c32-8bba-380690b93ad1-kube-api-access-qjt6t\") pod \"maas-api-e2e-inf-b-72cda6-64c8d78866-smljh\" (UID: \"b163dc4b-6403-4c32-8bba-380690b93ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"
Jul 16 21:19:30.119864 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:30.119825    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"
Jul 16 21:19:30.154343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:30.154318    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"
Jul 16 21:19:30.173671 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:30.173643    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6" event={"ID":"f7c97846-4513-40af-82b9-9a7a127d5841","Type":"ContainerStarted","Data":"fdb79431b3c725dfe52be95c982b08542cf27b8a5e4b9308132e769b9b22caca"}
Jul 16 21:19:30.173793 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:30.173701    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6" event={"ID":"f7c97846-4513-40af-82b9-9a7a127d5841","Type":"ContainerStarted","Data":"c78130b0ecf24df40be1a2c1baf21acba72f1c2ca9c7dc9af9592291f4a76595"}
Jul 16 21:19:30.221980 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:30.221924    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6" podStartSLOduration=1.221902298 podStartE2EDuration="1.221902298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:19:30.220506765 +0000 UTC m=+2808.242094533" watchObservedRunningTime="2026-07-16 21:19:30.221902298 +0000 UTC m=+2808.243490065"
Jul 16 21:19:30.292056 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:30.291979    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"]
Jul 16 21:19:30.303558 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:19:30.303529    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb163dc4b_6403_4c32_8bba_380690b93ad1.slice/crio-d20e264198eb13a8a92e56df20a56debbe4d1e38fe86a37b9e943e374525eab0 WatchSource:0}: Error finding container d20e264198eb13a8a92e56df20a56debbe4d1e38fe86a37b9e943e374525eab0: Status 404 returned error can't find the container with id d20e264198eb13a8a92e56df20a56debbe4d1e38fe86a37b9e943e374525eab0
Jul 16 21:19:31.179278 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:31.179193    2530 generic.go:356] "Generic (PLEG): container finished" podID="f7c97846-4513-40af-82b9-9a7a127d5841" containerID="fdb79431b3c725dfe52be95c982b08542cf27b8a5e4b9308132e769b9b22caca" exitCode=6
Jul 16 21:19:31.179729 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:31.179285    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6" event={"ID":"f7c97846-4513-40af-82b9-9a7a127d5841","Type":"ContainerDied","Data":"fdb79431b3c725dfe52be95c982b08542cf27b8a5e4b9308132e769b9b22caca"}
Jul 16 21:19:31.179729 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:31.179507    2530 scope.go:117] "RemoveContainer" containerID="fdb79431b3c725dfe52be95c982b08542cf27b8a5e4b9308132e769b9b22caca"
Jul 16 21:19:31.180846 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:31.180817    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh" event={"ID":"b163dc4b-6403-4c32-8bba-380690b93ad1","Type":"ContainerStarted","Data":"1b85e265d6c10638acf8be888d7d96249eefc80ec75ee607d86c13d84c74bc34"}
Jul 16 21:19:31.181001 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:31.180860    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh" event={"ID":"b163dc4b-6403-4c32-8bba-380690b93ad1","Type":"ContainerStarted","Data":"d20e264198eb13a8a92e56df20a56debbe4d1e38fe86a37b9e943e374525eab0"}
Jul 16 21:19:31.181001 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:31.180895    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"
Jul 16 21:19:31.221466 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:31.221405    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh" podStartSLOduration=1.7039309569999999 podStartE2EDuration="2.221384906s" totalImagesPullingTime="517.453949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:19:31.217304013 +0000 UTC m=+2809.238891779" watchObservedRunningTime="2026-07-16 21:19:31.221384906 +0000 UTC m=+2809.242972672"
Jul 16 21:19:32.187660 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:32.187578    2530 generic.go:356] "Generic (PLEG): container finished" podID="f7c97846-4513-40af-82b9-9a7a127d5841" containerID="6bcbd86a698f4b5ba1fa6ebb7f408d9a7e484f91c6720f2206b560d8f1adf75c" exitCode=6
Jul 16 21:19:32.188300 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:32.187678    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6" event={"ID":"f7c97846-4513-40af-82b9-9a7a127d5841","Type":"ContainerDied","Data":"6bcbd86a698f4b5ba1fa6ebb7f408d9a7e484f91c6720f2206b560d8f1adf75c"}
Jul 16 21:19:32.188300 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:32.187781    2530 scope.go:117] "RemoveContainer" containerID="fdb79431b3c725dfe52be95c982b08542cf27b8a5e4b9308132e769b9b22caca"
Jul 16 21:19:32.188300 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:32.187857    2530 scope.go:117] "RemoveContainer" containerID="6bcbd86a698f4b5ba1fa6ebb7f408d9a7e484f91c6720f2206b560d8f1adf75c"
Jul 16 21:19:32.188300 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:19:32.188136    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6_odh-ai-gateway-infra(f7c97846-4513-40af-82b9-9a7a127d5841)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6" podUID="f7c97846-4513-40af-82b9-9a7a127d5841"
Jul 16 21:19:32.250717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:32.250674    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr"]
Jul 16 21:19:32.255249 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:32.255226    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr"
Jul 16 21:19:32.262703 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:32.262118    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr"]
Jul 16 21:19:32.329538 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:32.329496    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c290aade-086f-4f66-91f3-021d0e1eb2d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr\" (UID: \"c290aade-086f-4f66-91f3-021d0e1eb2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr"
Jul 16 21:19:32.430443 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:32.430405    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c290aade-086f-4f66-91f3-021d0e1eb2d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr\" (UID: \"c290aade-086f-4f66-91f3-021d0e1eb2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr"
Jul 16 21:19:32.431012 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:32.430993    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c290aade-086f-4f66-91f3-021d0e1eb2d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr\" (UID: \"c290aade-086f-4f66-91f3-021d0e1eb2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr"
Jul 16 21:19:32.567199 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:32.567165    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr"
Jul 16 21:19:32.700844 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:32.700816    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr"]
Jul 16 21:19:32.702524 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:19:32.702495    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc290aade_086f_4f66_91f3_021d0e1eb2d9.slice/crio-00f94e0c7f7c95f35a53f23d3468dea648d60a1ab54534fe34413b1dbe96885f WatchSource:0}: Error finding container 00f94e0c7f7c95f35a53f23d3468dea648d60a1ab54534fe34413b1dbe96885f: Status 404 returned error can't find the container with id 00f94e0c7f7c95f35a53f23d3468dea648d60a1ab54534fe34413b1dbe96885f
Jul 16 21:19:33.193758 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:33.193664    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr" event={"ID":"c290aade-086f-4f66-91f3-021d0e1eb2d9","Type":"ContainerStarted","Data":"cc78dcf16a128bd90aa102efa4219796b4f44c9bfc095b532eeeabd335664741"}
Jul 16 21:19:33.193758 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:33.193717    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr" event={"ID":"c290aade-086f-4f66-91f3-021d0e1eb2d9","Type":"ContainerStarted","Data":"00f94e0c7f7c95f35a53f23d3468dea648d60a1ab54534fe34413b1dbe96885f"}
Jul 16 21:19:33.195425 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:33.195405    2530 scope.go:117] "RemoveContainer" containerID="6bcbd86a698f4b5ba1fa6ebb7f408d9a7e484f91c6720f2206b560d8f1adf75c"
Jul 16 21:19:33.195577 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:19:33.195560    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6_odh-ai-gateway-infra(f7c97846-4513-40af-82b9-9a7a127d5841)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6" podUID="f7c97846-4513-40af-82b9-9a7a127d5841"
Jul 16 21:19:33.209425 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:33.209383    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr" podStartSLOduration=1.209369357 podStartE2EDuration="1.209369357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:19:33.208712734 +0000 UTC m=+2811.230300534" watchObservedRunningTime="2026-07-16 21:19:33.209369357 +0000 UTC m=+2811.230957123"
Jul 16 21:19:34.157278 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:34.157230    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs"]
Jul 16 21:19:34.160420 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:34.160404    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs"
Jul 16 21:19:34.169498 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:34.169473    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs"]
Jul 16 21:19:34.201363 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:34.201329    2530 generic.go:356] "Generic (PLEG): container finished" podID="c290aade-086f-4f66-91f3-021d0e1eb2d9" containerID="cc78dcf16a128bd90aa102efa4219796b4f44c9bfc095b532eeeabd335664741" exitCode=6
Jul 16 21:19:34.201718 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:34.201422    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr" event={"ID":"c290aade-086f-4f66-91f3-021d0e1eb2d9","Type":"ContainerDied","Data":"cc78dcf16a128bd90aa102efa4219796b4f44c9bfc095b532eeeabd335664741"}
Jul 16 21:19:34.201718 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:34.201556    2530 scope.go:117] "RemoveContainer" containerID="cc78dcf16a128bd90aa102efa4219796b4f44c9bfc095b532eeeabd335664741"
Jul 16 21:19:34.245063 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:34.245039    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5dfc96f-b2ea-46e1-97a7-8f527f5794f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs\" (UID: \"c5dfc96f-b2ea-46e1-97a7-8f527f5794f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs"
Jul 16 21:19:34.345814 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:34.345773    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5dfc96f-b2ea-46e1-97a7-8f527f5794f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs\" (UID: \"c5dfc96f-b2ea-46e1-97a7-8f527f5794f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs"
Jul 16 21:19:34.346417 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:34.346397    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5dfc96f-b2ea-46e1-97a7-8f527f5794f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs\" (UID: \"c5dfc96f-b2ea-46e1-97a7-8f527f5794f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs"
Jul 16 21:19:34.471328 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:34.471301    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs"
Jul 16 21:19:34.601944 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:34.601918    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs"]
Jul 16 21:19:34.604069 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:19:34.604043    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5dfc96f_b2ea_46e1_97a7_8f527f5794f0.slice/crio-8c48f48800fea8a852eb7f176768a1bda1a51fc84f0399d43b02f46fb029c003 WatchSource:0}: Error finding container 8c48f48800fea8a852eb7f176768a1bda1a51fc84f0399d43b02f46fb029c003: Status 404 returned error can't find the container with id 8c48f48800fea8a852eb7f176768a1bda1a51fc84f0399d43b02f46fb029c003
Jul 16 21:19:35.207454 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:35.207419    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs" event={"ID":"c5dfc96f-b2ea-46e1-97a7-8f527f5794f0","Type":"ContainerStarted","Data":"fdb7ea65f1d192577d1d25c15af68849cce13787c50dff472a0c1e6a56aafa0b"}
Jul 16 21:19:35.207901 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:35.207483    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs" event={"ID":"c5dfc96f-b2ea-46e1-97a7-8f527f5794f0","Type":"ContainerStarted","Data":"8c48f48800fea8a852eb7f176768a1bda1a51fc84f0399d43b02f46fb029c003"}
Jul 16 21:19:35.209186 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:35.209164    2530 generic.go:356] "Generic (PLEG): container finished" podID="c290aade-086f-4f66-91f3-021d0e1eb2d9" containerID="20b5b31f7a594d12acb1e02e31c25f401b93e165042d1567724c55d76f0b7418" exitCode=6
Jul 16 21:19:35.209309 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:35.209212    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr" event={"ID":"c290aade-086f-4f66-91f3-021d0e1eb2d9","Type":"ContainerDied","Data":"20b5b31f7a594d12acb1e02e31c25f401b93e165042d1567724c55d76f0b7418"}
Jul 16 21:19:35.209309 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:35.209261    2530 scope.go:117] "RemoveContainer" containerID="cc78dcf16a128bd90aa102efa4219796b4f44c9bfc095b532eeeabd335664741"
Jul 16 21:19:35.209403 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:35.209372    2530 scope.go:117] "RemoveContainer" containerID="20b5b31f7a594d12acb1e02e31c25f401b93e165042d1567724c55d76f0b7418"
Jul 16 21:19:35.209588 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:19:35.209559    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr_odh-ai-gateway-infra(c290aade-086f-4f66-91f3-021d0e1eb2d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr" podUID="c290aade-086f-4f66-91f3-021d0e1eb2d9"
Jul 16 21:19:35.227526 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:35.227487    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs" podStartSLOduration=1.227474423 podStartE2EDuration="1.227474423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:19:35.226249748 +0000 UTC m=+2813.247837514" watchObservedRunningTime="2026-07-16 21:19:35.227474423 +0000 UTC m=+2813.249062189"
Jul 16 21:19:36.220903 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:36.220860    2530 generic.go:356] "Generic (PLEG): container finished" podID="c5dfc96f-b2ea-46e1-97a7-8f527f5794f0" containerID="fdb7ea65f1d192577d1d25c15af68849cce13787c50dff472a0c1e6a56aafa0b" exitCode=6
Jul 16 21:19:36.221390 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:36.220945    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs" event={"ID":"c5dfc96f-b2ea-46e1-97a7-8f527f5794f0","Type":"ContainerDied","Data":"fdb7ea65f1d192577d1d25c15af68849cce13787c50dff472a0c1e6a56aafa0b"}
Jul 16 21:19:36.221390 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:36.221166    2530 scope.go:117] "RemoveContainer" containerID="fdb7ea65f1d192577d1d25c15af68849cce13787c50dff472a0c1e6a56aafa0b"
Jul 16 21:19:36.222988 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:36.222972    2530 scope.go:117] "RemoveContainer" containerID="20b5b31f7a594d12acb1e02e31c25f401b93e165042d1567724c55d76f0b7418"
Jul 16 21:19:36.223201 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:19:36.223181    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr_odh-ai-gateway-infra(c290aade-086f-4f66-91f3-021d0e1eb2d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr" podUID="c290aade-086f-4f66-91f3-021d0e1eb2d9"
Jul 16 21:19:37.193837 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:37.193752    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"
Jul 16 21:19:37.230343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:37.230307    2530 generic.go:356] "Generic (PLEG): container finished" podID="c5dfc96f-b2ea-46e1-97a7-8f527f5794f0" containerID="ba04f11a6240be832d54cf15ca4d2eec1b7930ddf0b07c903eb1c2db6b117d13" exitCode=6
Jul 16 21:19:37.230805 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:37.230393    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs" event={"ID":"c5dfc96f-b2ea-46e1-97a7-8f527f5794f0","Type":"ContainerDied","Data":"ba04f11a6240be832d54cf15ca4d2eec1b7930ddf0b07c903eb1c2db6b117d13"}
Jul 16 21:19:37.230805 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:37.230463    2530 scope.go:117] "RemoveContainer" containerID="fdb7ea65f1d192577d1d25c15af68849cce13787c50dff472a0c1e6a56aafa0b"
Jul 16 21:19:37.230805 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:37.230677    2530 scope.go:117] "RemoveContainer" containerID="ba04f11a6240be832d54cf15ca4d2eec1b7930ddf0b07c903eb1c2db6b117d13"
Jul 16 21:19:37.230956 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:19:37.230915    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs_odh-ai-gateway-infra(c5dfc96f-b2ea-46e1-97a7-8f527f5794f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs" podUID="c5dfc96f-b2ea-46e1-97a7-8f527f5794f0"
Jul 16 21:19:38.237006 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:38.236978    2530 scope.go:117] "RemoveContainer" containerID="ba04f11a6240be832d54cf15ca4d2eec1b7930ddf0b07c903eb1c2db6b117d13"
Jul 16 21:19:38.237402 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:19:38.237208    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs_odh-ai-gateway-infra(c5dfc96f-b2ea-46e1-97a7-8f527f5794f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs" podUID="c5dfc96f-b2ea-46e1-97a7-8f527f5794f0"
Jul 16 21:19:45.502203 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:45.502171    2530 scope.go:117] "RemoveContainer" containerID="6bcbd86a698f4b5ba1fa6ebb7f408d9a7e484f91c6720f2206b560d8f1adf75c"
Jul 16 21:19:46.274142 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:46.274088    2530 generic.go:356] "Generic (PLEG): container finished" podID="f7c97846-4513-40af-82b9-9a7a127d5841" containerID="5dfb5fd6191466b61709cd988b762083252f8e6d7632c752e6522deebb944735" exitCode=6
Jul 16 21:19:46.274255 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:46.274169    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6" event={"ID":"f7c97846-4513-40af-82b9-9a7a127d5841","Type":"ContainerDied","Data":"5dfb5fd6191466b61709cd988b762083252f8e6d7632c752e6522deebb944735"}
Jul 16 21:19:46.274255 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:46.274236    2530 scope.go:117] "RemoveContainer" containerID="6bcbd86a698f4b5ba1fa6ebb7f408d9a7e484f91c6720f2206b560d8f1adf75c"
Jul 16 21:19:46.274601 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:46.274582    2530 scope.go:117] "RemoveContainer" containerID="5dfb5fd6191466b61709cd988b762083252f8e6d7632c752e6522deebb944735"
Jul 16 21:19:46.274778 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:19:46.274755    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6_odh-ai-gateway-infra(f7c97846-4513-40af-82b9-9a7a127d5841)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6" podUID="f7c97846-4513-40af-82b9-9a7a127d5841"
Jul 16 21:19:47.305142 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:47.305110    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6"]
Jul 16 21:19:47.438892 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:47.438871    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6"
Jul 16 21:19:47.453838 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:47.453812    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c97846-4513-40af-82b9-9a7a127d5841-maas-api-service-ca\") pod \"f7c97846-4513-40af-82b9-9a7a127d5841\" (UID: \"f7c97846-4513-40af-82b9-9a7a127d5841\") "
Jul 16 21:19:47.454124 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:47.454086    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7c97846-4513-40af-82b9-9a7a127d5841-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7c97846-4513-40af-82b9-9a7a127d5841" (UID: "f7c97846-4513-40af-82b9-9a7a127d5841"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:19:47.554780 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:47.554752    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7c97846-4513-40af-82b9-9a7a127d5841-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:19:48.288861 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:48.288826    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6" event={"ID":"f7c97846-4513-40af-82b9-9a7a127d5841","Type":"ContainerDied","Data":"c78130b0ecf24df40be1a2c1baf21acba72f1c2ca9c7dc9af9592291f4a76595"}
Jul 16 21:19:48.289056 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:48.288882    2530 scope.go:117] "RemoveContainer" containerID="5dfb5fd6191466b61709cd988b762083252f8e6d7632c752e6522deebb944735"
Jul 16 21:19:48.289056 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:48.288843    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6"
Jul 16 21:19:48.311439 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:48.311416    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6"]
Jul 16 21:19:48.321906 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:48.321879    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-hnpp6"]
Jul 16 21:19:48.502761 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:48.502735    2530 scope.go:117] "RemoveContainer" containerID="20b5b31f7a594d12acb1e02e31c25f401b93e165042d1567724c55d76f0b7418"
Jul 16 21:19:48.507904 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:48.507880    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7c97846-4513-40af-82b9-9a7a127d5841" path="/var/lib/kubelet/pods/f7c97846-4513-40af-82b9-9a7a127d5841/volumes"
Jul 16 21:19:49.295495 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:49.295465    2530 generic.go:356] "Generic (PLEG): container finished" podID="c290aade-086f-4f66-91f3-021d0e1eb2d9" containerID="6b43171283d28ad82f9115106a79768a7f84c014ee01d88576aa252ac0de1f03" exitCode=6
Jul 16 21:19:49.295618 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:49.295541    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr" event={"ID":"c290aade-086f-4f66-91f3-021d0e1eb2d9","Type":"ContainerDied","Data":"6b43171283d28ad82f9115106a79768a7f84c014ee01d88576aa252ac0de1f03"}
Jul 16 21:19:49.295618 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:49.295601    2530 scope.go:117] "RemoveContainer" containerID="20b5b31f7a594d12acb1e02e31c25f401b93e165042d1567724c55d76f0b7418"
Jul 16 21:19:49.295874 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:49.295850    2530 scope.go:117] "RemoveContainer" containerID="6b43171283d28ad82f9115106a79768a7f84c014ee01d88576aa252ac0de1f03"
Jul 16 21:19:49.296064 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:19:49.296040    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr_odh-ai-gateway-infra(c290aade-086f-4f66-91f3-021d0e1eb2d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr" podUID="c290aade-086f-4f66-91f3-021d0e1eb2d9"
Jul 16 21:19:50.320508 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:50.320475    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr"]
Jul 16 21:19:50.466954 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:50.466926    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr"
Jul 16 21:19:50.477019 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:50.476993    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c290aade-086f-4f66-91f3-021d0e1eb2d9-maas-api-service-ca\") pod \"c290aade-086f-4f66-91f3-021d0e1eb2d9\" (UID: \"c290aade-086f-4f66-91f3-021d0e1eb2d9\") "
Jul 16 21:19:50.477330 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:50.477300    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c290aade-086f-4f66-91f3-021d0e1eb2d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c290aade-086f-4f66-91f3-021d0e1eb2d9" (UID: "c290aade-086f-4f66-91f3-021d0e1eb2d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:19:50.507592 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:50.507566    2530 scope.go:117] "RemoveContainer" containerID="ba04f11a6240be832d54cf15ca4d2eec1b7930ddf0b07c903eb1c2db6b117d13"
Jul 16 21:19:50.578583 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:50.578522    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c290aade-086f-4f66-91f3-021d0e1eb2d9-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:19:51.308327 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:51.308279    2530 generic.go:356] "Generic (PLEG): container finished" podID="c5dfc96f-b2ea-46e1-97a7-8f527f5794f0" containerID="28435bb577f34f8b96c7075f2c5dabfc746c3b3b971eb87da1f18ee64ff3e137" exitCode=6
Jul 16 21:19:51.308489 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:51.308353    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs" event={"ID":"c5dfc96f-b2ea-46e1-97a7-8f527f5794f0","Type":"ContainerDied","Data":"28435bb577f34f8b96c7075f2c5dabfc746c3b3b971eb87da1f18ee64ff3e137"}
Jul 16 21:19:51.308489 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:51.308422    2530 scope.go:117] "RemoveContainer" containerID="ba04f11a6240be832d54cf15ca4d2eec1b7930ddf0b07c903eb1c2db6b117d13"
Jul 16 21:19:51.308764 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:51.308745    2530 scope.go:117] "RemoveContainer" containerID="28435bb577f34f8b96c7075f2c5dabfc746c3b3b971eb87da1f18ee64ff3e137"
Jul 16 21:19:51.308990 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:19:51.308967    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs_odh-ai-gateway-infra(c5dfc96f-b2ea-46e1-97a7-8f527f5794f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs" podUID="c5dfc96f-b2ea-46e1-97a7-8f527f5794f0"
Jul 16 21:19:51.309887 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:51.309868    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr" event={"ID":"c290aade-086f-4f66-91f3-021d0e1eb2d9","Type":"ContainerDied","Data":"00f94e0c7f7c95f35a53f23d3468dea648d60a1ab54534fe34413b1dbe96885f"}
Jul 16 21:19:51.309957 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:51.309922    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr"
Jul 16 21:19:51.322445 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:51.322337    2530 scope.go:117] "RemoveContainer" containerID="6b43171283d28ad82f9115106a79768a7f84c014ee01d88576aa252ac0de1f03"
Jul 16 21:19:51.339095 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:51.339074    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr"]
Jul 16 21:19:51.348270 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:51.348249    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fs8kr"]
Jul 16 21:19:52.333534 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:52.333502    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs"]
Jul 16 21:19:52.464252 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:52.464228    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs"
Jul 16 21:19:52.492933 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:52.492909    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5dfc96f-b2ea-46e1-97a7-8f527f5794f0-maas-api-service-ca\") pod \"c5dfc96f-b2ea-46e1-97a7-8f527f5794f0\" (UID: \"c5dfc96f-b2ea-46e1-97a7-8f527f5794f0\") "
Jul 16 21:19:52.493301 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:52.493278    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5dfc96f-b2ea-46e1-97a7-8f527f5794f0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c5dfc96f-b2ea-46e1-97a7-8f527f5794f0" (UID: "c5dfc96f-b2ea-46e1-97a7-8f527f5794f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:19:52.507130 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:52.507077    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c290aade-086f-4f66-91f3-021d0e1eb2d9" path="/var/lib/kubelet/pods/c290aade-086f-4f66-91f3-021d0e1eb2d9/volumes"
Jul 16 21:19:52.593612 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:52.593557    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5dfc96f-b2ea-46e1-97a7-8f527f5794f0-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:19:53.322910 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:53.322877    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs"
Jul 16 21:19:53.323130 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:53.322901    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs" event={"ID":"c5dfc96f-b2ea-46e1-97a7-8f527f5794f0","Type":"ContainerDied","Data":"8c48f48800fea8a852eb7f176768a1bda1a51fc84f0399d43b02f46fb029c003"}
Jul 16 21:19:53.323130 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:53.322970    2530 scope.go:117] "RemoveContainer" containerID="28435bb577f34f8b96c7075f2c5dabfc746c3b3b971eb87da1f18ee64ff3e137"
Jul 16 21:19:53.338175 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:53.338146    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs"]
Jul 16 21:19:53.344152 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:53.344128    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-l9gbs"]
Jul 16 21:19:54.507065 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:19:54.507033    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5dfc96f-b2ea-46e1-97a7-8f527f5794f0" path="/var/lib/kubelet/pods/c5dfc96f-b2ea-46e1-97a7-8f527f5794f0/volumes"
Jul 16 21:20:12.015327 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015296    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s"]
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015604    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7c97846-4513-40af-82b9-9a7a127d5841" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015615    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7c97846-4513-40af-82b9-9a7a127d5841" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015623    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c290aade-086f-4f66-91f3-021d0e1eb2d9" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015629    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c290aade-086f-4f66-91f3-021d0e1eb2d9" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015636    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5dfc96f-b2ea-46e1-97a7-8f527f5794f0" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015642    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5dfc96f-b2ea-46e1-97a7-8f527f5794f0" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015658    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c290aade-086f-4f66-91f3-021d0e1eb2d9" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015663    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c290aade-086f-4f66-91f3-021d0e1eb2d9" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015674    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7c97846-4513-40af-82b9-9a7a127d5841" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015679    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7c97846-4513-40af-82b9-9a7a127d5841" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015687    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5dfc96f-b2ea-46e1-97a7-8f527f5794f0" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015692    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5dfc96f-b2ea-46e1-97a7-8f527f5794f0" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015740    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7c97846-4513-40af-82b9-9a7a127d5841" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015747    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7c97846-4513-40af-82b9-9a7a127d5841" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015754    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c290aade-086f-4f66-91f3-021d0e1eb2d9" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015760    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5dfc96f-b2ea-46e1-97a7-8f527f5794f0" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015765    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7c97846-4513-40af-82b9-9a7a127d5841" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015771    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5dfc96f-b2ea-46e1-97a7-8f527f5794f0" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015776    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c290aade-086f-4f66-91f3-021d0e1eb2d9" containerName="revoke-keys"
Jul 16 21:20:12.015785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.015781    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5dfc96f-b2ea-46e1-97a7-8f527f5794f0" containerName="revoke-keys"
Jul 16 21:20:12.020294 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.020276    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s"
Jul 16 21:20:12.022766 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.022748    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:20:12.026023 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.025999    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s"]
Jul 16 21:20:12.143037 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.143009    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/775af836-8680-4a40-b92e-c993c69c3380-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s\" (UID: \"775af836-8680-4a40-b92e-c993c69c3380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s"
Jul 16 21:20:12.244266 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.244239    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/775af836-8680-4a40-b92e-c993c69c3380-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s\" (UID: \"775af836-8680-4a40-b92e-c993c69c3380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s"
Jul 16 21:20:12.244783 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.244766    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/775af836-8680-4a40-b92e-c993c69c3380-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s\" (UID: \"775af836-8680-4a40-b92e-c993c69c3380\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s"
Jul 16 21:20:12.331442 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.331383    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s"
Jul 16 21:20:12.490215 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:12.490183    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s"]
Jul 16 21:20:12.492030 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:20:12.492001    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775af836_8680_4a40_b92e_c993c69c3380.slice/crio-0e083b469afaef6e51fca54f5c5a0a7da7b7c4d36597424b4a2e70658fd65ae0 WatchSource:0}: Error finding container 0e083b469afaef6e51fca54f5c5a0a7da7b7c4d36597424b4a2e70658fd65ae0: Status 404 returned error can't find the container with id 0e083b469afaef6e51fca54f5c5a0a7da7b7c4d36597424b4a2e70658fd65ae0
Jul 16 21:20:13.415701 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:13.415669    2530 generic.go:356] "Generic (PLEG): container finished" podID="775af836-8680-4a40-b92e-c993c69c3380" containerID="6122050cdd84818a7d0e070b98f0aece1c41f478226d836f8b78b7e6c96ded94" exitCode=6
Jul 16 21:20:13.416026 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:13.415756    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s" event={"ID":"775af836-8680-4a40-b92e-c993c69c3380","Type":"ContainerDied","Data":"6122050cdd84818a7d0e070b98f0aece1c41f478226d836f8b78b7e6c96ded94"}
Jul 16 21:20:13.416026 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:13.415807    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s" event={"ID":"775af836-8680-4a40-b92e-c993c69c3380","Type":"ContainerStarted","Data":"0e083b469afaef6e51fca54f5c5a0a7da7b7c4d36597424b4a2e70658fd65ae0"}
Jul 16 21:20:13.416026 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:13.415905    2530 scope.go:117] "RemoveContainer" containerID="6122050cdd84818a7d0e070b98f0aece1c41f478226d836f8b78b7e6c96ded94"
Jul 16 21:20:14.422325 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:14.422240    2530 generic.go:356] "Generic (PLEG): container finished" podID="775af836-8680-4a40-b92e-c993c69c3380" containerID="19ccfcad76a732a3cae859197e93d5b2e1767a2112b26f26e68d704ec89aa1f0" exitCode=6
Jul 16 21:20:14.422714 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:14.422337    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s" event={"ID":"775af836-8680-4a40-b92e-c993c69c3380","Type":"ContainerDied","Data":"19ccfcad76a732a3cae859197e93d5b2e1767a2112b26f26e68d704ec89aa1f0"}
Jul 16 21:20:14.422714 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:14.422419    2530 scope.go:117] "RemoveContainer" containerID="6122050cdd84818a7d0e070b98f0aece1c41f478226d836f8b78b7e6c96ded94"
Jul 16 21:20:14.422714 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:14.422506    2530 scope.go:117] "RemoveContainer" containerID="19ccfcad76a732a3cae859197e93d5b2e1767a2112b26f26e68d704ec89aa1f0"
Jul 16 21:20:14.422874 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:20:14.422712    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s_odh-ai-gateway-infra(775af836-8680-4a40-b92e-c993c69c3380)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s" podUID="775af836-8680-4a40-b92e-c993c69c3380"
Jul 16 21:20:15.428858 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:15.428830    2530 scope.go:117] "RemoveContainer" containerID="19ccfcad76a732a3cae859197e93d5b2e1767a2112b26f26e68d704ec89aa1f0"
Jul 16 21:20:15.429273 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:20:15.429013    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s_odh-ai-gateway-infra(775af836-8680-4a40-b92e-c993c69c3380)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s" podUID="775af836-8680-4a40-b92e-c993c69c3380"
Jul 16 21:20:19.438003 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:19.437967    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84"]
Jul 16 21:20:19.438406 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:19.438355    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7c97846-4513-40af-82b9-9a7a127d5841" containerName="revoke-keys"
Jul 16 21:20:19.438406 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:19.438368    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7c97846-4513-40af-82b9-9a7a127d5841" containerName="revoke-keys"
Jul 16 21:20:19.438406 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:19.438377    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c290aade-086f-4f66-91f3-021d0e1eb2d9" containerName="revoke-keys"
Jul 16 21:20:19.438406 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:19.438382    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c290aade-086f-4f66-91f3-021d0e1eb2d9" containerName="revoke-keys"
Jul 16 21:20:19.438406 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:19.438388    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5dfc96f-b2ea-46e1-97a7-8f527f5794f0" containerName="revoke-keys"
Jul 16 21:20:19.438406 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:19.438395    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5dfc96f-b2ea-46e1-97a7-8f527f5794f0" containerName="revoke-keys"
Jul 16 21:20:19.438598 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:19.438449    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c290aade-086f-4f66-91f3-021d0e1eb2d9" containerName="revoke-keys"
Jul 16 21:20:19.442703 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:19.442683    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84"
Jul 16 21:20:19.449213 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:19.449186    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84"]
Jul 16 21:20:19.508154 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:19.508123    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/842535c6-c97d-433b-bbab-46a4a013d91b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84\" (UID: \"842535c6-c97d-433b-bbab-46a4a013d91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84"
Jul 16 21:20:19.608970 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:19.608943    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/842535c6-c97d-433b-bbab-46a4a013d91b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84\" (UID: \"842535c6-c97d-433b-bbab-46a4a013d91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84"
Jul 16 21:20:19.609536 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:19.609519    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/842535c6-c97d-433b-bbab-46a4a013d91b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84\" (UID: \"842535c6-c97d-433b-bbab-46a4a013d91b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84"
Jul 16 21:20:19.755929 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:19.755899    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84"
Jul 16 21:20:19.886288 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:19.886266    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84"]
Jul 16 21:20:19.888879 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:20:19.888854    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842535c6_c97d_433b_bbab_46a4a013d91b.slice/crio-17424754b5bf0a15214726cebdac66cde8e7a6eba7db1aa5833d724d750fd004 WatchSource:0}: Error finding container 17424754b5bf0a15214726cebdac66cde8e7a6eba7db1aa5833d724d750fd004: Status 404 returned error can't find the container with id 17424754b5bf0a15214726cebdac66cde8e7a6eba7db1aa5833d724d750fd004
Jul 16 21:20:20.452823 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:20.452794    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84" event={"ID":"842535c6-c97d-433b-bbab-46a4a013d91b","Type":"ContainerStarted","Data":"9d1299cc252338c0e33cac3622868f3d983ba4865d2e5d90c80f82b53facd3a4"}
Jul 16 21:20:20.453245 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:20.452844    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84" event={"ID":"842535c6-c97d-433b-bbab-46a4a013d91b","Type":"ContainerStarted","Data":"17424754b5bf0a15214726cebdac66cde8e7a6eba7db1aa5833d724d750fd004"}
Jul 16 21:20:20.468543 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:20.468498    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84" podStartSLOduration=1.468482658 podStartE2EDuration="1.468482658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:20:20.467684586 +0000 UTC m=+2858.489272352" watchObservedRunningTime="2026-07-16 21:20:20.468482658 +0000 UTC m=+2858.490070423"
Jul 16 21:20:21.458727 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:21.458683    2530 generic.go:356] "Generic (PLEG): container finished" podID="842535c6-c97d-433b-bbab-46a4a013d91b" containerID="9d1299cc252338c0e33cac3622868f3d983ba4865d2e5d90c80f82b53facd3a4" exitCode=6
Jul 16 21:20:21.459143 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:21.458743    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84" event={"ID":"842535c6-c97d-433b-bbab-46a4a013d91b","Type":"ContainerDied","Data":"9d1299cc252338c0e33cac3622868f3d983ba4865d2e5d90c80f82b53facd3a4"}
Jul 16 21:20:21.459143 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:21.458949    2530 scope.go:117] "RemoveContainer" containerID="9d1299cc252338c0e33cac3622868f3d983ba4865d2e5d90c80f82b53facd3a4"
Jul 16 21:20:22.293357 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:22.293319    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg"]
Jul 16 21:20:22.296564 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:22.296550    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg"
Jul 16 21:20:22.303904 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:22.303878    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg"]
Jul 16 21:20:22.427660 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:22.427625    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44938529-f44f-41a0-9b8b-fb33aca1870a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg\" (UID: \"44938529-f44f-41a0-9b8b-fb33aca1870a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg"
Jul 16 21:20:22.466536 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:22.466499    2530 generic.go:356] "Generic (PLEG): container finished" podID="842535c6-c97d-433b-bbab-46a4a013d91b" containerID="77f9ed0882bfe0573ca2603a1e343b1f878bbff57928a87dfa16f354bc76b198" exitCode=6
Jul 16 21:20:22.466953 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:22.466571    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84" event={"ID":"842535c6-c97d-433b-bbab-46a4a013d91b","Type":"ContainerDied","Data":"77f9ed0882bfe0573ca2603a1e343b1f878bbff57928a87dfa16f354bc76b198"}
Jul 16 21:20:22.466953 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:22.466627    2530 scope.go:117] "RemoveContainer" containerID="9d1299cc252338c0e33cac3622868f3d983ba4865d2e5d90c80f82b53facd3a4"
Jul 16 21:20:22.466953 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:22.466793    2530 scope.go:117] "RemoveContainer" containerID="77f9ed0882bfe0573ca2603a1e343b1f878bbff57928a87dfa16f354bc76b198"
Jul 16 21:20:22.467172 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:20:22.466971    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84_odh-ai-gateway-infra(842535c6-c97d-433b-bbab-46a4a013d91b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84" podUID="842535c6-c97d-433b-bbab-46a4a013d91b"
Jul 16 21:20:22.528517 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:22.528491    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44938529-f44f-41a0-9b8b-fb33aca1870a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg\" (UID: \"44938529-f44f-41a0-9b8b-fb33aca1870a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg"
Jul 16 21:20:22.529029 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:22.529013    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44938529-f44f-41a0-9b8b-fb33aca1870a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg\" (UID: \"44938529-f44f-41a0-9b8b-fb33aca1870a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg"
Jul 16 21:20:22.607985 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:22.607928    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg"
Jul 16 21:20:22.740320 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:22.740298    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg"]
Jul 16 21:20:22.741697 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:20:22.741671    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44938529_f44f_41a0_9b8b_fb33aca1870a.slice/crio-f3ae9f1cb2a41d9f71464a31f9a0a1777b3f8a0eebb5ee6b2737410781f031dc WatchSource:0}: Error finding container f3ae9f1cb2a41d9f71464a31f9a0a1777b3f8a0eebb5ee6b2737410781f031dc: Status 404 returned error can't find the container with id f3ae9f1cb2a41d9f71464a31f9a0a1777b3f8a0eebb5ee6b2737410781f031dc
Jul 16 21:20:23.472435 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:23.472406    2530 scope.go:117] "RemoveContainer" containerID="77f9ed0882bfe0573ca2603a1e343b1f878bbff57928a87dfa16f354bc76b198"
Jul 16 21:20:23.472792 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:20:23.472645    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84_odh-ai-gateway-infra(842535c6-c97d-433b-bbab-46a4a013d91b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84" podUID="842535c6-c97d-433b-bbab-46a4a013d91b"
Jul 16 21:20:23.473702 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:23.473677    2530 generic.go:356] "Generic (PLEG): container finished" podID="44938529-f44f-41a0-9b8b-fb33aca1870a" containerID="5273e1e2b300abbb7cc85dcdb65bfc80ce6224be6cf3cc6dedee4116ed12cdc8" exitCode=6
Jul 16 21:20:23.473803 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:23.473724    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg" event={"ID":"44938529-f44f-41a0-9b8b-fb33aca1870a","Type":"ContainerDied","Data":"5273e1e2b300abbb7cc85dcdb65bfc80ce6224be6cf3cc6dedee4116ed12cdc8"}
Jul 16 21:20:23.473803 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:23.473766    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg" event={"ID":"44938529-f44f-41a0-9b8b-fb33aca1870a","Type":"ContainerStarted","Data":"f3ae9f1cb2a41d9f71464a31f9a0a1777b3f8a0eebb5ee6b2737410781f031dc"}
Jul 16 21:20:23.473926 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:23.473913    2530 scope.go:117] "RemoveContainer" containerID="5273e1e2b300abbb7cc85dcdb65bfc80ce6224be6cf3cc6dedee4116ed12cdc8"
Jul 16 21:20:24.203136 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:24.203093    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq"]
Jul 16 21:20:24.206319 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:24.206302    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq"
Jul 16 21:20:24.213992 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:24.213968    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq"]
Jul 16 21:20:24.344920 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:24.344858    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq\" (UID: \"cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq"
Jul 16 21:20:24.445723 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:24.445698    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq\" (UID: \"cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq"
Jul 16 21:20:24.446299 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:24.446282    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq\" (UID: \"cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq"
Jul 16 21:20:24.481384 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:24.481354    2530 generic.go:356] "Generic (PLEG): container finished" podID="44938529-f44f-41a0-9b8b-fb33aca1870a" containerID="00f49ca020fb393efd3513f9714c7f8d2f4982d1a8a4b0b121f47cfb91fd6f80" exitCode=6
Jul 16 21:20:24.481772 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:24.481398    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg" event={"ID":"44938529-f44f-41a0-9b8b-fb33aca1870a","Type":"ContainerDied","Data":"00f49ca020fb393efd3513f9714c7f8d2f4982d1a8a4b0b121f47cfb91fd6f80"}
Jul 16 21:20:24.481772 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:24.481442    2530 scope.go:117] "RemoveContainer" containerID="5273e1e2b300abbb7cc85dcdb65bfc80ce6224be6cf3cc6dedee4116ed12cdc8"
Jul 16 21:20:24.481772 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:24.481642    2530 scope.go:117] "RemoveContainer" containerID="00f49ca020fb393efd3513f9714c7f8d2f4982d1a8a4b0b121f47cfb91fd6f80"
Jul 16 21:20:24.481931 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:20:24.481845    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg_odh-ai-gateway-infra(44938529-f44f-41a0-9b8b-fb33aca1870a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg" podUID="44938529-f44f-41a0-9b8b-fb33aca1870a"
Jul 16 21:20:24.517591 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:24.517572    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq"
Jul 16 21:20:24.644246 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:24.644216    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq"]
Jul 16 21:20:24.647604 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:20:24.647580    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd52f4ca_2cf3_4d59_aa4e_2fbe0bb2f72e.slice/crio-46e2f396868e32c7e52575b1530dc293be625761ba41be6e271efe45f1df9fe9 WatchSource:0}: Error finding container 46e2f396868e32c7e52575b1530dc293be625761ba41be6e271efe45f1df9fe9: Status 404 returned error can't find the container with id 46e2f396868e32c7e52575b1530dc293be625761ba41be6e271efe45f1df9fe9
Jul 16 21:20:25.487637 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:25.487546    2530 scope.go:117] "RemoveContainer" containerID="00f49ca020fb393efd3513f9714c7f8d2f4982d1a8a4b0b121f47cfb91fd6f80"
Jul 16 21:20:25.488017 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:20:25.487771    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg_odh-ai-gateway-infra(44938529-f44f-41a0-9b8b-fb33aca1870a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg" podUID="44938529-f44f-41a0-9b8b-fb33aca1870a"
Jul 16 21:20:25.488844 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:25.488820    2530 generic.go:356] "Generic (PLEG): container finished" podID="cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e" containerID="5179c1dff4842c16c0eb883ae60fe057bf4d7425ed3df6ac74d822594ae73a57" exitCode=6
Jul 16 21:20:25.488966 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:25.488854    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq" event={"ID":"cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e","Type":"ContainerDied","Data":"5179c1dff4842c16c0eb883ae60fe057bf4d7425ed3df6ac74d822594ae73a57"}
Jul 16 21:20:25.488966 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:25.488901    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq" event={"ID":"cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e","Type":"ContainerStarted","Data":"46e2f396868e32c7e52575b1530dc293be625761ba41be6e271efe45f1df9fe9"}
Jul 16 21:20:25.489134 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:25.489119    2530 scope.go:117] "RemoveContainer" containerID="5179c1dff4842c16c0eb883ae60fe057bf4d7425ed3df6ac74d822594ae73a57"
Jul 16 21:20:26.494880 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:26.494843    2530 generic.go:356] "Generic (PLEG): container finished" podID="cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e" containerID="be4dff2bb04d364f373bf23738cf1ef16530652b9c7f0da0af4805a2163c7c0e" exitCode=6
Jul 16 21:20:26.495270 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:26.494929    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq" event={"ID":"cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e","Type":"ContainerDied","Data":"be4dff2bb04d364f373bf23738cf1ef16530652b9c7f0da0af4805a2163c7c0e"}
Jul 16 21:20:26.495270 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:26.495017    2530 scope.go:117] "RemoveContainer" containerID="5179c1dff4842c16c0eb883ae60fe057bf4d7425ed3df6ac74d822594ae73a57"
Jul 16 21:20:26.495270 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:26.495241    2530 scope.go:117] "RemoveContainer" containerID="be4dff2bb04d364f373bf23738cf1ef16530652b9c7f0da0af4805a2163c7c0e"
Jul 16 21:20:26.495462 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:20:26.495440    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq_odh-ai-gateway-infra(cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq" podUID="cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e"
Jul 16 21:20:26.501850 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:26.501829    2530 scope.go:117] "RemoveContainer" containerID="19ccfcad76a732a3cae859197e93d5b2e1767a2112b26f26e68d704ec89aa1f0"
Jul 16 21:20:27.501582 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:27.501553    2530 generic.go:356] "Generic (PLEG): container finished" podID="775af836-8680-4a40-b92e-c993c69c3380" containerID="7f8cbf7a7ad3a97789631d9e83405b5428e9acfa27f003cf9399952ae3d40a62" exitCode=6
Jul 16 21:20:27.501972 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:27.501631    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s" event={"ID":"775af836-8680-4a40-b92e-c993c69c3380","Type":"ContainerDied","Data":"7f8cbf7a7ad3a97789631d9e83405b5428e9acfa27f003cf9399952ae3d40a62"}
Jul 16 21:20:27.501972 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:27.501704    2530 scope.go:117] "RemoveContainer" containerID="19ccfcad76a732a3cae859197e93d5b2e1767a2112b26f26e68d704ec89aa1f0"
Jul 16 21:20:27.501972 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:27.501919    2530 scope.go:117] "RemoveContainer" containerID="7f8cbf7a7ad3a97789631d9e83405b5428e9acfa27f003cf9399952ae3d40a62"
Jul 16 21:20:27.502180 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:20:27.502067    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s_odh-ai-gateway-infra(775af836-8680-4a40-b92e-c993c69c3380)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s" podUID="775af836-8680-4a40-b92e-c993c69c3380"
Jul 16 21:20:27.503370 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:27.503352    2530 scope.go:117] "RemoveContainer" containerID="be4dff2bb04d364f373bf23738cf1ef16530652b9c7f0da0af4805a2163c7c0e"
Jul 16 21:20:27.503537 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:20:27.503519    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq_odh-ai-gateway-infra(cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq" podUID="cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e"
Jul 16 21:20:28.530532 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.530498    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s"]
Jul 16 21:20:28.667180 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.667151    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s"
Jul 16 21:20:28.782937 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.782861    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/775af836-8680-4a40-b92e-c993c69c3380-maas-api-service-ca\") pod \"775af836-8680-4a40-b92e-c993c69c3380\" (UID: \"775af836-8680-4a40-b92e-c993c69c3380\") "
Jul 16 21:20:28.783226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.783204    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775af836-8680-4a40-b92e-c993c69c3380-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "775af836-8680-4a40-b92e-c993c69c3380" (UID: "775af836-8680-4a40-b92e-c993c69c3380"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:20:28.790633 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.790606    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c69b60-hkxpd"]
Jul 16 21:20:28.790950 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.790938    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="775af836-8680-4a40-b92e-c993c69c3380" containerName="revoke-keys"
Jul 16 21:20:28.790996 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.790951    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="775af836-8680-4a40-b92e-c993c69c3380" containerName="revoke-keys"
Jul 16 21:20:28.790996 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.790969    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="775af836-8680-4a40-b92e-c993c69c3380" containerName="revoke-keys"
Jul 16 21:20:28.790996 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.790975    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="775af836-8680-4a40-b92e-c993c69c3380" containerName="revoke-keys"
Jul 16 21:20:28.790996 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.790983    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="775af836-8680-4a40-b92e-c993c69c3380" containerName="revoke-keys"
Jul 16 21:20:28.790996 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.790988    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="775af836-8680-4a40-b92e-c993c69c3380" containerName="revoke-keys"
Jul 16 21:20:28.791184 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.791037    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="775af836-8680-4a40-b92e-c993c69c3380" containerName="revoke-keys"
Jul 16 21:20:28.791184 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.791049    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="775af836-8680-4a40-b92e-c993c69c3380" containerName="revoke-keys"
Jul 16 21:20:28.793895 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.793880    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c69b60-hkxpd"
Jul 16 21:20:28.802719 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.802694    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c69b60-hkxpd"]
Jul 16 21:20:28.884038 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.884008    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09612e85-d4bc-465f-8518-5d421687491b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c69b60-hkxpd\" (UID: \"09612e85-d4bc-465f-8518-5d421687491b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c69b60-hkxpd"
Jul 16 21:20:28.884186 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.884153    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/775af836-8680-4a40-b92e-c993c69c3380-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:20:28.985461 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.985433    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09612e85-d4bc-465f-8518-5d421687491b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c69b60-hkxpd\" (UID: \"09612e85-d4bc-465f-8518-5d421687491b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c69b60-hkxpd"
Jul 16 21:20:28.986058 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:28.986042    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09612e85-d4bc-465f-8518-5d421687491b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c69b60-hkxpd\" (UID: \"09612e85-d4bc-465f-8518-5d421687491b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c69b60-hkxpd"
Jul 16 21:20:29.105528 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:29.105461    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c69b60-hkxpd"
Jul 16 21:20:29.235650 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:29.235624    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c69b60-hkxpd"]
Jul 16 21:20:29.237996 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:20:29.237968    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09612e85_d4bc_465f_8518_5d421687491b.slice/crio-aa5892d4f50337153a0a7de57750ca25b3a47e408708ec8c447d623f2bc711bc WatchSource:0}: Error finding container aa5892d4f50337153a0a7de57750ca25b3a47e408708ec8c447d623f2bc711bc: Status 404 returned error can't find the container with id aa5892d4f50337153a0a7de57750ca25b3a47e408708ec8c447d623f2bc711bc
Jul 16 21:20:29.514343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:29.514311    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c69b60-hkxpd" event={"ID":"09612e85-d4bc-465f-8518-5d421687491b","Type":"ContainerStarted","Data":"aa5892d4f50337153a0a7de57750ca25b3a47e408708ec8c447d623f2bc711bc"}
Jul 16 21:20:29.515675 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:29.515655    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s"
Jul 16 21:20:29.515770 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:29.515654    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s" event={"ID":"775af836-8680-4a40-b92e-c993c69c3380","Type":"ContainerDied","Data":"0e083b469afaef6e51fca54f5c5a0a7da7b7c4d36597424b4a2e70658fd65ae0"}
Jul 16 21:20:29.515822 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:29.515776    2530 scope.go:117] "RemoveContainer" containerID="7f8cbf7a7ad3a97789631d9e83405b5428e9acfa27f003cf9399952ae3d40a62"
Jul 16 21:20:29.540669 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:29.540647    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s"]
Jul 16 21:20:29.546061 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:29.546040    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mtq5s"]
Jul 16 21:20:30.507876 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:30.507839    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="775af836-8680-4a40-b92e-c993c69c3380" path="/var/lib/kubelet/pods/775af836-8680-4a40-b92e-c993c69c3380/volumes"
Jul 16 21:20:30.521900 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:30.521868    2530 generic.go:356] "Generic (PLEG): container finished" podID="09612e85-d4bc-465f-8518-5d421687491b" containerID="a6bfc2fd68bdb2c3dc9937fe47691c001d8eaeb36c2b9646b26beffc05a28808" exitCode=0
Jul 16 21:20:30.522052 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:30.521917    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c69b60-hkxpd" event={"ID":"09612e85-d4bc-465f-8518-5d421687491b","Type":"ContainerDied","Data":"a6bfc2fd68bdb2c3dc9937fe47691c001d8eaeb36c2b9646b26beffc05a28808"}
Jul 16 21:20:31.658695 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:31.658656    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c69b60-hkxpd"
Jul 16 21:20:31.806913 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:31.806828    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09612e85-d4bc-465f-8518-5d421687491b-maas-api-service-ca\") pod \"09612e85-d4bc-465f-8518-5d421687491b\" (UID: \"09612e85-d4bc-465f-8518-5d421687491b\") "
Jul 16 21:20:31.807190 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:31.807171    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09612e85-d4bc-465f-8518-5d421687491b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "09612e85-d4bc-465f-8518-5d421687491b" (UID: "09612e85-d4bc-465f-8518-5d421687491b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:20:31.908032 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:31.908002    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09612e85-d4bc-465f-8518-5d421687491b-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:20:32.531763 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:32.531735    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c69b60-hkxpd"
Jul 16 21:20:32.531907 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:32.531776    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c69b60-hkxpd" event={"ID":"09612e85-d4bc-465f-8518-5d421687491b","Type":"ContainerDied","Data":"aa5892d4f50337153a0a7de57750ca25b3a47e408708ec8c447d623f2bc711bc"}
Jul 16 21:20:32.531907 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:32.531837    2530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa5892d4f50337153a0a7de57750ca25b3a47e408708ec8c447d623f2bc711bc"
Jul 16 21:20:35.502797 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:35.502731    2530 scope.go:117] "RemoveContainer" containerID="77f9ed0882bfe0573ca2603a1e343b1f878bbff57928a87dfa16f354bc76b198"
Jul 16 21:20:36.551486 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:36.551450    2530 generic.go:356] "Generic (PLEG): container finished" podID="842535c6-c97d-433b-bbab-46a4a013d91b" containerID="07a5c264f98a8b45413bb7bc28ce82364a6a44916969bd6396da4f5154c79180" exitCode=6
Jul 16 21:20:36.551957 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:36.551524    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84" event={"ID":"842535c6-c97d-433b-bbab-46a4a013d91b","Type":"ContainerDied","Data":"07a5c264f98a8b45413bb7bc28ce82364a6a44916969bd6396da4f5154c79180"}
Jul 16 21:20:36.551957 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:36.551583    2530 scope.go:117] "RemoveContainer" containerID="77f9ed0882bfe0573ca2603a1e343b1f878bbff57928a87dfa16f354bc76b198"
Jul 16 21:20:36.551957 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:36.551872    2530 scope.go:117] "RemoveContainer" containerID="07a5c264f98a8b45413bb7bc28ce82364a6a44916969bd6396da4f5154c79180"
Jul 16 21:20:36.552163 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:20:36.552073    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84_odh-ai-gateway-infra(842535c6-c97d-433b-bbab-46a4a013d91b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84" podUID="842535c6-c97d-433b-bbab-46a4a013d91b"
Jul 16 21:20:37.578874 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:37.578834    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84"]
Jul 16 21:20:37.715886 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:37.715862    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84"
Jul 16 21:20:37.857402 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:37.857301    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/842535c6-c97d-433b-bbab-46a4a013d91b-maas-api-service-ca\") pod \"842535c6-c97d-433b-bbab-46a4a013d91b\" (UID: \"842535c6-c97d-433b-bbab-46a4a013d91b\") "
Jul 16 21:20:37.857683 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:37.857661    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842535c6-c97d-433b-bbab-46a4a013d91b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "842535c6-c97d-433b-bbab-46a4a013d91b" (UID: "842535c6-c97d-433b-bbab-46a4a013d91b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:20:37.958403 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:37.958375    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/842535c6-c97d-433b-bbab-46a4a013d91b-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:20:38.564543 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:38.564517    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84"
Jul 16 21:20:38.564543 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:38.564527    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84" event={"ID":"842535c6-c97d-433b-bbab-46a4a013d91b","Type":"ContainerDied","Data":"17424754b5bf0a15214726cebdac66cde8e7a6eba7db1aa5833d724d750fd004"}
Jul 16 21:20:38.564699 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:38.564585    2530 scope.go:117] "RemoveContainer" containerID="07a5c264f98a8b45413bb7bc28ce82364a6a44916969bd6396da4f5154c79180"
Jul 16 21:20:38.580649 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:38.580626    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84"]
Jul 16 21:20:38.584499 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:38.584478    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-vmd84"]
Jul 16 21:20:39.217242 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.217212    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"]
Jul 16 21:20:39.217529 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.217491    2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7" podUID="38836870-ea92-4c5d-96ac-29ca53c88d3c" containerName="maas-api" containerID="cri-o://19d79218572b9ad30352a3934d9bde66aabba0092e442746e1ce6b696552cd97" gracePeriod=30
Jul 16 21:20:39.473243 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.473189    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"
Jul 16 21:20:39.568872 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.568846    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5792\" (UniqueName: \"kubernetes.io/projected/38836870-ea92-4c5d-96ac-29ca53c88d3c-kube-api-access-s5792\") pod \"38836870-ea92-4c5d-96ac-29ca53c88d3c\" (UID: \"38836870-ea92-4c5d-96ac-29ca53c88d3c\") "
Jul 16 21:20:39.569031 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.568908    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38836870-ea92-4c5d-96ac-29ca53c88d3c-maas-api-tls\") pod \"38836870-ea92-4c5d-96ac-29ca53c88d3c\" (UID: \"38836870-ea92-4c5d-96ac-29ca53c88d3c\") "
Jul 16 21:20:39.569031 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.569015    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38836870-ea92-4c5d-96ac-29ca53c88d3c-openshift-service-ca\") pod \"38836870-ea92-4c5d-96ac-29ca53c88d3c\" (UID: \"38836870-ea92-4c5d-96ac-29ca53c88d3c\") "
Jul 16 21:20:39.569377 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.569350    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38836870-ea92-4c5d-96ac-29ca53c88d3c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "38836870-ea92-4c5d-96ac-29ca53c88d3c" (UID: "38836870-ea92-4c5d-96ac-29ca53c88d3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:20:39.571361 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.571324    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38836870-ea92-4c5d-96ac-29ca53c88d3c-kube-api-access-s5792" (OuterVolumeSpecName: "kube-api-access-s5792") pod "38836870-ea92-4c5d-96ac-29ca53c88d3c" (UID: "38836870-ea92-4c5d-96ac-29ca53c88d3c"). InnerVolumeSpecName "kube-api-access-s5792". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:20:39.571461 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.571386    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38836870-ea92-4c5d-96ac-29ca53c88d3c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "38836870-ea92-4c5d-96ac-29ca53c88d3c" (UID: "38836870-ea92-4c5d-96ac-29ca53c88d3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:20:39.571694 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.571674    2530 generic.go:356] "Generic (PLEG): container finished" podID="38836870-ea92-4c5d-96ac-29ca53c88d3c" containerID="19d79218572b9ad30352a3934d9bde66aabba0092e442746e1ce6b696552cd97" exitCode=0
Jul 16 21:20:39.571764 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.571727    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"
Jul 16 21:20:39.571819 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.571759    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7" event={"ID":"38836870-ea92-4c5d-96ac-29ca53c88d3c","Type":"ContainerDied","Data":"19d79218572b9ad30352a3934d9bde66aabba0092e442746e1ce6b696552cd97"}
Jul 16 21:20:39.571870 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.571818    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7" event={"ID":"38836870-ea92-4c5d-96ac-29ca53c88d3c","Type":"ContainerDied","Data":"33884dab88b4bab5c0fcbdefaf6817cc42b987ed1a06b12c62df743046c8d2a0"}
Jul 16 21:20:39.571870 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.571850    2530 scope.go:117] "RemoveContainer" containerID="19d79218572b9ad30352a3934d9bde66aabba0092e442746e1ce6b696552cd97"
Jul 16 21:20:39.589697 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.589680    2530 scope.go:117] "RemoveContainer" containerID="19d79218572b9ad30352a3934d9bde66aabba0092e442746e1ce6b696552cd97"
Jul 16 21:20:39.589966 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:20:39.589924    2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19d79218572b9ad30352a3934d9bde66aabba0092e442746e1ce6b696552cd97\": container with ID starting with 19d79218572b9ad30352a3934d9bde66aabba0092e442746e1ce6b696552cd97 not found: ID does not exist" containerID="19d79218572b9ad30352a3934d9bde66aabba0092e442746e1ce6b696552cd97"
Jul 16 21:20:39.589966 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.589949    2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19d79218572b9ad30352a3934d9bde66aabba0092e442746e1ce6b696552cd97"} err="failed to get container status \"19d79218572b9ad30352a3934d9bde66aabba0092e442746e1ce6b696552cd97\": rpc error: code = NotFound desc = could not find container \"19d79218572b9ad30352a3934d9bde66aabba0092e442746e1ce6b696552cd97\": container with ID starting with 19d79218572b9ad30352a3934d9bde66aabba0092e442746e1ce6b696552cd97 not found: ID does not exist"
Jul 16 21:20:39.599989 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.599969    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"]
Jul 16 21:20:39.603817 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.603796    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c69b60-794bf6877b-zj6n7"]
Jul 16 21:20:39.670692 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.670668    2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5792\" (UniqueName: \"kubernetes.io/projected/38836870-ea92-4c5d-96ac-29ca53c88d3c-kube-api-access-s5792\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:20:39.670692 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.670692    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38836870-ea92-4c5d-96ac-29ca53c88d3c-maas-api-tls\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:20:39.670856 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:39.670705    2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38836870-ea92-4c5d-96ac-29ca53c88d3c-openshift-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:20:40.502980 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:40.502950    2530 scope.go:117] "RemoveContainer" containerID="00f49ca020fb393efd3513f9714c7f8d2f4982d1a8a4b0b121f47cfb91fd6f80"
Jul 16 21:20:40.509086 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:40.509067    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38836870-ea92-4c5d-96ac-29ca53c88d3c" path="/var/lib/kubelet/pods/38836870-ea92-4c5d-96ac-29ca53c88d3c/volumes"
Jul 16 21:20:40.509417 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:40.509395    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="842535c6-c97d-433b-bbab-46a4a013d91b" path="/var/lib/kubelet/pods/842535c6-c97d-433b-bbab-46a4a013d91b/volumes"
Jul 16 21:20:41.584453 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:41.584421    2530 generic.go:356] "Generic (PLEG): container finished" podID="44938529-f44f-41a0-9b8b-fb33aca1870a" containerID="31bf4d7e81dd85b7697354b8eb18be07f9926e79ddc0ef0d0af450bb1f4ded55" exitCode=6
Jul 16 21:20:41.584839 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:41.584498    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg" event={"ID":"44938529-f44f-41a0-9b8b-fb33aca1870a","Type":"ContainerDied","Data":"31bf4d7e81dd85b7697354b8eb18be07f9926e79ddc0ef0d0af450bb1f4ded55"}
Jul 16 21:20:41.584839 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:41.584561    2530 scope.go:117] "RemoveContainer" containerID="00f49ca020fb393efd3513f9714c7f8d2f4982d1a8a4b0b121f47cfb91fd6f80"
Jul 16 21:20:41.584930 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:41.584877    2530 scope.go:117] "RemoveContainer" containerID="31bf4d7e81dd85b7697354b8eb18be07f9926e79ddc0ef0d0af450bb1f4ded55"
Jul 16 21:20:41.585129 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:20:41.585084    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg_odh-ai-gateway-infra(44938529-f44f-41a0-9b8b-fb33aca1870a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg" podUID="44938529-f44f-41a0-9b8b-fb33aca1870a"
Jul 16 21:20:42.505186 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:42.505158    2530 scope.go:117] "RemoveContainer" containerID="be4dff2bb04d364f373bf23738cf1ef16530652b9c7f0da0af4805a2163c7c0e"
Jul 16 21:20:42.612577 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:42.612548    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg"]
Jul 16 21:20:42.753284 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:42.753262    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg"
Jul 16 21:20:42.896742 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:42.896721    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44938529-f44f-41a0-9b8b-fb33aca1870a-maas-api-service-ca\") pod \"44938529-f44f-41a0-9b8b-fb33aca1870a\" (UID: \"44938529-f44f-41a0-9b8b-fb33aca1870a\") "
Jul 16 21:20:42.897032 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:42.897011    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44938529-f44f-41a0-9b8b-fb33aca1870a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "44938529-f44f-41a0-9b8b-fb33aca1870a" (UID: "44938529-f44f-41a0-9b8b-fb33aca1870a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:20:42.998187 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:42.998163    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44938529-f44f-41a0-9b8b-fb33aca1870a-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:20:43.595387 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:43.595359    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg"
Jul 16 21:20:43.595532 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:43.595397    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg" event={"ID":"44938529-f44f-41a0-9b8b-fb33aca1870a","Type":"ContainerDied","Data":"f3ae9f1cb2a41d9f71464a31f9a0a1777b3f8a0eebb5ee6b2737410781f031dc"}
Jul 16 21:20:43.595532 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:43.595463    2530 scope.go:117] "RemoveContainer" containerID="31bf4d7e81dd85b7697354b8eb18be07f9926e79ddc0ef0d0af450bb1f4ded55"
Jul 16 21:20:43.597477 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:43.597453    2530 generic.go:356] "Generic (PLEG): container finished" podID="cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e" containerID="9958ca6a18b2809149542cab47967be341c8dabfd17875f4d310c72100644e45" exitCode=6
Jul 16 21:20:43.597610 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:43.597488    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq" event={"ID":"cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e","Type":"ContainerDied","Data":"9958ca6a18b2809149542cab47967be341c8dabfd17875f4d310c72100644e45"}
Jul 16 21:20:43.597761 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:43.597745    2530 scope.go:117] "RemoveContainer" containerID="9958ca6a18b2809149542cab47967be341c8dabfd17875f4d310c72100644e45"
Jul 16 21:20:43.597924 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:20:43.597907    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq_odh-ai-gateway-infra(cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq" podUID="cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e"
Jul 16 21:20:43.603988 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:43.603967    2530 scope.go:117] "RemoveContainer" containerID="be4dff2bb04d364f373bf23738cf1ef16530652b9c7f0da0af4805a2163c7c0e"
Jul 16 21:20:43.628243 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:43.628191    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg"]
Jul 16 21:20:43.630131 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:43.630084    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-srxsg"]
Jul 16 21:20:44.507618 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:44.507586    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44938529-f44f-41a0-9b8b-fb33aca1870a" path="/var/lib/kubelet/pods/44938529-f44f-41a0-9b8b-fb33aca1870a/volumes"
Jul 16 21:20:44.622665 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:44.622633    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq"]
Jul 16 21:20:44.754162 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:44.754137    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq"
Jul 16 21:20:44.913363 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:44.913285    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e-maas-api-service-ca\") pod \"cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e\" (UID: \"cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e\") "
Jul 16 21:20:44.913606 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:44.913587    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e" (UID: "cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:20:45.013783 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:45.013752    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:20:45.609171 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:45.609133    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq" event={"ID":"cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e","Type":"ContainerDied","Data":"46e2f396868e32c7e52575b1530dc293be625761ba41be6e271efe45f1df9fe9"}
Jul 16 21:20:45.609171 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:45.609152    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq"
Jul 16 21:20:45.609447 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:45.609193    2530 scope.go:117] "RemoveContainer" containerID="9958ca6a18b2809149542cab47967be341c8dabfd17875f4d310c72100644e45"
Jul 16 21:20:45.636450 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:45.636424    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq"]
Jul 16 21:20:45.643803 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:45.643780    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-k74mq"]
Jul 16 21:20:46.507121 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:46.507067    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e" path="/var/lib/kubelet/pods/cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e/volumes"
Jul 16 21:20:54.957865 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.957825    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72cda6-2zsfg"]
Jul 16 21:20:54.958272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958166    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44938529-f44f-41a0-9b8b-fb33aca1870a" containerName="revoke-keys"
Jul 16 21:20:54.958272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958178    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="44938529-f44f-41a0-9b8b-fb33aca1870a" containerName="revoke-keys"
Jul 16 21:20:54.958272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958189    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="842535c6-c97d-433b-bbab-46a4a013d91b" containerName="revoke-keys"
Jul 16 21:20:54.958272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958194    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="842535c6-c97d-433b-bbab-46a4a013d91b" containerName="revoke-keys"
Jul 16 21:20:54.958272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958203    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e" containerName="revoke-keys"
Jul 16 21:20:54.958272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958208    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e" containerName="revoke-keys"
Jul 16 21:20:54.958272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958218    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44938529-f44f-41a0-9b8b-fb33aca1870a" containerName="revoke-keys"
Jul 16 21:20:54.958272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958223    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="44938529-f44f-41a0-9b8b-fb33aca1870a" containerName="revoke-keys"
Jul 16 21:20:54.958272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958229    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44938529-f44f-41a0-9b8b-fb33aca1870a" containerName="revoke-keys"
Jul 16 21:20:54.958272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958235    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="44938529-f44f-41a0-9b8b-fb33aca1870a" containerName="revoke-keys"
Jul 16 21:20:54.958272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958244    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="842535c6-c97d-433b-bbab-46a4a013d91b" containerName="revoke-keys"
Jul 16 21:20:54.958272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958248    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="842535c6-c97d-433b-bbab-46a4a013d91b" containerName="revoke-keys"
Jul 16 21:20:54.958272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958255    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="842535c6-c97d-433b-bbab-46a4a013d91b" containerName="revoke-keys"
Jul 16 21:20:54.958272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958260    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="842535c6-c97d-433b-bbab-46a4a013d91b" containerName="revoke-keys"
Jul 16 21:20:54.958272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958267    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38836870-ea92-4c5d-96ac-29ca53c88d3c" containerName="maas-api"
Jul 16 21:20:54.958272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958272    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="38836870-ea92-4c5d-96ac-29ca53c88d3c" containerName="maas-api"
Jul 16 21:20:54.958272 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958279    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e" containerName="revoke-keys"
Jul 16 21:20:54.958786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958284    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e" containerName="revoke-keys"
Jul 16 21:20:54.958786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958291    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09612e85-d4bc-465f-8518-5d421687491b" containerName="revoke-keys"
Jul 16 21:20:54.958786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958296    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="09612e85-d4bc-465f-8518-5d421687491b" containerName="revoke-keys"
Jul 16 21:20:54.958786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958345    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="842535c6-c97d-433b-bbab-46a4a013d91b" containerName="revoke-keys"
Jul 16 21:20:54.958786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958351    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e" containerName="revoke-keys"
Jul 16 21:20:54.958786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958359    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="44938529-f44f-41a0-9b8b-fb33aca1870a" containerName="revoke-keys"
Jul 16 21:20:54.958786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958365    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="38836870-ea92-4c5d-96ac-29ca53c88d3c" containerName="maas-api"
Jul 16 21:20:54.958786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958371    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="09612e85-d4bc-465f-8518-5d421687491b" containerName="revoke-keys"
Jul 16 21:20:54.958786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958377    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e" containerName="revoke-keys"
Jul 16 21:20:54.958786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958382    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="842535c6-c97d-433b-bbab-46a4a013d91b" containerName="revoke-keys"
Jul 16 21:20:54.958786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958386    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="44938529-f44f-41a0-9b8b-fb33aca1870a" containerName="revoke-keys"
Jul 16 21:20:54.958786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958390    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="842535c6-c97d-433b-bbab-46a4a013d91b" containerName="revoke-keys"
Jul 16 21:20:54.958786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.958395    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="775af836-8680-4a40-b92e-c993c69c3380" containerName="revoke-keys"
Jul 16 21:20:54.961280 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.961264    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72cda6-2zsfg"
Jul 16 21:20:54.964149 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.964130    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:20:54.970683 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.970652    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72cda6-2zsfg"]
Jul 16 21:20:54.995034 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:54.995011    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b5f5425-622b-41a3-aa8a-68abe4efe80e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-72cda6-2zsfg\" (UID: \"3b5f5425-622b-41a3-aa8a-68abe4efe80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72cda6-2zsfg"
Jul 16 21:20:55.095599 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:55.095566    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b5f5425-622b-41a3-aa8a-68abe4efe80e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-72cda6-2zsfg\" (UID: \"3b5f5425-622b-41a3-aa8a-68abe4efe80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72cda6-2zsfg"
Jul 16 21:20:55.096222 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:55.096205    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b5f5425-622b-41a3-aa8a-68abe4efe80e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-72cda6-2zsfg\" (UID: \"3b5f5425-622b-41a3-aa8a-68abe4efe80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72cda6-2zsfg"
Jul 16 21:20:55.272152 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:55.272046    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72cda6-2zsfg"
Jul 16 21:20:55.419898 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:55.419865    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72cda6-2zsfg"]
Jul 16 21:20:55.423572 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:20:55.423541    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5f5425_622b_41a3_aa8a_68abe4efe80e.slice/crio-680f13745859c1f035e8090e5b3a6f7a49af1d225e2e5cec280439961d8fdefa WatchSource:0}: Error finding container 680f13745859c1f035e8090e5b3a6f7a49af1d225e2e5cec280439961d8fdefa: Status 404 returned error can't find the container with id 680f13745859c1f035e8090e5b3a6f7a49af1d225e2e5cec280439961d8fdefa
Jul 16 21:20:55.656604 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:55.656569    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72cda6-2zsfg" event={"ID":"3b5f5425-622b-41a3-aa8a-68abe4efe80e","Type":"ContainerStarted","Data":"680f13745859c1f035e8090e5b3a6f7a49af1d225e2e5cec280439961d8fdefa"}
Jul 16 21:20:56.662554 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:56.662517    2530 generic.go:356] "Generic (PLEG): container finished" podID="3b5f5425-622b-41a3-aa8a-68abe4efe80e" containerID="5fb42ea7219ebbf58ba02db83c28545882f84e30b673d9eb2bb86b6bc0eaf096" exitCode=0
Jul 16 21:20:56.662932 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:56.662565    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72cda6-2zsfg" event={"ID":"3b5f5425-622b-41a3-aa8a-68abe4efe80e","Type":"ContainerDied","Data":"5fb42ea7219ebbf58ba02db83c28545882f84e30b673d9eb2bb86b6bc0eaf096"}
Jul 16 21:20:57.811421 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:57.811396    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72cda6-2zsfg"
Jul 16 21:20:57.921464 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:57.921386    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b5f5425-622b-41a3-aa8a-68abe4efe80e-maas-api-service-ca\") pod \"3b5f5425-622b-41a3-aa8a-68abe4efe80e\" (UID: \"3b5f5425-622b-41a3-aa8a-68abe4efe80e\") "
Jul 16 21:20:57.921686 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:57.921666    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5f5425-622b-41a3-aa8a-68abe4efe80e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b5f5425-622b-41a3-aa8a-68abe4efe80e" (UID: "3b5f5425-622b-41a3-aa8a-68abe4efe80e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:20:58.022287 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:58.022253    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b5f5425-622b-41a3-aa8a-68abe4efe80e-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:20:58.672736 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:58.672706    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72cda6-2zsfg"
Jul 16 21:20:58.672905 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:58.672745    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72cda6-2zsfg" event={"ID":"3b5f5425-622b-41a3-aa8a-68abe4efe80e","Type":"ContainerDied","Data":"680f13745859c1f035e8090e5b3a6f7a49af1d225e2e5cec280439961d8fdefa"}
Jul 16 21:20:58.672905 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:20:58.672790    2530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="680f13745859c1f035e8090e5b3a6f7a49af1d225e2e5cec280439961d8fdefa"
Jul 16 21:21:02.061605 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:02.061566    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd"]
Jul 16 21:21:02.061989 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:02.061882    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b5f5425-622b-41a3-aa8a-68abe4efe80e" containerName="revoke-keys"
Jul 16 21:21:02.061989 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:02.061892    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b5f5425-622b-41a3-aa8a-68abe4efe80e" containerName="revoke-keys"
Jul 16 21:21:02.061989 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:02.061904    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e" containerName="revoke-keys"
Jul 16 21:21:02.061989 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:02.061910    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e" containerName="revoke-keys"
Jul 16 21:21:02.061989 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:02.061957    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd52f4ca-2cf3-4d59-aa4e-2fbe0bb2f72e" containerName="revoke-keys"
Jul 16 21:21:02.061989 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:02.061965    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b5f5425-622b-41a3-aa8a-68abe4efe80e" containerName="revoke-keys"
Jul 16 21:21:02.061989 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:02.061975    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="44938529-f44f-41a0-9b8b-fb33aca1870a" containerName="revoke-keys"
Jul 16 21:21:02.064954 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:02.064937    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd"
Jul 16 21:21:02.067759 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:02.067742    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:21:02.074361 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:02.074336    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd"]
Jul 16 21:21:02.158854 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:02.158821    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bda1e1d-6199-4d10-9342-44f8828bc096-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-njptd\" (UID: \"9bda1e1d-6199-4d10-9342-44f8828bc096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd"
Jul 16 21:21:02.259467 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:02.259433    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bda1e1d-6199-4d10-9342-44f8828bc096-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-njptd\" (UID: \"9bda1e1d-6199-4d10-9342-44f8828bc096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd"
Jul 16 21:21:02.260087 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:02.260066    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bda1e1d-6199-4d10-9342-44f8828bc096-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-njptd\" (UID: \"9bda1e1d-6199-4d10-9342-44f8828bc096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd"
Jul 16 21:21:02.376513 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:02.376431    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd"
Jul 16 21:21:02.508662 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:02.508637    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd"]
Jul 16 21:21:02.510204 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:21:02.510179    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bda1e1d_6199_4d10_9342_44f8828bc096.slice/crio-136008a042de48162c64be7f82ff68b259984cba57ba1d0f54332e94718c6deb WatchSource:0}: Error finding container 136008a042de48162c64be7f82ff68b259984cba57ba1d0f54332e94718c6deb: Status 404 returned error can't find the container with id 136008a042de48162c64be7f82ff68b259984cba57ba1d0f54332e94718c6deb
Jul 16 21:21:02.691016 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:02.690931    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd" event={"ID":"9bda1e1d-6199-4d10-9342-44f8828bc096","Type":"ContainerStarted","Data":"136008a042de48162c64be7f82ff68b259984cba57ba1d0f54332e94718c6deb"}
Jul 16 21:21:03.696798 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:03.696761    2530 generic.go:356] "Generic (PLEG): container finished" podID="9bda1e1d-6199-4d10-9342-44f8828bc096" containerID="9c4e0bd61d4eddbd529a215b1c3dfd4539a39c0e4d7e35649316b0d8693f35cc" exitCode=6
Jul 16 21:21:03.697313 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:03.696845    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd" event={"ID":"9bda1e1d-6199-4d10-9342-44f8828bc096","Type":"ContainerDied","Data":"9c4e0bd61d4eddbd529a215b1c3dfd4539a39c0e4d7e35649316b0d8693f35cc"}
Jul 16 21:21:03.697313 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:03.697042    2530 scope.go:117] "RemoveContainer" containerID="9c4e0bd61d4eddbd529a215b1c3dfd4539a39c0e4d7e35649316b0d8693f35cc"
Jul 16 21:21:04.702719 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:04.702634    2530 generic.go:356] "Generic (PLEG): container finished" podID="9bda1e1d-6199-4d10-9342-44f8828bc096" containerID="5de026785593f418cd36a173263553928d574bf946a80c1604842ae115d55f21" exitCode=6
Jul 16 21:21:04.702719 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:04.702702    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd" event={"ID":"9bda1e1d-6199-4d10-9342-44f8828bc096","Type":"ContainerDied","Data":"5de026785593f418cd36a173263553928d574bf946a80c1604842ae115d55f21"}
Jul 16 21:21:04.703173 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:04.702762    2530 scope.go:117] "RemoveContainer" containerID="9c4e0bd61d4eddbd529a215b1c3dfd4539a39c0e4d7e35649316b0d8693f35cc"
Jul 16 21:21:04.703173 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:04.702934    2530 scope.go:117] "RemoveContainer" containerID="5de026785593f418cd36a173263553928d574bf946a80c1604842ae115d55f21"
Jul 16 21:21:04.703251 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:04.703168    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-njptd_odh-ai-gateway-infra(9bda1e1d-6199-4d10-9342-44f8828bc096)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd" podUID="9bda1e1d-6199-4d10-9342-44f8828bc096"
Jul 16 21:21:05.396702 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:05.396663    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"]
Jul 16 21:21:05.396959 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:05.396927    2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh" podUID="b163dc4b-6403-4c32-8bba-380690b93ad1" containerName="maas-api" containerID="cri-o://1b85e265d6c10638acf8be888d7d96249eefc80ec75ee607d86c13d84c74bc34" gracePeriod=30
Jul 16 21:21:05.708424 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:05.708401    2530 scope.go:117] "RemoveContainer" containerID="5de026785593f418cd36a173263553928d574bf946a80c1604842ae115d55f21"
Jul 16 21:21:05.708797 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:05.708553    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-njptd_odh-ai-gateway-infra(9bda1e1d-6199-4d10-9342-44f8828bc096)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd" podUID="9bda1e1d-6199-4d10-9342-44f8828bc096"
Jul 16 21:21:06.645113 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.645081    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"
Jul 16 21:21:06.695301 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.695274    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjt6t\" (UniqueName: \"kubernetes.io/projected/b163dc4b-6403-4c32-8bba-380690b93ad1-kube-api-access-qjt6t\") pod \"b163dc4b-6403-4c32-8bba-380690b93ad1\" (UID: \"b163dc4b-6403-4c32-8bba-380690b93ad1\") "
Jul 16 21:21:06.695420 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.695356    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b163dc4b-6403-4c32-8bba-380690b93ad1-maas-api-tls\") pod \"b163dc4b-6403-4c32-8bba-380690b93ad1\" (UID: \"b163dc4b-6403-4c32-8bba-380690b93ad1\") "
Jul 16 21:21:06.695420 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.695401    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b163dc4b-6403-4c32-8bba-380690b93ad1-openshift-service-ca\") pod \"b163dc4b-6403-4c32-8bba-380690b93ad1\" (UID: \"b163dc4b-6403-4c32-8bba-380690b93ad1\") "
Jul 16 21:21:06.695773 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.695748    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b163dc4b-6403-4c32-8bba-380690b93ad1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b163dc4b-6403-4c32-8bba-380690b93ad1" (UID: "b163dc4b-6403-4c32-8bba-380690b93ad1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:21:06.697411 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.697389    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b163dc4b-6403-4c32-8bba-380690b93ad1-kube-api-access-qjt6t" (OuterVolumeSpecName: "kube-api-access-qjt6t") pod "b163dc4b-6403-4c32-8bba-380690b93ad1" (UID: "b163dc4b-6403-4c32-8bba-380690b93ad1"). InnerVolumeSpecName "kube-api-access-qjt6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:21:06.697491 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.697422    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b163dc4b-6403-4c32-8bba-380690b93ad1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b163dc4b-6403-4c32-8bba-380690b93ad1" (UID: "b163dc4b-6403-4c32-8bba-380690b93ad1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:21:06.713556 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.713531    2530 generic.go:356] "Generic (PLEG): container finished" podID="b163dc4b-6403-4c32-8bba-380690b93ad1" containerID="1b85e265d6c10638acf8be888d7d96249eefc80ec75ee607d86c13d84c74bc34" exitCode=0
Jul 16 21:21:06.713840 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.713578    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"
Jul 16 21:21:06.713840 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.713616    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh" event={"ID":"b163dc4b-6403-4c32-8bba-380690b93ad1","Type":"ContainerDied","Data":"1b85e265d6c10638acf8be888d7d96249eefc80ec75ee607d86c13d84c74bc34"}
Jul 16 21:21:06.713840 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.713674    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh" event={"ID":"b163dc4b-6403-4c32-8bba-380690b93ad1","Type":"ContainerDied","Data":"d20e264198eb13a8a92e56df20a56debbe4d1e38fe86a37b9e943e374525eab0"}
Jul 16 21:21:06.713840 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.713701    2530 scope.go:117] "RemoveContainer" containerID="1b85e265d6c10638acf8be888d7d96249eefc80ec75ee607d86c13d84c74bc34"
Jul 16 21:21:06.721996 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.721979    2530 scope.go:117] "RemoveContainer" containerID="1b85e265d6c10638acf8be888d7d96249eefc80ec75ee607d86c13d84c74bc34"
Jul 16 21:21:06.722261 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:06.722243    2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b85e265d6c10638acf8be888d7d96249eefc80ec75ee607d86c13d84c74bc34\": container with ID starting with 1b85e265d6c10638acf8be888d7d96249eefc80ec75ee607d86c13d84c74bc34 not found: ID does not exist" containerID="1b85e265d6c10638acf8be888d7d96249eefc80ec75ee607d86c13d84c74bc34"
Jul 16 21:21:06.722326 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.722270    2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b85e265d6c10638acf8be888d7d96249eefc80ec75ee607d86c13d84c74bc34"} err="failed to get container status \"1b85e265d6c10638acf8be888d7d96249eefc80ec75ee607d86c13d84c74bc34\": rpc error: code = NotFound desc = could not find container \"1b85e265d6c10638acf8be888d7d96249eefc80ec75ee607d86c13d84c74bc34\": container with ID starting with 1b85e265d6c10638acf8be888d7d96249eefc80ec75ee607d86c13d84c74bc34 not found: ID does not exist"
Jul 16 21:21:06.734432 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.734384    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"]
Jul 16 21:21:06.737753 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.737735    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-72cda6-64c8d78866-smljh"]
Jul 16 21:21:06.796595 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.796572    2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjt6t\" (UniqueName: \"kubernetes.io/projected/b163dc4b-6403-4c32-8bba-380690b93ad1-kube-api-access-qjt6t\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:21:06.796595 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.796593    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b163dc4b-6403-4c32-8bba-380690b93ad1-maas-api-tls\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:21:06.796723 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:06.796602    2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b163dc4b-6403-4c32-8bba-380690b93ad1-openshift-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:21:08.507373 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:08.507341    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b163dc4b-6403-4c32-8bba-380690b93ad1" path="/var/lib/kubelet/pods/b163dc4b-6403-4c32-8bba-380690b93ad1/volumes"
Jul 16 21:21:09.479519 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:09.479481    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n"]
Jul 16 21:21:09.479831 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:09.479819    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b163dc4b-6403-4c32-8bba-380690b93ad1" containerName="maas-api"
Jul 16 21:21:09.479876 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:09.479833    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b163dc4b-6403-4c32-8bba-380690b93ad1" containerName="maas-api"
Jul 16 21:21:09.479959 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:09.479880    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b163dc4b-6403-4c32-8bba-380690b93ad1" containerName="maas-api"
Jul 16 21:21:09.484229 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:09.484212    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n"
Jul 16 21:21:09.490506 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:09.490483    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n"]
Jul 16 21:21:09.518172 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:09.518146    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/894e1873-d7c9-44c2-a9bb-1603f9cdf989-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n\" (UID: \"894e1873-d7c9-44c2-a9bb-1603f9cdf989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n"
Jul 16 21:21:09.619440 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:09.619408    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/894e1873-d7c9-44c2-a9bb-1603f9cdf989-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n\" (UID: \"894e1873-d7c9-44c2-a9bb-1603f9cdf989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n"
Jul 16 21:21:09.619937 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:09.619918    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/894e1873-d7c9-44c2-a9bb-1603f9cdf989-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n\" (UID: \"894e1873-d7c9-44c2-a9bb-1603f9cdf989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n"
Jul 16 21:21:09.795816 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:09.795752    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n"
Jul 16 21:21:10.131732 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:10.131707    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n"]
Jul 16 21:21:10.133626 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:21:10.133597    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894e1873_d7c9_44c2_a9bb_1603f9cdf989.slice/crio-aae29d4a3897670616af37fa24c900c70d7dfd1e217f14d5e14b09c9d625a473 WatchSource:0}: Error finding container aae29d4a3897670616af37fa24c900c70d7dfd1e217f14d5e14b09c9d625a473: Status 404 returned error can't find the container with id aae29d4a3897670616af37fa24c900c70d7dfd1e217f14d5e14b09c9d625a473
Jul 16 21:21:10.734252 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:10.734224    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n" event={"ID":"894e1873-d7c9-44c2-a9bb-1603f9cdf989","Type":"ContainerStarted","Data":"b0737d7deac5a5717ab3d7f2eebbdd6be9ae53db31fc351321e3e1f23fc6ff36"}
Jul 16 21:21:10.734583 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:10.734273    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n" event={"ID":"894e1873-d7c9-44c2-a9bb-1603f9cdf989","Type":"ContainerStarted","Data":"aae29d4a3897670616af37fa24c900c70d7dfd1e217f14d5e14b09c9d625a473"}
Jul 16 21:21:10.734583 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:10.734443    2530 scope.go:117] "RemoveContainer" containerID="b0737d7deac5a5717ab3d7f2eebbdd6be9ae53db31fc351321e3e1f23fc6ff36"
Jul 16 21:21:11.739948 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:11.739910    2530 generic.go:356] "Generic (PLEG): container finished" podID="894e1873-d7c9-44c2-a9bb-1603f9cdf989" containerID="b0737d7deac5a5717ab3d7f2eebbdd6be9ae53db31fc351321e3e1f23fc6ff36" exitCode=6
Jul 16 21:21:11.739948 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:11.739942    2530 generic.go:356] "Generic (PLEG): container finished" podID="894e1873-d7c9-44c2-a9bb-1603f9cdf989" containerID="1ad49df9b2c4bc588058bb0ba9e153785cb32ee77b6de531ca662c78e43ee69a" exitCode=6
Jul 16 21:21:11.740510 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:11.739988    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n" event={"ID":"894e1873-d7c9-44c2-a9bb-1603f9cdf989","Type":"ContainerDied","Data":"b0737d7deac5a5717ab3d7f2eebbdd6be9ae53db31fc351321e3e1f23fc6ff36"}
Jul 16 21:21:11.740510 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:11.740052    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n" event={"ID":"894e1873-d7c9-44c2-a9bb-1603f9cdf989","Type":"ContainerDied","Data":"1ad49df9b2c4bc588058bb0ba9e153785cb32ee77b6de531ca662c78e43ee69a"}
Jul 16 21:21:11.740510 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:11.740080    2530 scope.go:117] "RemoveContainer" containerID="b0737d7deac5a5717ab3d7f2eebbdd6be9ae53db31fc351321e3e1f23fc6ff36"
Jul 16 21:21:11.740510 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:11.740248    2530 scope.go:117] "RemoveContainer" containerID="1ad49df9b2c4bc588058bb0ba9e153785cb32ee77b6de531ca662c78e43ee69a"
Jul 16 21:21:11.740510 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:11.740490    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n_odh-ai-gateway-infra(894e1873-d7c9-44c2-a9bb-1603f9cdf989)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n" podUID="894e1873-d7c9-44c2-a9bb-1603f9cdf989"
Jul 16 21:21:11.752933 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:11.752915    2530 scope.go:117] "RemoveContainer" containerID="b0737d7deac5a5717ab3d7f2eebbdd6be9ae53db31fc351321e3e1f23fc6ff36"
Jul 16 21:21:11.753215 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:11.753195    2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0737d7deac5a5717ab3d7f2eebbdd6be9ae53db31fc351321e3e1f23fc6ff36\": container with ID starting with b0737d7deac5a5717ab3d7f2eebbdd6be9ae53db31fc351321e3e1f23fc6ff36 not found: ID does not exist" containerID="b0737d7deac5a5717ab3d7f2eebbdd6be9ae53db31fc351321e3e1f23fc6ff36"
Jul 16 21:21:11.753303 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:11.753226    2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0737d7deac5a5717ab3d7f2eebbdd6be9ae53db31fc351321e3e1f23fc6ff36"} err="failed to get container status \"b0737d7deac5a5717ab3d7f2eebbdd6be9ae53db31fc351321e3e1f23fc6ff36\": rpc error: code = NotFound desc = could not find container \"b0737d7deac5a5717ab3d7f2eebbdd6be9ae53db31fc351321e3e1f23fc6ff36\": container with ID starting with b0737d7deac5a5717ab3d7f2eebbdd6be9ae53db31fc351321e3e1f23fc6ff36 not found: ID does not exist"
Jul 16 21:21:12.747542 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:12.747518    2530 scope.go:117] "RemoveContainer" containerID="1ad49df9b2c4bc588058bb0ba9e153785cb32ee77b6de531ca662c78e43ee69a"
Jul 16 21:21:12.748048 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:12.747745    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n_odh-ai-gateway-infra(894e1873-d7c9-44c2-a9bb-1603f9cdf989)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n" podUID="894e1873-d7c9-44c2-a9bb-1603f9cdf989"
Jul 16 21:21:18.502506 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:18.502469    2530 scope.go:117] "RemoveContainer" containerID="5de026785593f418cd36a173263553928d574bf946a80c1604842ae115d55f21"
Jul 16 21:21:18.774584 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:18.774522    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd" event={"ID":"9bda1e1d-6199-4d10-9342-44f8828bc096","Type":"ContainerStarted","Data":"8dc662998ee1829bbc5c5b336425a50e31a894316acdf6e45eb685ce099cd511"}
Jul 16 21:21:18.797438 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:18.797394    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd" podStartSLOduration=16.797381691 podStartE2EDuration="16.797381691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:21:18.796064286 +0000 UTC m=+2916.817652052" watchObservedRunningTime="2026-07-16 21:21:18.797381691 +0000 UTC m=+2916.818969479"
Jul 16 21:21:19.780574 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:19.780544    2530 generic.go:356] "Generic (PLEG): container finished" podID="9bda1e1d-6199-4d10-9342-44f8828bc096" containerID="8dc662998ee1829bbc5c5b336425a50e31a894316acdf6e45eb685ce099cd511" exitCode=6
Jul 16 21:21:19.781026 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:19.780608    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd" event={"ID":"9bda1e1d-6199-4d10-9342-44f8828bc096","Type":"ContainerDied","Data":"8dc662998ee1829bbc5c5b336425a50e31a894316acdf6e45eb685ce099cd511"}
Jul 16 21:21:19.781026 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:19.780670    2530 scope.go:117] "RemoveContainer" containerID="5de026785593f418cd36a173263553928d574bf946a80c1604842ae115d55f21"
Jul 16 21:21:19.781026 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:19.780959    2530 scope.go:117] "RemoveContainer" containerID="8dc662998ee1829bbc5c5b336425a50e31a894316acdf6e45eb685ce099cd511"
Jul 16 21:21:19.781216 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:19.781196    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-njptd_odh-ai-gateway-infra(9bda1e1d-6199-4d10-9342-44f8828bc096)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd" podUID="9bda1e1d-6199-4d10-9342-44f8828bc096"
Jul 16 21:21:19.807709 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:19.807688    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd"]
Jul 16 21:21:20.918703 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:20.918674    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd"
Jul 16 21:21:21.012351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:21.012307    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bda1e1d-6199-4d10-9342-44f8828bc096-maas-api-service-ca\") pod \"9bda1e1d-6199-4d10-9342-44f8828bc096\" (UID: \"9bda1e1d-6199-4d10-9342-44f8828bc096\") "
Jul 16 21:21:21.012626 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:21.012606    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bda1e1d-6199-4d10-9342-44f8828bc096-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9bda1e1d-6199-4d10-9342-44f8828bc096" (UID: "9bda1e1d-6199-4d10-9342-44f8828bc096"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:21:21.113147 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:21.113073    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bda1e1d-6199-4d10-9342-44f8828bc096-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:21:21.800254 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:21.800223    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd"
Jul 16 21:21:21.800437 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:21.800218    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd" event={"ID":"9bda1e1d-6199-4d10-9342-44f8828bc096","Type":"ContainerDied","Data":"136008a042de48162c64be7f82ff68b259984cba57ba1d0f54332e94718c6deb"}
Jul 16 21:21:21.800437 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:21.800404    2530 scope.go:117] "RemoveContainer" containerID="8dc662998ee1829bbc5c5b336425a50e31a894316acdf6e45eb685ce099cd511"
Jul 16 21:21:21.820540 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:21.820516    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd"]
Jul 16 21:21:21.824518 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:21.824499    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-njptd"]
Jul 16 21:21:22.358995 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:22.358963    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8"]
Jul 16 21:21:22.359376 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:22.359291    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bda1e1d-6199-4d10-9342-44f8828bc096" containerName="revoke-keys"
Jul 16 21:21:22.359376 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:22.359302    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bda1e1d-6199-4d10-9342-44f8828bc096" containerName="revoke-keys"
Jul 16 21:21:22.359376 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:22.359313    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bda1e1d-6199-4d10-9342-44f8828bc096" containerName="revoke-keys"
Jul 16 21:21:22.359376 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:22.359318    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bda1e1d-6199-4d10-9342-44f8828bc096" containerName="revoke-keys"
Jul 16 21:21:22.359376 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:22.359339    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bda1e1d-6199-4d10-9342-44f8828bc096" containerName="revoke-keys"
Jul 16 21:21:22.359376 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:22.359345    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bda1e1d-6199-4d10-9342-44f8828bc096" containerName="revoke-keys"
Jul 16 21:21:22.359567 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:22.359394    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bda1e1d-6199-4d10-9342-44f8828bc096" containerName="revoke-keys"
Jul 16 21:21:22.359567 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:22.359401    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bda1e1d-6199-4d10-9342-44f8828bc096" containerName="revoke-keys"
Jul 16 21:21:22.359567 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:22.359406    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bda1e1d-6199-4d10-9342-44f8828bc096" containerName="revoke-keys"
Jul 16 21:21:22.363443 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:22.363412    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8"
Jul 16 21:21:22.369889 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:22.369864    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8"]
Jul 16 21:21:22.423836 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:22.423803    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69d36a4-9866-4797-b0f2-970cb770e26a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8\" (UID: \"c69d36a4-9866-4797-b0f2-970cb770e26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8"
Jul 16 21:21:22.507648 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:22.507609    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bda1e1d-6199-4d10-9342-44f8828bc096" path="/var/lib/kubelet/pods/9bda1e1d-6199-4d10-9342-44f8828bc096/volumes"
Jul 16 21:21:22.525133 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:22.525085    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69d36a4-9866-4797-b0f2-970cb770e26a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8\" (UID: \"c69d36a4-9866-4797-b0f2-970cb770e26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8"
Jul 16 21:21:22.525662 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:22.525645    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69d36a4-9866-4797-b0f2-970cb770e26a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8\" (UID: \"c69d36a4-9866-4797-b0f2-970cb770e26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8"
Jul 16 21:21:22.675418 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:22.675356    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8"
Jul 16 21:21:22.809189 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:22.809166    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8"]
Jul 16 21:21:22.811272 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:21:22.811243    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69d36a4_9866_4797_b0f2_970cb770e26a.slice/crio-bc078c4a2c226b6dfd8e1f8dd54443ae9a02c358dcf96dda8c9c399d1a9fe2fd WatchSource:0}: Error finding container bc078c4a2c226b6dfd8e1f8dd54443ae9a02c358dcf96dda8c9c399d1a9fe2fd: Status 404 returned error can't find the container with id bc078c4a2c226b6dfd8e1f8dd54443ae9a02c358dcf96dda8c9c399d1a9fe2fd
Jul 16 21:21:23.502298 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:23.502273    2530 scope.go:117] "RemoveContainer" containerID="1ad49df9b2c4bc588058bb0ba9e153785cb32ee77b6de531ca662c78e43ee69a"
Jul 16 21:21:23.811352 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:23.811325    2530 generic.go:356] "Generic (PLEG): container finished" podID="c69d36a4-9866-4797-b0f2-970cb770e26a" containerID="2c43e43463bfa041a2bb716027a2f347a4c2ff5aa8886076d7f64c17fdecf0cb" exitCode=6
Jul 16 21:21:23.811496 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:23.811412    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8" event={"ID":"c69d36a4-9866-4797-b0f2-970cb770e26a","Type":"ContainerDied","Data":"2c43e43463bfa041a2bb716027a2f347a4c2ff5aa8886076d7f64c17fdecf0cb"}
Jul 16 21:21:23.811496 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:23.811481    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8" event={"ID":"c69d36a4-9866-4797-b0f2-970cb770e26a","Type":"ContainerStarted","Data":"bc078c4a2c226b6dfd8e1f8dd54443ae9a02c358dcf96dda8c9c399d1a9fe2fd"}
Jul 16 21:21:23.811595 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:23.811564    2530 scope.go:117] "RemoveContainer" containerID="2c43e43463bfa041a2bb716027a2f347a4c2ff5aa8886076d7f64c17fdecf0cb"
Jul 16 21:21:23.813674 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:23.813645    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n" event={"ID":"894e1873-d7c9-44c2-a9bb-1603f9cdf989","Type":"ContainerStarted","Data":"bbad5ec9946c6afcbd8db9c728353ada15dee56e49049cb99317bafade2b48ff"}
Jul 16 21:21:23.841334 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:23.841284    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n" podStartSLOduration=14.841271586 podStartE2EDuration="14.841271586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:21:23.840159423 +0000 UTC m=+2921.861747189" watchObservedRunningTime="2026-07-16 21:21:23.841271586 +0000 UTC m=+2921.862859352"
Jul 16 21:21:24.255466 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.255441    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm"]
Jul 16 21:21:24.258553 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.258535    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm"
Jul 16 21:21:24.268938 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.268916    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm"]
Jul 16 21:21:24.340910 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.340851    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1a3012-29c7-4110-af98-dcf6a3360d94-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm\" (UID: \"6c1a3012-29c7-4110-af98-dcf6a3360d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm"
Jul 16 21:21:24.441551 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.441502    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1a3012-29c7-4110-af98-dcf6a3360d94-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm\" (UID: \"6c1a3012-29c7-4110-af98-dcf6a3360d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm"
Jul 16 21:21:24.442265 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.442244    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1a3012-29c7-4110-af98-dcf6a3360d94-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm\" (UID: \"6c1a3012-29c7-4110-af98-dcf6a3360d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm"
Jul 16 21:21:24.493759 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.493723    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"]
Jul 16 21:21:24.496990 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.496974    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"
Jul 16 21:21:24.499609 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.499580    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ed076e-serving-cert\""
Jul 16 21:21:24.509655 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.509635    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"]
Jul 16 21:21:24.597976 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.597915    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm"
Jul 16 21:21:24.642863 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.642805    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e-maas-api-tls\") pod \"maas-api-e2e-api-a-ed076e-c975cb44d-44nx2\" (UID: \"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"
Jul 16 21:21:24.642985 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.642866    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e-openshift-service-ca\") pod \"maas-api-e2e-api-a-ed076e-c975cb44d-44nx2\" (UID: \"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"
Jul 16 21:21:24.642985 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.642911    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqqqs\" (UniqueName: \"kubernetes.io/projected/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e-kube-api-access-lqqqs\") pod \"maas-api-e2e-api-a-ed076e-c975cb44d-44nx2\" (UID: \"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"
Jul 16 21:21:24.734672 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.734645    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm"]
Jul 16 21:21:24.737065 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:21:24.737037    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1a3012_29c7_4110_af98_dcf6a3360d94.slice/crio-2e7505253414236abd470bd7ac15eb57776fc36347488997c2692ccde38022c8 WatchSource:0}: Error finding container 2e7505253414236abd470bd7ac15eb57776fc36347488997c2692ccde38022c8: Status 404 returned error can't find the container with id 2e7505253414236abd470bd7ac15eb57776fc36347488997c2692ccde38022c8
Jul 16 21:21:24.743669 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.743647    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e-openshift-service-ca\") pod \"maas-api-e2e-api-a-ed076e-c975cb44d-44nx2\" (UID: \"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"
Jul 16 21:21:24.743747 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.743688    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqqqs\" (UniqueName: \"kubernetes.io/projected/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e-kube-api-access-lqqqs\") pod \"maas-api-e2e-api-a-ed076e-c975cb44d-44nx2\" (UID: \"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"
Jul 16 21:21:24.743805 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.743789    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e-maas-api-tls\") pod \"maas-api-e2e-api-a-ed076e-c975cb44d-44nx2\" (UID: \"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"
Jul 16 21:21:24.744380 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.744360    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e-openshift-service-ca\") pod \"maas-api-e2e-api-a-ed076e-c975cb44d-44nx2\" (UID: \"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"
Jul 16 21:21:24.746302 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.746279    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e-maas-api-tls\") pod \"maas-api-e2e-api-a-ed076e-c975cb44d-44nx2\" (UID: \"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"
Jul 16 21:21:24.752226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.752200    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqqqs\" (UniqueName: \"kubernetes.io/projected/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e-kube-api-access-lqqqs\") pod \"maas-api-e2e-api-a-ed076e-c975cb44d-44nx2\" (UID: \"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"
Jul 16 21:21:24.808723 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.808698    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"
Jul 16 21:21:24.821501 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.821479    2530 generic.go:356] "Generic (PLEG): container finished" podID="894e1873-d7c9-44c2-a9bb-1603f9cdf989" containerID="bbad5ec9946c6afcbd8db9c728353ada15dee56e49049cb99317bafade2b48ff" exitCode=6
Jul 16 21:21:24.821571 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.821548    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n" event={"ID":"894e1873-d7c9-44c2-a9bb-1603f9cdf989","Type":"ContainerDied","Data":"bbad5ec9946c6afcbd8db9c728353ada15dee56e49049cb99317bafade2b48ff"}
Jul 16 21:21:24.821634 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.821607    2530 scope.go:117] "RemoveContainer" containerID="1ad49df9b2c4bc588058bb0ba9e153785cb32ee77b6de531ca662c78e43ee69a"
Jul 16 21:21:24.821888 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.821871    2530 scope.go:117] "RemoveContainer" containerID="bbad5ec9946c6afcbd8db9c728353ada15dee56e49049cb99317bafade2b48ff"
Jul 16 21:21:24.822131 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:24.822079    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n_odh-ai-gateway-infra(894e1873-d7c9-44c2-a9bb-1603f9cdf989)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n" podUID="894e1873-d7c9-44c2-a9bb-1603f9cdf989"
Jul 16 21:21:24.823746 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.823716    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm" event={"ID":"6c1a3012-29c7-4110-af98-dcf6a3360d94","Type":"ContainerStarted","Data":"2e7505253414236abd470bd7ac15eb57776fc36347488997c2692ccde38022c8"}
Jul 16 21:21:24.826294 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.826262    2530 generic.go:356] "Generic (PLEG): container finished" podID="c69d36a4-9866-4797-b0f2-970cb770e26a" containerID="775d23ac95a5420ceb2409fa3f7a7f50fed00dcbb8c5b04ac9c69f6313792610" exitCode=6
Jul 16 21:21:24.826399 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.826288    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8" event={"ID":"c69d36a4-9866-4797-b0f2-970cb770e26a","Type":"ContainerDied","Data":"775d23ac95a5420ceb2409fa3f7a7f50fed00dcbb8c5b04ac9c69f6313792610"}
Jul 16 21:21:24.826508 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.826492    2530 scope.go:117] "RemoveContainer" containerID="775d23ac95a5420ceb2409fa3f7a7f50fed00dcbb8c5b04ac9c69f6313792610"
Jul 16 21:21:24.826701 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:24.826679    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8_odh-ai-gateway-infra(c69d36a4-9866-4797-b0f2-970cb770e26a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8" podUID="c69d36a4-9866-4797-b0f2-970cb770e26a"
Jul 16 21:21:24.838151 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.838076    2530 scope.go:117] "RemoveContainer" containerID="2c43e43463bfa041a2bb716027a2f347a4c2ff5aa8886076d7f64c17fdecf0cb"
Jul 16 21:21:24.853525 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.853505    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n"]
Jul 16 21:21:24.946641 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.946616    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"]
Jul 16 21:21:24.948043 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:21:24.948020    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1ab3c6_d5d6_4935_8d04_14e54ae00e6e.slice/crio-84eecd4282b6e9f146eeb82a9525bc7ba9372402e7a306ee35cc9906764bfefa WatchSource:0}: Error finding container 84eecd4282b6e9f146eeb82a9525bc7ba9372402e7a306ee35cc9906764bfefa: Status 404 returned error can't find the container with id 84eecd4282b6e9f146eeb82a9525bc7ba9372402e7a306ee35cc9906764bfefa
Jul 16 21:21:24.949362 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:24.949345    2530 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:21:25.833489 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:25.833459    2530 generic.go:356] "Generic (PLEG): container finished" podID="6c1a3012-29c7-4110-af98-dcf6a3360d94" containerID="6c57cde031301d0e9944155a4d2cf1489db34d82d0651e9920c274008200b3aa" exitCode=6
Jul 16 21:21:25.833841 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:25.833502    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm" event={"ID":"6c1a3012-29c7-4110-af98-dcf6a3360d94","Type":"ContainerDied","Data":"6c57cde031301d0e9944155a4d2cf1489db34d82d0651e9920c274008200b3aa"}
Jul 16 21:21:25.833841 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:25.833706    2530 scope.go:117] "RemoveContainer" containerID="6c57cde031301d0e9944155a4d2cf1489db34d82d0651e9920c274008200b3aa"
Jul 16 21:21:25.834938 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:25.834913    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2" event={"ID":"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e","Type":"ContainerStarted","Data":"84eecd4282b6e9f146eeb82a9525bc7ba9372402e7a306ee35cc9906764bfefa"}
Jul 16 21:21:25.836806 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:25.836786    2530 scope.go:117] "RemoveContainer" containerID="775d23ac95a5420ceb2409fa3f7a7f50fed00dcbb8c5b04ac9c69f6313792610"
Jul 16 21:21:25.837018 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:25.836992    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8_odh-ai-gateway-infra(c69d36a4-9866-4797-b0f2-970cb770e26a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8" podUID="c69d36a4-9866-4797-b0f2-970cb770e26a"
Jul 16 21:21:25.981817 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:25.981794    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n"
Jul 16 21:21:26.056045 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:26.056015    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/894e1873-d7c9-44c2-a9bb-1603f9cdf989-maas-api-service-ca\") pod \"894e1873-d7c9-44c2-a9bb-1603f9cdf989\" (UID: \"894e1873-d7c9-44c2-a9bb-1603f9cdf989\") "
Jul 16 21:21:26.056355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:26.056335    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/894e1873-d7c9-44c2-a9bb-1603f9cdf989-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "894e1873-d7c9-44c2-a9bb-1603f9cdf989" (UID: "894e1873-d7c9-44c2-a9bb-1603f9cdf989"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:21:26.157048 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:26.157024    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/894e1873-d7c9-44c2-a9bb-1603f9cdf989-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:21:26.842161 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:26.842133    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n"
Jul 16 21:21:26.842533 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:26.842140    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n" event={"ID":"894e1873-d7c9-44c2-a9bb-1603f9cdf989","Type":"ContainerDied","Data":"aae29d4a3897670616af37fa24c900c70d7dfd1e217f14d5e14b09c9d625a473"}
Jul 16 21:21:26.842533 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:26.842269    2530 scope.go:117] "RemoveContainer" containerID="bbad5ec9946c6afcbd8db9c728353ada15dee56e49049cb99317bafade2b48ff"
Jul 16 21:21:26.844188 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:26.844162    2530 generic.go:356] "Generic (PLEG): container finished" podID="6c1a3012-29c7-4110-af98-dcf6a3360d94" containerID="51a1422e1bdae34a86a6512a1411bf97a68b357267537bceb16a702b5523776d" exitCode=6
Jul 16 21:21:26.844285 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:26.844253    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm" event={"ID":"6c1a3012-29c7-4110-af98-dcf6a3360d94","Type":"ContainerDied","Data":"51a1422e1bdae34a86a6512a1411bf97a68b357267537bceb16a702b5523776d"}
Jul 16 21:21:26.844497 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:26.844480    2530 scope.go:117] "RemoveContainer" containerID="51a1422e1bdae34a86a6512a1411bf97a68b357267537bceb16a702b5523776d"
Jul 16 21:21:26.844687 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:26.844663    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm_odh-ai-gateway-infra(6c1a3012-29c7-4110-af98-dcf6a3360d94)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm" podUID="6c1a3012-29c7-4110-af98-dcf6a3360d94"
Jul 16 21:21:26.845936 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:26.845913    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2" event={"ID":"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e","Type":"ContainerStarted","Data":"cd917b1ea45ff20aa2ba874e4eec9163c6427c621e7b9f78b74a7c63b7a20aad"}
Jul 16 21:21:26.846032 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:26.846016    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"
Jul 16 21:21:26.850853 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:26.850837    2530 scope.go:117] "RemoveContainer" containerID="6c57cde031301d0e9944155a4d2cf1489db34d82d0651e9920c274008200b3aa"
Jul 16 21:21:26.860053 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:26.860030    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n"]
Jul 16 21:21:26.862846 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:26.862825    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-9c46n"]
Jul 16 21:21:26.893388 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:26.893352    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2" podStartSLOduration=1.9421205590000001 podStartE2EDuration="2.893339137s" totalImagesPullingTime="951.218578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:21:26.892735669 +0000 UTC m=+2924.914323436" watchObservedRunningTime="2026-07-16 21:21:26.893339137 +0000 UTC m=+2924.914926899"
Jul 16 21:21:27.854062 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:27.854020    2530 scope.go:117] "RemoveContainer" containerID="51a1422e1bdae34a86a6512a1411bf97a68b357267537bceb16a702b5523776d"
Jul 16 21:21:27.854537 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:27.854252    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm_odh-ai-gateway-infra(6c1a3012-29c7-4110-af98-dcf6a3360d94)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm" podUID="6c1a3012-29c7-4110-af98-dcf6a3360d94"
Jul 16 21:21:28.507846 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:28.507814    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="894e1873-d7c9-44c2-a9bb-1603f9cdf989" path="/var/lib/kubelet/pods/894e1873-d7c9-44c2-a9bb-1603f9cdf989/volumes"
Jul 16 21:21:32.425577 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.425547    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"]
Jul 16 21:21:32.425948 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.425871    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="894e1873-d7c9-44c2-a9bb-1603f9cdf989" containerName="revoke-keys"
Jul 16 21:21:32.425948 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.425881    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="894e1873-d7c9-44c2-a9bb-1603f9cdf989" containerName="revoke-keys"
Jul 16 21:21:32.425948 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.425894    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="894e1873-d7c9-44c2-a9bb-1603f9cdf989" containerName="revoke-keys"
Jul 16 21:21:32.425948 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.425899    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="894e1873-d7c9-44c2-a9bb-1603f9cdf989" containerName="revoke-keys"
Jul 16 21:21:32.425948 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.425913    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="894e1873-d7c9-44c2-a9bb-1603f9cdf989" containerName="revoke-keys"
Jul 16 21:21:32.425948 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.425919    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="894e1873-d7c9-44c2-a9bb-1603f9cdf989" containerName="revoke-keys"
Jul 16 21:21:32.426186 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.425967    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="894e1873-d7c9-44c2-a9bb-1603f9cdf989" containerName="revoke-keys"
Jul 16 21:21:32.426186 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.425975    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="894e1873-d7c9-44c2-a9bb-1603f9cdf989" containerName="revoke-keys"
Jul 16 21:21:32.430292 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.430273    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"
Jul 16 21:21:32.432973 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.432953    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-df0daf-serving-cert\""
Jul 16 21:21:32.441784 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.441763    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"]
Jul 16 21:21:32.513281 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.513251    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2fbj\" (UniqueName: \"kubernetes.io/projected/cfa5eb32-916f-4577-b474-12917f1bbd75-kube-api-access-p2fbj\") pod \"maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf\" (UID: \"cfa5eb32-916f-4577-b474-12917f1bbd75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"
Jul 16 21:21:32.513452 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.513347    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa5eb32-916f-4577-b474-12917f1bbd75-maas-api-tls\") pod \"maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf\" (UID: \"cfa5eb32-916f-4577-b474-12917f1bbd75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"
Jul 16 21:21:32.513452 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.513404    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa5eb32-916f-4577-b474-12917f1bbd75-openshift-service-ca\") pod \"maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf\" (UID: \"cfa5eb32-916f-4577-b474-12917f1bbd75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"
Jul 16 21:21:32.614231 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.614193    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa5eb32-916f-4577-b474-12917f1bbd75-maas-api-tls\") pod \"maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf\" (UID: \"cfa5eb32-916f-4577-b474-12917f1bbd75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"
Jul 16 21:21:32.614418 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.614255    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa5eb32-916f-4577-b474-12917f1bbd75-openshift-service-ca\") pod \"maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf\" (UID: \"cfa5eb32-916f-4577-b474-12917f1bbd75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"
Jul 16 21:21:32.614418 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.614295    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2fbj\" (UniqueName: \"kubernetes.io/projected/cfa5eb32-916f-4577-b474-12917f1bbd75-kube-api-access-p2fbj\") pod \"maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf\" (UID: \"cfa5eb32-916f-4577-b474-12917f1bbd75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"
Jul 16 21:21:32.614418 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:32.614347    2530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-serving-cert: secret "maas-api-e2e-api-b-df0daf-serving-cert" not found
Jul 16 21:21:32.614588 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:32.614423    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfa5eb32-916f-4577-b474-12917f1bbd75-maas-api-tls podName:cfa5eb32-916f-4577-b474-12917f1bbd75 nodeName:}" failed. No retries permitted until 2026-07-16 21:21:33.114406004 +0000 UTC m=+2931.135993750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cfa5eb32-916f-4577-b474-12917f1bbd75-maas-api-tls") pod "maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf" (UID: "cfa5eb32-916f-4577-b474-12917f1bbd75") : secret "maas-api-e2e-api-b-df0daf-serving-cert" not found
Jul 16 21:21:32.614985 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.614963    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa5eb32-916f-4577-b474-12917f1bbd75-openshift-service-ca\") pod \"maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf\" (UID: \"cfa5eb32-916f-4577-b474-12917f1bbd75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"
Jul 16 21:21:32.623316 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.623295    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2fbj\" (UniqueName: \"kubernetes.io/projected/cfa5eb32-916f-4577-b474-12917f1bbd75-kube-api-access-p2fbj\") pod \"maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf\" (UID: \"cfa5eb32-916f-4577-b474-12917f1bbd75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"
Jul 16 21:21:32.859718 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:32.859694    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"
Jul 16 21:21:33.119281 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:33.119192    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa5eb32-916f-4577-b474-12917f1bbd75-maas-api-tls\") pod \"maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf\" (UID: \"cfa5eb32-916f-4577-b474-12917f1bbd75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"
Jul 16 21:21:33.119455 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:33.119324    2530 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-serving-cert: secret "maas-api-e2e-api-b-df0daf-serving-cert" not found
Jul 16 21:21:33.119455 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:33.119384    2530 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfa5eb32-916f-4577-b474-12917f1bbd75-maas-api-tls podName:cfa5eb32-916f-4577-b474-12917f1bbd75 nodeName:}" failed. No retries permitted until 2026-07-16 21:21:34.119366784 +0000 UTC m=+2932.140954528 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cfa5eb32-916f-4577-b474-12917f1bbd75-maas-api-tls") pod "maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf" (UID: "cfa5eb32-916f-4577-b474-12917f1bbd75") : secret "maas-api-e2e-api-b-df0daf-serving-cert" not found
Jul 16 21:21:34.128556 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:34.128471    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa5eb32-916f-4577-b474-12917f1bbd75-maas-api-tls\") pod \"maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf\" (UID: \"cfa5eb32-916f-4577-b474-12917f1bbd75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"
Jul 16 21:21:34.131218 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:34.131193    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa5eb32-916f-4577-b474-12917f1bbd75-maas-api-tls\") pod \"maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf\" (UID: \"cfa5eb32-916f-4577-b474-12917f1bbd75\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"
Jul 16 21:21:34.240209 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:34.240176    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"
Jul 16 21:21:34.372299 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:34.372198    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"]
Jul 16 21:21:34.374571 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:21:34.374529    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa5eb32_916f_4577_b474_12917f1bbd75.slice/crio-8d072e347dca39118291c48749950479e0964cc262e5d977ac8606688518f3dc WatchSource:0}: Error finding container 8d072e347dca39118291c48749950479e0964cc262e5d977ac8606688518f3dc: Status 404 returned error can't find the container with id 8d072e347dca39118291c48749950479e0964cc262e5d977ac8606688518f3dc
Jul 16 21:21:34.890626 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:34.890593    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf" event={"ID":"cfa5eb32-916f-4577-b474-12917f1bbd75","Type":"ContainerStarted","Data":"105b8ad9075630ce6939a663124e04cc0fc776fa781db10999a4bbe5630fa359"}
Jul 16 21:21:34.890788 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:34.890649    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf" event={"ID":"cfa5eb32-916f-4577-b474-12917f1bbd75","Type":"ContainerStarted","Data":"8d072e347dca39118291c48749950479e0964cc262e5d977ac8606688518f3dc"}
Jul 16 21:21:34.890788 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:34.890729    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"
Jul 16 21:21:34.912767 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:34.912720    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf" podStartSLOduration=2.598966952 podStartE2EDuration="2.912706032s" totalImagesPullingTime="313.73908ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:21:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:21:34.91017304 +0000 UTC m=+2932.931760848" watchObservedRunningTime="2026-07-16 21:21:34.912706032 +0000 UTC m=+2932.934293797"
Jul 16 21:21:37.502759 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:37.502722    2530 scope.go:117] "RemoveContainer" containerID="775d23ac95a5420ceb2409fa3f7a7f50fed00dcbb8c5b04ac9c69f6313792610"
Jul 16 21:21:37.906296 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:37.906225    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8" event={"ID":"c69d36a4-9866-4797-b0f2-970cb770e26a","Type":"ContainerStarted","Data":"7a8602d64797ebdd1c3765a1f77b1538738831950e9d21c15510efc64a887091"}
Jul 16 21:21:37.923669 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:37.923621    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8" podStartSLOduration=15.923605831 podStartE2EDuration="15.923605831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:21:37.921406091 +0000 UTC m=+2935.942993857" watchObservedRunningTime="2026-07-16 21:21:37.923605831 +0000 UTC m=+2935.945193596"
Jul 16 21:21:38.912716 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:38.912680    2530 generic.go:356] "Generic (PLEG): container finished" podID="c69d36a4-9866-4797-b0f2-970cb770e26a" containerID="7a8602d64797ebdd1c3765a1f77b1538738831950e9d21c15510efc64a887091" exitCode=6
Jul 16 21:21:38.913137 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:38.912755    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8" event={"ID":"c69d36a4-9866-4797-b0f2-970cb770e26a","Type":"ContainerDied","Data":"7a8602d64797ebdd1c3765a1f77b1538738831950e9d21c15510efc64a887091"}
Jul 16 21:21:38.913137 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:38.912816    2530 scope.go:117] "RemoveContainer" containerID="775d23ac95a5420ceb2409fa3f7a7f50fed00dcbb8c5b04ac9c69f6313792610"
Jul 16 21:21:38.913137 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:38.913054    2530 scope.go:117] "RemoveContainer" containerID="7a8602d64797ebdd1c3765a1f77b1538738831950e9d21c15510efc64a887091"
Jul 16 21:21:38.913277 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:38.913253    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8_odh-ai-gateway-infra(c69d36a4-9866-4797-b0f2-970cb770e26a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8" podUID="c69d36a4-9866-4797-b0f2-970cb770e26a"
Jul 16 21:21:38.937760 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:38.937736    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8"]
Jul 16 21:21:40.050324 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:40.050300    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8"
Jul 16 21:21:40.177328 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:40.177261    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69d36a4-9866-4797-b0f2-970cb770e26a-maas-api-service-ca\") pod \"c69d36a4-9866-4797-b0f2-970cb770e26a\" (UID: \"c69d36a4-9866-4797-b0f2-970cb770e26a\") "
Jul 16 21:21:40.177593 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:40.177572    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c69d36a4-9866-4797-b0f2-970cb770e26a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c69d36a4-9866-4797-b0f2-970cb770e26a" (UID: "c69d36a4-9866-4797-b0f2-970cb770e26a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:21:40.277755 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:40.277727    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69d36a4-9866-4797-b0f2-970cb770e26a-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:21:40.901837 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:40.901757    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"
Jul 16 21:21:40.924640 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:40.924611    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8"
Jul 16 21:21:40.924802 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:40.924656    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8" event={"ID":"c69d36a4-9866-4797-b0f2-970cb770e26a","Type":"ContainerDied","Data":"bc078c4a2c226b6dfd8e1f8dd54443ae9a02c358dcf96dda8c9c399d1a9fe2fd"}
Jul 16 21:21:40.924802 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:40.924713    2530 scope.go:117] "RemoveContainer" containerID="7a8602d64797ebdd1c3765a1f77b1538738831950e9d21c15510efc64a887091"
Jul 16 21:21:40.943398 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:40.943376    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8"]
Jul 16 21:21:40.950739 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:40.950718    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-vqvg8"]
Jul 16 21:21:41.502518 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:41.502493    2530 scope.go:117] "RemoveContainer" containerID="51a1422e1bdae34a86a6512a1411bf97a68b357267537bceb16a702b5523776d"
Jul 16 21:21:41.931902 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:41.931875    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm" event={"ID":"6c1a3012-29c7-4110-af98-dcf6a3360d94","Type":"ContainerStarted","Data":"d2c1a9308cee57b80f51d7acb92be690498837333b76a22b97ea62799a607a75"}
Jul 16 21:21:41.949860 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:41.949818    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm" podStartSLOduration=17.949804062 podStartE2EDuration="17.949804062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:21:41.948434685 +0000 UTC m=+2939.970022451" watchObservedRunningTime="2026-07-16 21:21:41.949804062 +0000 UTC m=+2939.971391830"
Jul 16 21:21:42.506916 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:42.506887    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c69d36a4-9866-4797-b0f2-970cb770e26a" path="/var/lib/kubelet/pods/c69d36a4-9866-4797-b0f2-970cb770e26a/volumes"
Jul 16 21:21:42.946042 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:42.945954    2530 generic.go:356] "Generic (PLEG): container finished" podID="6c1a3012-29c7-4110-af98-dcf6a3360d94" containerID="d2c1a9308cee57b80f51d7acb92be690498837333b76a22b97ea62799a607a75" exitCode=6
Jul 16 21:21:42.946269 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:42.946037    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm" event={"ID":"6c1a3012-29c7-4110-af98-dcf6a3360d94","Type":"ContainerDied","Data":"d2c1a9308cee57b80f51d7acb92be690498837333b76a22b97ea62799a607a75"}
Jul 16 21:21:42.946269 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:42.946165    2530 scope.go:117] "RemoveContainer" containerID="51a1422e1bdae34a86a6512a1411bf97a68b357267537bceb16a702b5523776d"
Jul 16 21:21:42.946457 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:42.946443    2530 scope.go:117] "RemoveContainer" containerID="d2c1a9308cee57b80f51d7acb92be690498837333b76a22b97ea62799a607a75"
Jul 16 21:21:42.946654 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:42.946636    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm_odh-ai-gateway-infra(6c1a3012-29c7-4110-af98-dcf6a3360d94)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm" podUID="6c1a3012-29c7-4110-af98-dcf6a3360d94"
Jul 16 21:21:42.957577 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:42.957559    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm"]
Jul 16 21:21:44.095364 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.095340    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm"
Jul 16 21:21:44.209745 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.209721    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1a3012-29c7-4110-af98-dcf6a3360d94-maas-api-service-ca\") pod \"6c1a3012-29c7-4110-af98-dcf6a3360d94\" (UID: \"6c1a3012-29c7-4110-af98-dcf6a3360d94\") "
Jul 16 21:21:44.210065 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.210042    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1a3012-29c7-4110-af98-dcf6a3360d94-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c1a3012-29c7-4110-af98-dcf6a3360d94" (UID: "6c1a3012-29c7-4110-af98-dcf6a3360d94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:21:44.310936 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.310905    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1a3012-29c7-4110-af98-dcf6a3360d94-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:21:44.853165 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.853076    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed076e-67mcb"]
Jul 16 21:21:44.853456 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.853440    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69d36a4-9866-4797-b0f2-970cb770e26a" containerName="revoke-keys"
Jul 16 21:21:44.853509 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.853459    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69d36a4-9866-4797-b0f2-970cb770e26a" containerName="revoke-keys"
Jul 16 21:21:44.853509 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.853475    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1a3012-29c7-4110-af98-dcf6a3360d94" containerName="revoke-keys"
Jul 16 21:21:44.853509 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.853481    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1a3012-29c7-4110-af98-dcf6a3360d94" containerName="revoke-keys"
Jul 16 21:21:44.853509 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.853487    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69d36a4-9866-4797-b0f2-970cb770e26a" containerName="revoke-keys"
Jul 16 21:21:44.853509 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.853493    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69d36a4-9866-4797-b0f2-970cb770e26a" containerName="revoke-keys"
Jul 16 21:21:44.853509 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.853505    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1a3012-29c7-4110-af98-dcf6a3360d94" containerName="revoke-keys"
Jul 16 21:21:44.853509 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.853509    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1a3012-29c7-4110-af98-dcf6a3360d94" containerName="revoke-keys"
Jul 16 21:21:44.853717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.853563    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="894e1873-d7c9-44c2-a9bb-1603f9cdf989" containerName="revoke-keys"
Jul 16 21:21:44.853717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.853574    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1a3012-29c7-4110-af98-dcf6a3360d94" containerName="revoke-keys"
Jul 16 21:21:44.853717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.853580    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1a3012-29c7-4110-af98-dcf6a3360d94" containerName="revoke-keys"
Jul 16 21:21:44.853717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.853586    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1a3012-29c7-4110-af98-dcf6a3360d94" containerName="revoke-keys"
Jul 16 21:21:44.853717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.853592    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c69d36a4-9866-4797-b0f2-970cb770e26a" containerName="revoke-keys"
Jul 16 21:21:44.853717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.853602    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c69d36a4-9866-4797-b0f2-970cb770e26a" containerName="revoke-keys"
Jul 16 21:21:44.853717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.853610    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c69d36a4-9866-4797-b0f2-970cb770e26a" containerName="revoke-keys"
Jul 16 21:21:44.856429 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.856413    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed076e-67mcb"
Jul 16 21:21:44.864663 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.864640    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed076e-67mcb"]
Jul 16 21:21:44.957318 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.957283    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm" event={"ID":"6c1a3012-29c7-4110-af98-dcf6a3360d94","Type":"ContainerDied","Data":"2e7505253414236abd470bd7ac15eb57776fc36347488997c2692ccde38022c8"}
Jul 16 21:21:44.957473 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.957319    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm"
Jul 16 21:21:44.957473 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.957353    2530 scope.go:117] "RemoveContainer" containerID="d2c1a9308cee57b80f51d7acb92be690498837333b76a22b97ea62799a607a75"
Jul 16 21:21:44.971965 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.971945    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm"]
Jul 16 21:21:44.976257 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:44.976239    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-js2tm"]
Jul 16 21:21:45.016614 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:45.016590    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e0bbd2-48bd-415e-a200-16f70e4d8a17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ed076e-67mcb\" (UID: \"39e0bbd2-48bd-415e-a200-16f70e4d8a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed076e-67mcb"
Jul 16 21:21:45.117192 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:45.117123    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e0bbd2-48bd-415e-a200-16f70e4d8a17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ed076e-67mcb\" (UID: \"39e0bbd2-48bd-415e-a200-16f70e4d8a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed076e-67mcb"
Jul 16 21:21:45.117657 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:45.117639    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e0bbd2-48bd-415e-a200-16f70e4d8a17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ed076e-67mcb\" (UID: \"39e0bbd2-48bd-415e-a200-16f70e4d8a17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed076e-67mcb"
Jul 16 21:21:45.167161 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:45.167137    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed076e-67mcb"
Jul 16 21:21:45.314950 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:45.314924    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed076e-67mcb"]
Jul 16 21:21:45.316863 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:21:45.316839    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e0bbd2_48bd_415e_a200_16f70e4d8a17.slice/crio-0d51894d5229c163ef3e0bec4382e7aae07ace6c95a060f537f51c61281aee4f WatchSource:0}: Error finding container 0d51894d5229c163ef3e0bec4382e7aae07ace6c95a060f537f51c61281aee4f: Status 404 returned error can't find the container with id 0d51894d5229c163ef3e0bec4382e7aae07ace6c95a060f537f51c61281aee4f
Jul 16 21:21:45.967001 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:45.965605    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed076e-67mcb" event={"ID":"39e0bbd2-48bd-415e-a200-16f70e4d8a17","Type":"ContainerStarted","Data":"b47ddacee732c3a97b33bb41502962be60181ba425db8e88c5c6b5fd4a104da0"}
Jul 16 21:21:45.967001 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:45.965671    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed076e-67mcb" event={"ID":"39e0bbd2-48bd-415e-a200-16f70e4d8a17","Type":"ContainerStarted","Data":"0d51894d5229c163ef3e0bec4382e7aae07ace6c95a060f537f51c61281aee4f"}
Jul 16 21:21:45.981666 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:45.981610    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed076e-67mcb" podStartSLOduration=1.981596275 podStartE2EDuration="1.981596275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:21:45.979947585 +0000 UTC m=+2944.001535349" watchObservedRunningTime="2026-07-16 21:21:45.981596275 +0000 UTC m=+2944.003184040"
Jul 16 21:21:46.507533 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:46.507499    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c1a3012-29c7-4110-af98-dcf6a3360d94" path="/var/lib/kubelet/pods/6c1a3012-29c7-4110-af98-dcf6a3360d94/volumes"
Jul 16 21:21:46.974073 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:46.974044    2530 generic.go:356] "Generic (PLEG): container finished" podID="39e0bbd2-48bd-415e-a200-16f70e4d8a17" containerID="b47ddacee732c3a97b33bb41502962be60181ba425db8e88c5c6b5fd4a104da0" exitCode=0
Jul 16 21:21:46.974247 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:46.974080    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed076e-67mcb" event={"ID":"39e0bbd2-48bd-415e-a200-16f70e4d8a17","Type":"ContainerDied","Data":"b47ddacee732c3a97b33bb41502962be60181ba425db8e88c5c6b5fd4a104da0"}
Jul 16 21:21:48.105375 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:48.105351    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed076e-67mcb"
Jul 16 21:21:48.240990 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:48.240924    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e0bbd2-48bd-415e-a200-16f70e4d8a17-maas-api-service-ca\") pod \"39e0bbd2-48bd-415e-a200-16f70e4d8a17\" (UID: \"39e0bbd2-48bd-415e-a200-16f70e4d8a17\") "
Jul 16 21:21:48.241285 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:48.241262    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e0bbd2-48bd-415e-a200-16f70e4d8a17-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "39e0bbd2-48bd-415e-a200-16f70e4d8a17" (UID: "39e0bbd2-48bd-415e-a200-16f70e4d8a17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:21:48.342139 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:48.342091    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39e0bbd2-48bd-415e-a200-16f70e4d8a17-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:21:48.984188 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:48.984138    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed076e-67mcb" event={"ID":"39e0bbd2-48bd-415e-a200-16f70e4d8a17","Type":"ContainerDied","Data":"0d51894d5229c163ef3e0bec4382e7aae07ace6c95a060f537f51c61281aee4f"}
Jul 16 21:21:48.984355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:48.984210    2530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d51894d5229c163ef3e0bec4382e7aae07ace6c95a060f537f51c61281aee4f"
Jul 16 21:21:48.984355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:48.984161    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ed076e-67mcb"
Jul 16 21:21:52.105962 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:52.105924    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg"]
Jul 16 21:21:52.106417 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:52.106289    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1a3012-29c7-4110-af98-dcf6a3360d94" containerName="revoke-keys"
Jul 16 21:21:52.106417 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:52.106300    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1a3012-29c7-4110-af98-dcf6a3360d94" containerName="revoke-keys"
Jul 16 21:21:52.106417 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:52.106318    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69d36a4-9866-4797-b0f2-970cb770e26a" containerName="revoke-keys"
Jul 16 21:21:52.106417 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:52.106324    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69d36a4-9866-4797-b0f2-970cb770e26a" containerName="revoke-keys"
Jul 16 21:21:52.106417 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:52.106334    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39e0bbd2-48bd-415e-a200-16f70e4d8a17" containerName="revoke-keys"
Jul 16 21:21:52.106417 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:52.106339    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="39e0bbd2-48bd-415e-a200-16f70e4d8a17" containerName="revoke-keys"
Jul 16 21:21:52.106417 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:52.106393    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="39e0bbd2-48bd-415e-a200-16f70e4d8a17" containerName="revoke-keys"
Jul 16 21:21:52.109271 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:52.109253    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg"
Jul 16 21:21:52.111535 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:52.111513    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:21:52.116390 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:52.116362    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg"]
Jul 16 21:21:52.170304 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:52.170281    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4365dc-49f8-42ed-9b0d-0000626b792a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-p67mg\" (UID: \"eb4365dc-49f8-42ed-9b0d-0000626b792a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg"
Jul 16 21:21:52.271541 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:52.271516    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4365dc-49f8-42ed-9b0d-0000626b792a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-p67mg\" (UID: \"eb4365dc-49f8-42ed-9b0d-0000626b792a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg"
Jul 16 21:21:52.272086 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:52.272068    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4365dc-49f8-42ed-9b0d-0000626b792a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-p67mg\" (UID: \"eb4365dc-49f8-42ed-9b0d-0000626b792a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg"
Jul 16 21:21:52.420535 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:52.420472    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg"
Jul 16 21:21:52.551971 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:52.551904    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg"]
Jul 16 21:21:52.554746 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:21:52.554714    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4365dc_49f8_42ed_9b0d_0000626b792a.slice/crio-0f67796d0ad6dc3a503b104f367f8f0130a3ab7433e649313d5fe3c34ad252e8 WatchSource:0}: Error finding container 0f67796d0ad6dc3a503b104f367f8f0130a3ab7433e649313d5fe3c34ad252e8: Status 404 returned error can't find the container with id 0f67796d0ad6dc3a503b104f367f8f0130a3ab7433e649313d5fe3c34ad252e8
Jul 16 21:21:53.003922 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:53.003851    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg" event={"ID":"eb4365dc-49f8-42ed-9b0d-0000626b792a","Type":"ContainerStarted","Data":"8a6081063edc205979123430ebf0db3a1ef9509ba9c729401100a90a8d8ad6bb"}
Jul 16 21:21:53.003922 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:53.003913    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg" event={"ID":"eb4365dc-49f8-42ed-9b0d-0000626b792a","Type":"ContainerStarted","Data":"0f67796d0ad6dc3a503b104f367f8f0130a3ab7433e649313d5fe3c34ad252e8"}
Jul 16 21:21:53.023070 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:53.023027    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg" podStartSLOduration=1.023011235 podStartE2EDuration="1.023011235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:21:53.021478898 +0000 UTC m=+2951.043066665" watchObservedRunningTime="2026-07-16 21:21:53.023011235 +0000 UTC m=+2951.044599000"
Jul 16 21:21:54.009799 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:54.009764    2530 generic.go:356] "Generic (PLEG): container finished" podID="eb4365dc-49f8-42ed-9b0d-0000626b792a" containerID="8a6081063edc205979123430ebf0db3a1ef9509ba9c729401100a90a8d8ad6bb" exitCode=6
Jul 16 21:21:54.010218 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:54.009845    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg" event={"ID":"eb4365dc-49f8-42ed-9b0d-0000626b792a","Type":"ContainerDied","Data":"8a6081063edc205979123430ebf0db3a1ef9509ba9c729401100a90a8d8ad6bb"}
Jul 16 21:21:54.010218 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:54.010004    2530 scope.go:117] "RemoveContainer" containerID="8a6081063edc205979123430ebf0db3a1ef9509ba9c729401100a90a8d8ad6bb"
Jul 16 21:21:55.016614 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:55.016535    2530 generic.go:356] "Generic (PLEG): container finished" podID="eb4365dc-49f8-42ed-9b0d-0000626b792a" containerID="95f79fa9bdda8b5475ea9d052d72cbc15ae479531345ebef37b17825908c859b" exitCode=6
Jul 16 21:21:55.016982 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:55.016625    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg" event={"ID":"eb4365dc-49f8-42ed-9b0d-0000626b792a","Type":"ContainerDied","Data":"95f79fa9bdda8b5475ea9d052d72cbc15ae479531345ebef37b17825908c859b"}
Jul 16 21:21:55.016982 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:55.016686    2530 scope.go:117] "RemoveContainer" containerID="8a6081063edc205979123430ebf0db3a1ef9509ba9c729401100a90a8d8ad6bb"
Jul 16 21:21:55.016982 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:55.016853    2530 scope.go:117] "RemoveContainer" containerID="95f79fa9bdda8b5475ea9d052d72cbc15ae479531345ebef37b17825908c859b"
Jul 16 21:21:55.017134 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:55.017048    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-p67mg_odh-ai-gateway-infra(eb4365dc-49f8-42ed-9b0d-0000626b792a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg" podUID="eb4365dc-49f8-42ed-9b0d-0000626b792a"
Jul 16 21:21:55.785875 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:55.785836    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"]
Jul 16 21:21:55.786222 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:55.786187    2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2" podUID="ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e" containerName="maas-api" containerID="cri-o://cd917b1ea45ff20aa2ba874e4eec9163c6427c621e7b9f78b74a7c63b7a20aad" gracePeriod=30
Jul 16 21:21:56.023702 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:56.023673    2530 generic.go:356] "Generic (PLEG): container finished" podID="ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e" containerID="cd917b1ea45ff20aa2ba874e4eec9163c6427c621e7b9f78b74a7c63b7a20aad" exitCode=0
Jul 16 21:21:56.024138 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:56.023753    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2" event={"ID":"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e","Type":"ContainerDied","Data":"cd917b1ea45ff20aa2ba874e4eec9163c6427c621e7b9f78b74a7c63b7a20aad"}
Jul 16 21:21:56.024138 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:56.023819    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2" event={"ID":"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e","Type":"ContainerDied","Data":"84eecd4282b6e9f146eeb82a9525bc7ba9372402e7a306ee35cc9906764bfefa"}
Jul 16 21:21:56.024138 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:56.023844    2530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84eecd4282b6e9f146eeb82a9525bc7ba9372402e7a306ee35cc9906764bfefa"
Jul 16 21:21:56.025887 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:56.025868    2530 scope.go:117] "RemoveContainer" containerID="95f79fa9bdda8b5475ea9d052d72cbc15ae479531345ebef37b17825908c859b"
Jul 16 21:21:56.026061 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:21:56.026044    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-p67mg_odh-ai-gateway-infra(eb4365dc-49f8-42ed-9b0d-0000626b792a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg" podUID="eb4365dc-49f8-42ed-9b0d-0000626b792a"
Jul 16 21:21:56.037446 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:56.037394    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"
Jul 16 21:21:56.104073 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:56.104044    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e-openshift-service-ca\") pod \"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e\" (UID: \"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e\") "
Jul 16 21:21:56.104243 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:56.104080    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqqqs\" (UniqueName: \"kubernetes.io/projected/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e-kube-api-access-lqqqs\") pod \"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e\" (UID: \"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e\") "
Jul 16 21:21:56.104243 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:56.104198    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e-maas-api-tls\") pod \"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e\" (UID: \"ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e\") "
Jul 16 21:21:56.104475 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:56.104448    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e" (UID: "ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:21:56.106396 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:56.106378    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e" (UID: "ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:21:56.106396 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:56.106384    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e-kube-api-access-lqqqs" (OuterVolumeSpecName: "kube-api-access-lqqqs") pod "ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e" (UID: "ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e"). InnerVolumeSpecName "kube-api-access-lqqqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:21:56.204720 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:56.204693    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e-maas-api-tls\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:21:56.204720 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:56.204715    2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e-openshift-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:21:56.204860 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:56.204724    2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqqqs\" (UniqueName: \"kubernetes.io/projected/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e-kube-api-access-lqqqs\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:21:57.029921 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:57.029892    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"
Jul 16 21:21:57.046576 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:57.046541    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"]
Jul 16 21:21:57.051207 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:57.051184    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ed076e-c975cb44d-44nx2"]
Jul 16 21:21:58.507432 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:58.507398    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e" path="/var/lib/kubelet/pods/ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e/volumes"
Jul 16 21:21:59.523413 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:59.523380    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l"]
Jul 16 21:21:59.523791 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:59.523709    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e" containerName="maas-api"
Jul 16 21:21:59.523791 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:59.523719    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e" containerName="maas-api"
Jul 16 21:21:59.523791 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:59.523781    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca1ab3c6-d5d6-4935-8d04-14e54ae00e6e" containerName="maas-api"
Jul 16 21:21:59.529742 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:59.527472    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l"
Jul 16 21:21:59.535616 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:59.535589    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l"]
Jul 16 21:21:59.631442 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:59.631409    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb88f814-50e8-466a-b2f6-d5d22fa89f38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l\" (UID: \"eb88f814-50e8-466a-b2f6-d5d22fa89f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l"
Jul 16 21:21:59.732025 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:59.731990    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb88f814-50e8-466a-b2f6-d5d22fa89f38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l\" (UID: \"eb88f814-50e8-466a-b2f6-d5d22fa89f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l"
Jul 16 21:21:59.732594 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:59.732577    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb88f814-50e8-466a-b2f6-d5d22fa89f38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l\" (UID: \"eb88f814-50e8-466a-b2f6-d5d22fa89f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l"
Jul 16 21:21:59.843964 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:59.843896    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l"
Jul 16 21:21:59.974163 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:21:59.974138    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l"]
Jul 16 21:21:59.976952 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:21:59.976924    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb88f814_50e8_466a_b2f6_d5d22fa89f38.slice/crio-af53c341964581e4cdfc0f0af26f968f5502a06b669ab1a87e198b60682533dc WatchSource:0}: Error finding container af53c341964581e4cdfc0f0af26f968f5502a06b669ab1a87e198b60682533dc: Status 404 returned error can't find the container with id af53c341964581e4cdfc0f0af26f968f5502a06b669ab1a87e198b60682533dc
Jul 16 21:22:00.043845 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:00.043813    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l" event={"ID":"eb88f814-50e8-466a-b2f6-d5d22fa89f38","Type":"ContainerStarted","Data":"af53c341964581e4cdfc0f0af26f968f5502a06b669ab1a87e198b60682533dc"}
Jul 16 21:22:01.050417 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:01.050386    2530 generic.go:356] "Generic (PLEG): container finished" podID="eb88f814-50e8-466a-b2f6-d5d22fa89f38" containerID="9884716ab7dd591c3f53fc7c71ebfd347b3beb9a386f9e65a7f810c2fe597d52" exitCode=6
Jul 16 21:22:01.050759 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:01.050472    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l" event={"ID":"eb88f814-50e8-466a-b2f6-d5d22fa89f38","Type":"ContainerDied","Data":"9884716ab7dd591c3f53fc7c71ebfd347b3beb9a386f9e65a7f810c2fe597d52"}
Jul 16 21:22:01.050759 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:01.050630    2530 scope.go:117] "RemoveContainer" containerID="9884716ab7dd591c3f53fc7c71ebfd347b3beb9a386f9e65a7f810c2fe597d52"
Jul 16 21:22:02.056840 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:02.056811    2530 generic.go:356] "Generic (PLEG): container finished" podID="eb88f814-50e8-466a-b2f6-d5d22fa89f38" containerID="a6bc6537a641d52a6bc08a3550ca63a5ca32901108dc413d0f01f665ee90c972" exitCode=6
Jul 16 21:22:02.057317 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:02.056863    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l" event={"ID":"eb88f814-50e8-466a-b2f6-d5d22fa89f38","Type":"ContainerDied","Data":"a6bc6537a641d52a6bc08a3550ca63a5ca32901108dc413d0f01f665ee90c972"}
Jul 16 21:22:02.057317 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:02.056917    2530 scope.go:117] "RemoveContainer" containerID="9884716ab7dd591c3f53fc7c71ebfd347b3beb9a386f9e65a7f810c2fe597d52"
Jul 16 21:22:02.057317 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:02.057141    2530 scope.go:117] "RemoveContainer" containerID="a6bc6537a641d52a6bc08a3550ca63a5ca32901108dc413d0f01f665ee90c972"
Jul 16 21:22:02.057495 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:22:02.057391    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l_odh-ai-gateway-infra(eb88f814-50e8-466a-b2f6-d5d22fa89f38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l" podUID="eb88f814-50e8-466a-b2f6-d5d22fa89f38"
Jul 16 21:22:03.062998 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:03.062970    2530 scope.go:117] "RemoveContainer" containerID="a6bc6537a641d52a6bc08a3550ca63a5ca32901108dc413d0f01f665ee90c972"
Jul 16 21:22:03.063482 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:22:03.063218    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l_odh-ai-gateway-infra(eb88f814-50e8-466a-b2f6-d5d22fa89f38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l" podUID="eb88f814-50e8-466a-b2f6-d5d22fa89f38"
Jul 16 21:22:07.502765 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:07.502734    2530 scope.go:117] "RemoveContainer" containerID="95f79fa9bdda8b5475ea9d052d72cbc15ae479531345ebef37b17825908c859b"
Jul 16 21:22:08.088316 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:08.088234    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg" event={"ID":"eb4365dc-49f8-42ed-9b0d-0000626b792a","Type":"ContainerStarted","Data":"cfb0b139a4fc29b5d0dc252f515c8dd3ae8dd99f6ffcd5abba5a87bc00e1d569"}
Jul 16 21:22:09.095534 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:09.095501    2530 generic.go:356] "Generic (PLEG): container finished" podID="eb4365dc-49f8-42ed-9b0d-0000626b792a" containerID="cfb0b139a4fc29b5d0dc252f515c8dd3ae8dd99f6ffcd5abba5a87bc00e1d569" exitCode=6
Jul 16 21:22:09.096021 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:09.095575    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg" event={"ID":"eb4365dc-49f8-42ed-9b0d-0000626b792a","Type":"ContainerDied","Data":"cfb0b139a4fc29b5d0dc252f515c8dd3ae8dd99f6ffcd5abba5a87bc00e1d569"}
Jul 16 21:22:09.096021 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:09.095642    2530 scope.go:117] "RemoveContainer" containerID="95f79fa9bdda8b5475ea9d052d72cbc15ae479531345ebef37b17825908c859b"
Jul 16 21:22:09.096021 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:09.095952    2530 scope.go:117] "RemoveContainer" containerID="cfb0b139a4fc29b5d0dc252f515c8dd3ae8dd99f6ffcd5abba5a87bc00e1d569"
Jul 16 21:22:09.096232 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:22:09.096164    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-p67mg_odh-ai-gateway-infra(eb4365dc-49f8-42ed-9b0d-0000626b792a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg" podUID="eb4365dc-49f8-42ed-9b0d-0000626b792a"
Jul 16 21:22:09.113499 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:09.113475    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg"]
Jul 16 21:22:10.234593 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:10.234570    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg"
Jul 16 21:22:10.314881 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:10.314855    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4365dc-49f8-42ed-9b0d-0000626b792a-maas-api-service-ca\") pod \"eb4365dc-49f8-42ed-9b0d-0000626b792a\" (UID: \"eb4365dc-49f8-42ed-9b0d-0000626b792a\") "
Jul 16 21:22:10.315234 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:10.315212    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb4365dc-49f8-42ed-9b0d-0000626b792a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb4365dc-49f8-42ed-9b0d-0000626b792a" (UID: "eb4365dc-49f8-42ed-9b0d-0000626b792a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:22:10.416426 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:10.416359    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4365dc-49f8-42ed-9b0d-0000626b792a-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:22:10.974681 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:10.974645    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j"]
Jul 16 21:22:10.974979 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:10.974966    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb4365dc-49f8-42ed-9b0d-0000626b792a" containerName="revoke-keys"
Jul 16 21:22:10.974979 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:10.974979    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb4365dc-49f8-42ed-9b0d-0000626b792a" containerName="revoke-keys"
Jul 16 21:22:10.975070 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:10.975002    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb4365dc-49f8-42ed-9b0d-0000626b792a" containerName="revoke-keys"
Jul 16 21:22:10.975070 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:10.975007    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb4365dc-49f8-42ed-9b0d-0000626b792a" containerName="revoke-keys"
Jul 16 21:22:10.975070 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:10.975058    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb4365dc-49f8-42ed-9b0d-0000626b792a" containerName="revoke-keys"
Jul 16 21:22:10.975070 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:10.975067    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb4365dc-49f8-42ed-9b0d-0000626b792a" containerName="revoke-keys"
Jul 16 21:22:10.975252 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:10.975075    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb4365dc-49f8-42ed-9b0d-0000626b792a" containerName="revoke-keys"
Jul 16 21:22:10.978070 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:10.978055    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j"
Jul 16 21:22:10.985020 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:10.984986    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j"]
Jul 16 21:22:11.022263 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:11.022231    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2916edeb-4ced-40b7-8a26-5b2541ce61ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j\" (UID: \"2916edeb-4ced-40b7-8a26-5b2541ce61ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j"
Jul 16 21:22:11.106541 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:11.106510    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg"
Jul 16 21:22:11.106723 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:11.106542    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg" event={"ID":"eb4365dc-49f8-42ed-9b0d-0000626b792a","Type":"ContainerDied","Data":"0f67796d0ad6dc3a503b104f367f8f0130a3ab7433e649313d5fe3c34ad252e8"}
Jul 16 21:22:11.106723 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:11.106601    2530 scope.go:117] "RemoveContainer" containerID="cfb0b139a4fc29b5d0dc252f515c8dd3ae8dd99f6ffcd5abba5a87bc00e1d569"
Jul 16 21:22:11.121475 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:11.121453    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg"]
Jul 16 21:22:11.122909 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:11.122888    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2916edeb-4ced-40b7-8a26-5b2541ce61ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j\" (UID: \"2916edeb-4ced-40b7-8a26-5b2541ce61ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j"
Jul 16 21:22:11.123500 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:11.123480    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2916edeb-4ced-40b7-8a26-5b2541ce61ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j\" (UID: \"2916edeb-4ced-40b7-8a26-5b2541ce61ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j"
Jul 16 21:22:11.128456 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:11.128437    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-p67mg"]
Jul 16 21:22:11.290085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:11.290028    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j"
Jul 16 21:22:11.425874 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:11.425825    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j"]
Jul 16 21:22:11.428326 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:22:11.428295    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2916edeb_4ced_40b7_8a26_5b2541ce61ed.slice/crio-6bb33d81c6ef1e1c8f188cc4b8800ffa649bfdbc4b8ef4e48ad7a5000ea30b4c WatchSource:0}: Error finding container 6bb33d81c6ef1e1c8f188cc4b8800ffa649bfdbc4b8ef4e48ad7a5000ea30b4c: Status 404 returned error can't find the container with id 6bb33d81c6ef1e1c8f188cc4b8800ffa649bfdbc4b8ef4e48ad7a5000ea30b4c
Jul 16 21:22:12.113613 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:12.113578    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j" event={"ID":"2916edeb-4ced-40b7-8a26-5b2541ce61ed","Type":"ContainerStarted","Data":"ace035c90c13fe5a85c11f9d6fa7e6e097b2c52dea328dd8149a8b271c7a2e80"}
Jul 16 21:22:12.113787 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:12.113629    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j" event={"ID":"2916edeb-4ced-40b7-8a26-5b2541ce61ed","Type":"ContainerStarted","Data":"6bb33d81c6ef1e1c8f188cc4b8800ffa649bfdbc4b8ef4e48ad7a5000ea30b4c"}
Jul 16 21:22:12.128958 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:12.128918    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j" podStartSLOduration=2.128903392 podStartE2EDuration="2.128903392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:22:12.126950127 +0000 UTC m=+2970.148537884" watchObservedRunningTime="2026-07-16 21:22:12.128903392 +0000 UTC m=+2970.150491159"
Jul 16 21:22:12.507897 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:12.507867    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb4365dc-49f8-42ed-9b0d-0000626b792a" path="/var/lib/kubelet/pods/eb4365dc-49f8-42ed-9b0d-0000626b792a/volumes"
Jul 16 21:22:12.631778 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:12.631746    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt"]
Jul 16 21:22:12.632123 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:12.632094    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb4365dc-49f8-42ed-9b0d-0000626b792a" containerName="revoke-keys"
Jul 16 21:22:12.632123 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:12.632121    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb4365dc-49f8-42ed-9b0d-0000626b792a" containerName="revoke-keys"
Jul 16 21:22:12.635169 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:12.635154    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt"
Jul 16 21:22:12.642431 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:12.642408    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt"]
Jul 16 21:22:12.738395 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:12.738369    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa0f7cd-e6e6-4084-b636-cd2e91651cfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt\" (UID: \"4aa0f7cd-e6e6-4084-b636-cd2e91651cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt"
Jul 16 21:22:12.839387 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:12.839321    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa0f7cd-e6e6-4084-b636-cd2e91651cfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt\" (UID: \"4aa0f7cd-e6e6-4084-b636-cd2e91651cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt"
Jul 16 21:22:12.839932 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:12.839909    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa0f7cd-e6e6-4084-b636-cd2e91651cfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt\" (UID: \"4aa0f7cd-e6e6-4084-b636-cd2e91651cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt"
Jul 16 21:22:12.945919 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:12.945896    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt"
Jul 16 21:22:13.075033 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:13.075010    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt"]
Jul 16 21:22:13.076982 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:22:13.076954    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa0f7cd_e6e6_4084_b636_cd2e91651cfe.slice/crio-77ae56a48d8fe0f05559ecfcca649818e3033bea5aeed857eafb37bd7f4f5cf4 WatchSource:0}: Error finding container 77ae56a48d8fe0f05559ecfcca649818e3033bea5aeed857eafb37bd7f4f5cf4: Status 404 returned error can't find the container with id 77ae56a48d8fe0f05559ecfcca649818e3033bea5aeed857eafb37bd7f4f5cf4
Jul 16 21:22:13.118723 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:13.118699    2530 generic.go:356] "Generic (PLEG): container finished" podID="2916edeb-4ced-40b7-8a26-5b2541ce61ed" containerID="ace035c90c13fe5a85c11f9d6fa7e6e097b2c52dea328dd8149a8b271c7a2e80" exitCode=0
Jul 16 21:22:13.118807 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:13.118767    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j" event={"ID":"2916edeb-4ced-40b7-8a26-5b2541ce61ed","Type":"ContainerDied","Data":"ace035c90c13fe5a85c11f9d6fa7e6e097b2c52dea328dd8149a8b271c7a2e80"}
Jul 16 21:22:13.119943 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:13.119921    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt" event={"ID":"4aa0f7cd-e6e6-4084-b636-cd2e91651cfe","Type":"ContainerStarted","Data":"77ae56a48d8fe0f05559ecfcca649818e3033bea5aeed857eafb37bd7f4f5cf4"}
Jul 16 21:22:14.125535 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:14.125497    2530 generic.go:356] "Generic (PLEG): container finished" podID="4aa0f7cd-e6e6-4084-b636-cd2e91651cfe" containerID="62068cee8fa5055a8b48c7e8cdc657870af17e660ba74653d6636d54dc377110" exitCode=6
Jul 16 21:22:14.125940 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:14.125579    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt" event={"ID":"4aa0f7cd-e6e6-4084-b636-cd2e91651cfe","Type":"ContainerDied","Data":"62068cee8fa5055a8b48c7e8cdc657870af17e660ba74653d6636d54dc377110"}
Jul 16 21:22:14.125940 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:14.125761    2530 scope.go:117] "RemoveContainer" containerID="62068cee8fa5055a8b48c7e8cdc657870af17e660ba74653d6636d54dc377110"
Jul 16 21:22:14.335045 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:14.335023    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j"
Jul 16 21:22:14.449259 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:14.449193    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2916edeb-4ced-40b7-8a26-5b2541ce61ed-maas-api-service-ca\") pod \"2916edeb-4ced-40b7-8a26-5b2541ce61ed\" (UID: \"2916edeb-4ced-40b7-8a26-5b2541ce61ed\") "
Jul 16 21:22:14.449512 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:14.449491    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2916edeb-4ced-40b7-8a26-5b2541ce61ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2916edeb-4ced-40b7-8a26-5b2541ce61ed" (UID: "2916edeb-4ced-40b7-8a26-5b2541ce61ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:22:14.511229 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:14.511208    2530 scope.go:117] "RemoveContainer" containerID="a6bc6537a641d52a6bc08a3550ca63a5ca32901108dc413d0f01f665ee90c972"
Jul 16 21:22:14.550307 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:14.550286    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2916edeb-4ced-40b7-8a26-5b2541ce61ed-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:22:15.132129 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:15.132023    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j" event={"ID":"2916edeb-4ced-40b7-8a26-5b2541ce61ed","Type":"ContainerDied","Data":"6bb33d81c6ef1e1c8f188cc4b8800ffa649bfdbc4b8ef4e48ad7a5000ea30b4c"}
Jul 16 21:22:15.132129 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:15.132088    2530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bb33d81c6ef1e1c8f188cc4b8800ffa649bfdbc4b8ef4e48ad7a5000ea30b4c"
Jul 16 21:22:15.132129 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:15.132035    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j"
Jul 16 21:22:15.136619 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:15.136589    2530 generic.go:356] "Generic (PLEG): container finished" podID="4aa0f7cd-e6e6-4084-b636-cd2e91651cfe" containerID="0e9d619aec5f6a6002fafd3127bb98ac5795078cd78a8f0ac3c1e1e0c941d94e" exitCode=6
Jul 16 21:22:15.136779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:15.136679    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt" event={"ID":"4aa0f7cd-e6e6-4084-b636-cd2e91651cfe","Type":"ContainerDied","Data":"0e9d619aec5f6a6002fafd3127bb98ac5795078cd78a8f0ac3c1e1e0c941d94e"}
Jul 16 21:22:15.136779 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:15.136739    2530 scope.go:117] "RemoveContainer" containerID="62068cee8fa5055a8b48c7e8cdc657870af17e660ba74653d6636d54dc377110"
Jul 16 21:22:15.136915 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:15.136900    2530 scope.go:117] "RemoveContainer" containerID="0e9d619aec5f6a6002fafd3127bb98ac5795078cd78a8f0ac3c1e1e0c941d94e"
Jul 16 21:22:15.137115 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:22:15.137077    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt_odh-ai-gateway-infra(4aa0f7cd-e6e6-4084-b636-cd2e91651cfe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt" podUID="4aa0f7cd-e6e6-4084-b636-cd2e91651cfe"
Jul 16 21:22:15.139058 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:15.139036    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l" event={"ID":"eb88f814-50e8-466a-b2f6-d5d22fa89f38","Type":"ContainerStarted","Data":"9020983df9b1317491e1d21d752657319e5e3a4cd571e2d1e06264533c7bf346"}
Jul 16 21:22:15.171961 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:15.171915    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l" podStartSLOduration=16.171903827 podStartE2EDuration="16.171903827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:22:15.171150706 +0000 UTC m=+2973.192738477" watchObservedRunningTime="2026-07-16 21:22:15.171903827 +0000 UTC m=+2973.193491593"
Jul 16 21:22:16.145644 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:16.145616    2530 scope.go:117] "RemoveContainer" containerID="0e9d619aec5f6a6002fafd3127bb98ac5795078cd78a8f0ac3c1e1e0c941d94e"
Jul 16 21:22:16.146077 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:22:16.145835    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt_odh-ai-gateway-infra(4aa0f7cd-e6e6-4084-b636-cd2e91651cfe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt" podUID="4aa0f7cd-e6e6-4084-b636-cd2e91651cfe"
Jul 16 21:22:16.147278 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:16.147251    2530 generic.go:356] "Generic (PLEG): container finished" podID="eb88f814-50e8-466a-b2f6-d5d22fa89f38" containerID="9020983df9b1317491e1d21d752657319e5e3a4cd571e2d1e06264533c7bf346" exitCode=6
Jul 16 21:22:16.147398 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:16.147312    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l" event={"ID":"eb88f814-50e8-466a-b2f6-d5d22fa89f38","Type":"ContainerDied","Data":"9020983df9b1317491e1d21d752657319e5e3a4cd571e2d1e06264533c7bf346"}
Jul 16 21:22:16.147398 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:16.147364    2530 scope.go:117] "RemoveContainer" containerID="a6bc6537a641d52a6bc08a3550ca63a5ca32901108dc413d0f01f665ee90c972"
Jul 16 21:22:16.147605 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:16.147590    2530 scope.go:117] "RemoveContainer" containerID="9020983df9b1317491e1d21d752657319e5e3a4cd571e2d1e06264533c7bf346"
Jul 16 21:22:16.147799 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:22:16.147781    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l_odh-ai-gateway-infra(eb88f814-50e8-466a-b2f6-d5d22fa89f38)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l" podUID="eb88f814-50e8-466a-b2f6-d5d22fa89f38"
Jul 16 21:22:16.188362 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:16.188342    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l"]
Jul 16 21:22:17.283345 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:17.283316    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l"
Jul 16 21:22:17.373729 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:17.373704    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb88f814-50e8-466a-b2f6-d5d22fa89f38-maas-api-service-ca\") pod \"eb88f814-50e8-466a-b2f6-d5d22fa89f38\" (UID: \"eb88f814-50e8-466a-b2f6-d5d22fa89f38\") "
Jul 16 21:22:17.374034 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:17.374014    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb88f814-50e8-466a-b2f6-d5d22fa89f38-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb88f814-50e8-466a-b2f6-d5d22fa89f38" (UID: "eb88f814-50e8-466a-b2f6-d5d22fa89f38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:22:17.475137 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:17.475094    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb88f814-50e8-466a-b2f6-d5d22fa89f38-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:22:18.158654 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:18.158623    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l"
Jul 16 21:22:18.158810 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:18.158665    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l" event={"ID":"eb88f814-50e8-466a-b2f6-d5d22fa89f38","Type":"ContainerDied","Data":"af53c341964581e4cdfc0f0af26f968f5502a06b669ab1a87e198b60682533dc"}
Jul 16 21:22:18.158810 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:18.158728    2530 scope.go:117] "RemoveContainer" containerID="9020983df9b1317491e1d21d752657319e5e3a4cd571e2d1e06264533c7bf346"
Jul 16 21:22:18.180421 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:18.180395    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l"]
Jul 16 21:22:18.187234 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:18.187213    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-gp27l"]
Jul 16 21:22:18.508021 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:18.507993    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb88f814-50e8-466a-b2f6-d5d22fa89f38" path="/var/lib/kubelet/pods/eb88f814-50e8-466a-b2f6-d5d22fa89f38/volumes"
Jul 16 21:22:21.401206 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:21.401171    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"]
Jul 16 21:22:21.401697 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:21.401382    2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf" podUID="cfa5eb32-916f-4577-b474-12917f1bbd75" containerName="maas-api" containerID="cri-o://105b8ad9075630ce6939a663124e04cc0fc776fa781db10999a4bbe5630fa359" gracePeriod=30
Jul 16 21:22:21.657194 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:21.657137    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"
Jul 16 21:22:21.707921 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:21.707885    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2fbj\" (UniqueName: \"kubernetes.io/projected/cfa5eb32-916f-4577-b474-12917f1bbd75-kube-api-access-p2fbj\") pod \"cfa5eb32-916f-4577-b474-12917f1bbd75\" (UID: \"cfa5eb32-916f-4577-b474-12917f1bbd75\") "
Jul 16 21:22:21.707921 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:21.707921    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa5eb32-916f-4577-b474-12917f1bbd75-openshift-service-ca\") pod \"cfa5eb32-916f-4577-b474-12917f1bbd75\" (UID: \"cfa5eb32-916f-4577-b474-12917f1bbd75\") "
Jul 16 21:22:21.708144 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:21.707970    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa5eb32-916f-4577-b474-12917f1bbd75-maas-api-tls\") pod \"cfa5eb32-916f-4577-b474-12917f1bbd75\" (UID: \"cfa5eb32-916f-4577-b474-12917f1bbd75\") "
Jul 16 21:22:21.708348 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:21.708320    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa5eb32-916f-4577-b474-12917f1bbd75-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cfa5eb32-916f-4577-b474-12917f1bbd75" (UID: "cfa5eb32-916f-4577-b474-12917f1bbd75"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:22:21.710070 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:21.710046    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa5eb32-916f-4577-b474-12917f1bbd75-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cfa5eb32-916f-4577-b474-12917f1bbd75" (UID: "cfa5eb32-916f-4577-b474-12917f1bbd75"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:22:21.710070 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:21.710062    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa5eb32-916f-4577-b474-12917f1bbd75-kube-api-access-p2fbj" (OuterVolumeSpecName: "kube-api-access-p2fbj") pod "cfa5eb32-916f-4577-b474-12917f1bbd75" (UID: "cfa5eb32-916f-4577-b474-12917f1bbd75"). InnerVolumeSpecName "kube-api-access-p2fbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:22:21.809418 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:21.809391    2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2fbj\" (UniqueName: \"kubernetes.io/projected/cfa5eb32-916f-4577-b474-12917f1bbd75-kube-api-access-p2fbj\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:22:21.809418 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:21.809417    2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa5eb32-916f-4577-b474-12917f1bbd75-openshift-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:22:21.809576 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:21.809427    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa5eb32-916f-4577-b474-12917f1bbd75-maas-api-tls\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:22:22.185744 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:22.185710    2530 generic.go:356] "Generic (PLEG): container finished" podID="cfa5eb32-916f-4577-b474-12917f1bbd75" containerID="105b8ad9075630ce6939a663124e04cc0fc776fa781db10999a4bbe5630fa359" exitCode=0
Jul 16 21:22:22.185899 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:22.185754    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"
Jul 16 21:22:22.185899 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:22.185792    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf" event={"ID":"cfa5eb32-916f-4577-b474-12917f1bbd75","Type":"ContainerDied","Data":"105b8ad9075630ce6939a663124e04cc0fc776fa781db10999a4bbe5630fa359"}
Jul 16 21:22:22.185899 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:22.185842    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf" event={"ID":"cfa5eb32-916f-4577-b474-12917f1bbd75","Type":"ContainerDied","Data":"8d072e347dca39118291c48749950479e0964cc262e5d977ac8606688518f3dc"}
Jul 16 21:22:22.185899 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:22.185864    2530 scope.go:117] "RemoveContainer" containerID="105b8ad9075630ce6939a663124e04cc0fc776fa781db10999a4bbe5630fa359"
Jul 16 21:22:22.194660 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:22.194648    2530 scope.go:117] "RemoveContainer" containerID="105b8ad9075630ce6939a663124e04cc0fc776fa781db10999a4bbe5630fa359"
Jul 16 21:22:22.194919 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:22:22.194900    2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"105b8ad9075630ce6939a663124e04cc0fc776fa781db10999a4bbe5630fa359\": container with ID starting with 105b8ad9075630ce6939a663124e04cc0fc776fa781db10999a4bbe5630fa359 not found: ID does not exist" containerID="105b8ad9075630ce6939a663124e04cc0fc776fa781db10999a4bbe5630fa359"
Jul 16 21:22:22.194966 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:22.194927    2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105b8ad9075630ce6939a663124e04cc0fc776fa781db10999a4bbe5630fa359"} err="failed to get container status \"105b8ad9075630ce6939a663124e04cc0fc776fa781db10999a4bbe5630fa359\": rpc error: code = NotFound desc = could not find container \"105b8ad9075630ce6939a663124e04cc0fc776fa781db10999a4bbe5630fa359\": container with ID starting with 105b8ad9075630ce6939a663124e04cc0fc776fa781db10999a4bbe5630fa359 not found: ID does not exist"
Jul 16 21:22:22.206838 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:22.206817    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"]
Jul 16 21:22:22.210489 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:22.210471    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-df0daf-75c7f576c9-rqgnf"]
Jul 16 21:22:22.507396 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:22.507365    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfa5eb32-916f-4577-b474-12917f1bbd75" path="/var/lib/kubelet/pods/cfa5eb32-916f-4577-b474-12917f1bbd75/volumes"
Jul 16 21:22:27.502026 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:27.501992    2530 scope.go:117] "RemoveContainer" containerID="0e9d619aec5f6a6002fafd3127bb98ac5795078cd78a8f0ac3c1e1e0c941d94e"
Jul 16 21:22:28.216075 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:28.216044    2530 generic.go:356] "Generic (PLEG): container finished" podID="4aa0f7cd-e6e6-4084-b636-cd2e91651cfe" containerID="745e5ac04c715dd75a89c73c9f01c011fed591ac1c05c188d8adbecc50172129" exitCode=6
Jul 16 21:22:28.216249 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:28.216141    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt" event={"ID":"4aa0f7cd-e6e6-4084-b636-cd2e91651cfe","Type":"ContainerDied","Data":"745e5ac04c715dd75a89c73c9f01c011fed591ac1c05c188d8adbecc50172129"}
Jul 16 21:22:28.216249 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:28.216218    2530 scope.go:117] "RemoveContainer" containerID="0e9d619aec5f6a6002fafd3127bb98ac5795078cd78a8f0ac3c1e1e0c941d94e"
Jul 16 21:22:28.216549 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:28.216531    2530 scope.go:117] "RemoveContainer" containerID="745e5ac04c715dd75a89c73c9f01c011fed591ac1c05c188d8adbecc50172129"
Jul 16 21:22:28.216746 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:22:28.216727    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt_odh-ai-gateway-infra(4aa0f7cd-e6e6-4084-b636-cd2e91651cfe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt" podUID="4aa0f7cd-e6e6-4084-b636-cd2e91651cfe"
Jul 16 21:22:29.239505 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:29.239476    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt"]
Jul 16 21:22:29.375139 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:29.375089    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt"
Jul 16 21:22:29.473014 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:29.472987    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa0f7cd-e6e6-4084-b636-cd2e91651cfe-maas-api-service-ca\") pod \"4aa0f7cd-e6e6-4084-b636-cd2e91651cfe\" (UID: \"4aa0f7cd-e6e6-4084-b636-cd2e91651cfe\") "
Jul 16 21:22:29.473335 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:29.473308    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa0f7cd-e6e6-4084-b636-cd2e91651cfe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4aa0f7cd-e6e6-4084-b636-cd2e91651cfe" (UID: "4aa0f7cd-e6e6-4084-b636-cd2e91651cfe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:22:29.574196 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:29.574129    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa0f7cd-e6e6-4084-b636-cd2e91651cfe-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:22:30.228797 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:30.228760    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt" event={"ID":"4aa0f7cd-e6e6-4084-b636-cd2e91651cfe","Type":"ContainerDied","Data":"77ae56a48d8fe0f05559ecfcca649818e3033bea5aeed857eafb37bd7f4f5cf4"}
Jul 16 21:22:30.228979 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:30.228811    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt"
Jul 16 21:22:30.228979 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:30.228819    2530 scope.go:117] "RemoveContainer" containerID="745e5ac04c715dd75a89c73c9f01c011fed591ac1c05c188d8adbecc50172129"
Jul 16 21:22:30.251720 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:30.251695    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt"]
Jul 16 21:22:30.258583 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:30.258562    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-lxhgt"]
Jul 16 21:22:30.507409 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:30.507345    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aa0f7cd-e6e6-4084-b636-cd2e91651cfe" path="/var/lib/kubelet/pods/4aa0f7cd-e6e6-4084-b636-cd2e91651cfe/volumes"
Jul 16 21:22:40.827313 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827278    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"]
Jul 16 21:22:40.827824 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827764    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa5eb32-916f-4577-b474-12917f1bbd75" containerName="maas-api"
Jul 16 21:22:40.827824 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827787    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa5eb32-916f-4577-b474-12917f1bbd75" containerName="maas-api"
Jul 16 21:22:40.827824 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827802    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2916edeb-4ced-40b7-8a26-5b2541ce61ed" containerName="revoke-keys"
Jul 16 21:22:40.827824 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827810    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="2916edeb-4ced-40b7-8a26-5b2541ce61ed" containerName="revoke-keys"
Jul 16 21:22:40.827824 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827822    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb88f814-50e8-466a-b2f6-d5d22fa89f38" containerName="revoke-keys"
Jul 16 21:22:40.828000 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827830    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb88f814-50e8-466a-b2f6-d5d22fa89f38" containerName="revoke-keys"
Jul 16 21:22:40.828000 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827850    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa0f7cd-e6e6-4084-b636-cd2e91651cfe" containerName="revoke-keys"
Jul 16 21:22:40.828000 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827858    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa0f7cd-e6e6-4084-b636-cd2e91651cfe" containerName="revoke-keys"
Jul 16 21:22:40.828000 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827867    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb88f814-50e8-466a-b2f6-d5d22fa89f38" containerName="revoke-keys"
Jul 16 21:22:40.828000 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827872    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb88f814-50e8-466a-b2f6-d5d22fa89f38" containerName="revoke-keys"
Jul 16 21:22:40.828000 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827882    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa0f7cd-e6e6-4084-b636-cd2e91651cfe" containerName="revoke-keys"
Jul 16 21:22:40.828000 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827888    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa0f7cd-e6e6-4084-b636-cd2e91651cfe" containerName="revoke-keys"
Jul 16 21:22:40.828000 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827942    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="2916edeb-4ced-40b7-8a26-5b2541ce61ed" containerName="revoke-keys"
Jul 16 21:22:40.828000 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827949    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb88f814-50e8-466a-b2f6-d5d22fa89f38" containerName="revoke-keys"
Jul 16 21:22:40.828000 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827957    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aa0f7cd-e6e6-4084-b636-cd2e91651cfe" containerName="revoke-keys"
Jul 16 21:22:40.828000 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827963    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aa0f7cd-e6e6-4084-b636-cd2e91651cfe" containerName="revoke-keys"
Jul 16 21:22:40.828000 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827969    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb88f814-50e8-466a-b2f6-d5d22fa89f38" containerName="revoke-keys"
Jul 16 21:22:40.828000 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827977    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aa0f7cd-e6e6-4084-b636-cd2e91651cfe" containerName="revoke-keys"
Jul 16 21:22:40.828000 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.827986    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfa5eb32-916f-4577-b474-12917f1bbd75" containerName="maas-api"
Jul 16 21:22:40.832044 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.832021    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"
Jul 16 21:22:40.834662 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.834635    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-9331e3-serving-cert\""
Jul 16 21:22:40.849269 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.849223    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"]
Jul 16 21:22:40.967356 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.967322    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eee1489-6c54-4c80-957f-3de730a58320-maas-api-tls\") pod \"maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5\" (UID: \"4eee1489-6c54-4c80-957f-3de730a58320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"
Jul 16 21:22:40.967356 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.967357    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eee1489-6c54-4c80-957f-3de730a58320-openshift-service-ca\") pod \"maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5\" (UID: \"4eee1489-6c54-4c80-957f-3de730a58320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"
Jul 16 21:22:40.967559 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:40.967445    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6wdl\" (UniqueName: \"kubernetes.io/projected/4eee1489-6c54-4c80-957f-3de730a58320-kube-api-access-l6wdl\") pod \"maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5\" (UID: \"4eee1489-6c54-4c80-957f-3de730a58320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"
Jul 16 21:22:41.068749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:41.068718    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6wdl\" (UniqueName: \"kubernetes.io/projected/4eee1489-6c54-4c80-957f-3de730a58320-kube-api-access-l6wdl\") pod \"maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5\" (UID: \"4eee1489-6c54-4c80-957f-3de730a58320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"
Jul 16 21:22:41.068925 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:41.068774    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eee1489-6c54-4c80-957f-3de730a58320-maas-api-tls\") pod \"maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5\" (UID: \"4eee1489-6c54-4c80-957f-3de730a58320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"
Jul 16 21:22:41.068925 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:41.068801    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eee1489-6c54-4c80-957f-3de730a58320-openshift-service-ca\") pod \"maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5\" (UID: \"4eee1489-6c54-4c80-957f-3de730a58320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"
Jul 16 21:22:41.069539 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:41.069522    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eee1489-6c54-4c80-957f-3de730a58320-openshift-service-ca\") pod \"maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5\" (UID: \"4eee1489-6c54-4c80-957f-3de730a58320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"
Jul 16 21:22:41.071506 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:41.071486    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eee1489-6c54-4c80-957f-3de730a58320-maas-api-tls\") pod \"maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5\" (UID: \"4eee1489-6c54-4c80-957f-3de730a58320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"
Jul 16 21:22:41.077798 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:41.077742    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6wdl\" (UniqueName: \"kubernetes.io/projected/4eee1489-6c54-4c80-957f-3de730a58320-kube-api-access-l6wdl\") pod \"maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5\" (UID: \"4eee1489-6c54-4c80-957f-3de730a58320\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"
Jul 16 21:22:41.145329 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:41.145300    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"
Jul 16 21:22:41.298486 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:41.298462    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"]
Jul 16 21:22:41.299736 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:22:41.299710    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eee1489_6c54_4c80_957f_3de730a58320.slice/crio-505d5c5ad0acd81c499d171cec37afcd6b0107984552d0acbac4afe9e69ba58e WatchSource:0}: Error finding container 505d5c5ad0acd81c499d171cec37afcd6b0107984552d0acbac4afe9e69ba58e: Status 404 returned error can't find the container with id 505d5c5ad0acd81c499d171cec37afcd6b0107984552d0acbac4afe9e69ba58e
Jul 16 21:22:42.149685 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.149651    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5"]
Jul 16 21:22:42.150060 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.149992    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb88f814-50e8-466a-b2f6-d5d22fa89f38" containerName="revoke-keys"
Jul 16 21:22:42.150060 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.150003    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb88f814-50e8-466a-b2f6-d5d22fa89f38" containerName="revoke-keys"
Jul 16 21:22:42.150060 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.150018    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aa0f7cd-e6e6-4084-b636-cd2e91651cfe" containerName="revoke-keys"
Jul 16 21:22:42.150060 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.150028    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aa0f7cd-e6e6-4084-b636-cd2e91651cfe" containerName="revoke-keys"
Jul 16 21:22:42.150213 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.150115    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb88f814-50e8-466a-b2f6-d5d22fa89f38" containerName="revoke-keys"
Jul 16 21:22:42.153113 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.153081    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5"
Jul 16 21:22:42.155360 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.155342    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:22:42.161006 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.160987    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5"]
Jul 16 21:22:42.279450 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.279422    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80cd16aa-13ab-48ad-ad60-6b8b7d270a9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5\" (UID: \"80cd16aa-13ab-48ad-ad60-6b8b7d270a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5"
Jul 16 21:22:42.286937 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.286904    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5" event={"ID":"4eee1489-6c54-4c80-957f-3de730a58320","Type":"ContainerStarted","Data":"bc3c34a7156cc889a503abaf715d1a4f64848ec36520a6cf305bc6e3e842810b"}
Jul 16 21:22:42.287075 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.286954    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5" event={"ID":"4eee1489-6c54-4c80-957f-3de730a58320","Type":"ContainerStarted","Data":"505d5c5ad0acd81c499d171cec37afcd6b0107984552d0acbac4afe9e69ba58e"}
Jul 16 21:22:42.287075 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.286988    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"
Jul 16 21:22:42.304412 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.304363    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5" podStartSLOduration=1.91912934 podStartE2EDuration="2.304344749s" totalImagesPullingTime="385.215409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:22:42.303223254 +0000 UTC m=+3000.324811061" watchObservedRunningTime="2026-07-16 21:22:42.304344749 +0000 UTC m=+3000.325932517"
Jul 16 21:22:42.380013 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.379981    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80cd16aa-13ab-48ad-ad60-6b8b7d270a9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5\" (UID: \"80cd16aa-13ab-48ad-ad60-6b8b7d270a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5"
Jul 16 21:22:42.380819 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.380792    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80cd16aa-13ab-48ad-ad60-6b8b7d270a9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5\" (UID: \"80cd16aa-13ab-48ad-ad60-6b8b7d270a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5"
Jul 16 21:22:42.468044 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.468021    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:22:42.475856 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.475838    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5"
Jul 16 21:22:42.611043 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:42.611020    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5"]
Jul 16 21:22:42.613379 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:22:42.613351    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80cd16aa_13ab_48ad_ad60_6b8b7d270a9f.slice/crio-b9e864b152cf7d5d9d51ce28d0c03341ea9df680397e10b3aecd34dbea9af020 WatchSource:0}: Error finding container b9e864b152cf7d5d9d51ce28d0c03341ea9df680397e10b3aecd34dbea9af020: Status 404 returned error can't find the container with id b9e864b152cf7d5d9d51ce28d0c03341ea9df680397e10b3aecd34dbea9af020
Jul 16 21:22:43.293049 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:43.293015    2530 generic.go:356] "Generic (PLEG): container finished" podID="80cd16aa-13ab-48ad-ad60-6b8b7d270a9f" containerID="91a9447f1a0ab4866cb5e973cb20792cf15c7e12958f7c682496b144080d9432" exitCode=6
Jul 16 21:22:43.293380 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:43.293073    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5" event={"ID":"80cd16aa-13ab-48ad-ad60-6b8b7d270a9f","Type":"ContainerDied","Data":"91a9447f1a0ab4866cb5e973cb20792cf15c7e12958f7c682496b144080d9432"}
Jul 16 21:22:43.293380 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:43.293172    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5" event={"ID":"80cd16aa-13ab-48ad-ad60-6b8b7d270a9f","Type":"ContainerStarted","Data":"b9e864b152cf7d5d9d51ce28d0c03341ea9df680397e10b3aecd34dbea9af020"}
Jul 16 21:22:43.293380 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:43.293237    2530 scope.go:117] "RemoveContainer" containerID="91a9447f1a0ab4866cb5e973cb20792cf15c7e12958f7c682496b144080d9432"
Jul 16 21:22:44.299987 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:44.299958    2530 generic.go:356] "Generic (PLEG): container finished" podID="80cd16aa-13ab-48ad-ad60-6b8b7d270a9f" containerID="c01a371c4ffa794cee47ca3583224feed51a4ae35a7c1dd7a2261f73326fd772" exitCode=6
Jul 16 21:22:44.300394 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:44.300043    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5" event={"ID":"80cd16aa-13ab-48ad-ad60-6b8b7d270a9f","Type":"ContainerDied","Data":"c01a371c4ffa794cee47ca3583224feed51a4ae35a7c1dd7a2261f73326fd772"}
Jul 16 21:22:44.300394 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:44.300126    2530 scope.go:117] "RemoveContainer" containerID="91a9447f1a0ab4866cb5e973cb20792cf15c7e12958f7c682496b144080d9432"
Jul 16 21:22:44.300394 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:44.300267    2530 scope.go:117] "RemoveContainer" containerID="c01a371c4ffa794cee47ca3583224feed51a4ae35a7c1dd7a2261f73326fd772"
Jul 16 21:22:44.300529 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:22:44.300441    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5_odh-ai-gateway-infra(80cd16aa-13ab-48ad-ad60-6b8b7d270a9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5" podUID="80cd16aa-13ab-48ad-ad60-6b8b7d270a9f"
Jul 16 21:22:45.306849 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:45.306823    2530 scope.go:117] "RemoveContainer" containerID="c01a371c4ffa794cee47ca3583224feed51a4ae35a7c1dd7a2261f73326fd772"
Jul 16 21:22:45.307343 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:22:45.307048    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5_odh-ai-gateway-infra(80cd16aa-13ab-48ad-ad60-6b8b7d270a9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5" podUID="80cd16aa-13ab-48ad-ad60-6b8b7d270a9f"
Jul 16 21:22:48.298336 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:48.298306    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"
Jul 16 21:22:48.871334 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:48.871299    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"]
Jul 16 21:22:48.876260 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:48.876238    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"
Jul 16 21:22:48.878754 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:48.878731    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-fb5e77-serving-cert\""
Jul 16 21:22:48.886869 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:48.886844    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"]
Jul 16 21:22:49.031852 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:49.031824    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab39e48-0068-48e4-8441-d5b4a000e9ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg\" (UID: \"7ab39e48-0068-48e4-8441-d5b4a000e9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"
Jul 16 21:22:49.031992 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:49.031865    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ckn5\" (UniqueName: \"kubernetes.io/projected/7ab39e48-0068-48e4-8441-d5b4a000e9ff-kube-api-access-8ckn5\") pod \"maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg\" (UID: \"7ab39e48-0068-48e4-8441-d5b4a000e9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"
Jul 16 21:22:49.031992 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:49.031903    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab39e48-0068-48e4-8441-d5b4a000e9ff-openshift-service-ca\") pod \"maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg\" (UID: \"7ab39e48-0068-48e4-8441-d5b4a000e9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"
Jul 16 21:22:49.132730 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:49.132668    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab39e48-0068-48e4-8441-d5b4a000e9ff-openshift-service-ca\") pod \"maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg\" (UID: \"7ab39e48-0068-48e4-8441-d5b4a000e9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"
Jul 16 21:22:49.132856 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:49.132735    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab39e48-0068-48e4-8441-d5b4a000e9ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg\" (UID: \"7ab39e48-0068-48e4-8441-d5b4a000e9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"
Jul 16 21:22:49.132856 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:49.132759    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ckn5\" (UniqueName: \"kubernetes.io/projected/7ab39e48-0068-48e4-8441-d5b4a000e9ff-kube-api-access-8ckn5\") pod \"maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg\" (UID: \"7ab39e48-0068-48e4-8441-d5b4a000e9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"
Jul 16 21:22:49.133307 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:49.133289    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab39e48-0068-48e4-8441-d5b4a000e9ff-openshift-service-ca\") pod \"maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg\" (UID: \"7ab39e48-0068-48e4-8441-d5b4a000e9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"
Jul 16 21:22:49.135350 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:49.135332    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab39e48-0068-48e4-8441-d5b4a000e9ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg\" (UID: \"7ab39e48-0068-48e4-8441-d5b4a000e9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"
Jul 16 21:22:49.143379 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:49.143356    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ckn5\" (UniqueName: \"kubernetes.io/projected/7ab39e48-0068-48e4-8441-d5b4a000e9ff-kube-api-access-8ckn5\") pod \"maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg\" (UID: \"7ab39e48-0068-48e4-8441-d5b4a000e9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"
Jul 16 21:22:49.186911 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:49.186889    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"
Jul 16 21:22:49.324971 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:49.324950    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"]
Jul 16 21:22:49.326490 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:22:49.326464    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab39e48_0068_48e4_8441_d5b4a000e9ff.slice/crio-f787ecf4ad4d7ccd44cd17e5f42b0c45c8234ace4777b61851e4dd843807cf80 WatchSource:0}: Error finding container f787ecf4ad4d7ccd44cd17e5f42b0c45c8234ace4777b61851e4dd843807cf80: Status 404 returned error can't find the container with id f787ecf4ad4d7ccd44cd17e5f42b0c45c8234ace4777b61851e4dd843807cf80
Jul 16 21:22:49.569791 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:49.569759    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld"]
Jul 16 21:22:49.574355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:49.574332    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld"
Jul 16 21:22:49.582856 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:49.582834    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld"]
Jul 16 21:22:49.737487 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:49.737461    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d0b32e-2b8b-47f7-9a4f-9386b218aa9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld\" (UID: \"68d0b32e-2b8b-47f7-9a4f-9386b218aa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld"
Jul 16 21:22:49.838314 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:49.838255    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d0b32e-2b8b-47f7-9a4f-9386b218aa9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld\" (UID: \"68d0b32e-2b8b-47f7-9a4f-9386b218aa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld"
Jul 16 21:22:49.838843 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:49.838818    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d0b32e-2b8b-47f7-9a4f-9386b218aa9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld\" (UID: \"68d0b32e-2b8b-47f7-9a4f-9386b218aa9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld"
Jul 16 21:22:49.885407 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:49.885372    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld"
Jul 16 21:22:50.020784 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:50.020757    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld"]
Jul 16 21:22:50.022654 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:22:50.022627    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d0b32e_2b8b_47f7_9a4f_9386b218aa9f.slice/crio-bac73a2d68a39c34fe0592989cd053efaaa6e0b9f1bfeeac440d1777b1d6e79b WatchSource:0}: Error finding container bac73a2d68a39c34fe0592989cd053efaaa6e0b9f1bfeeac440d1777b1d6e79b: Status 404 returned error can't find the container with id bac73a2d68a39c34fe0592989cd053efaaa6e0b9f1bfeeac440d1777b1d6e79b
Jul 16 21:22:50.332284 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:50.332247    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld" event={"ID":"68d0b32e-2b8b-47f7-9a4f-9386b218aa9f","Type":"ContainerStarted","Data":"7790b7460610c37c953826dedf07a6442f3726d9497aada84764138c7e6f51be"}
Jul 16 21:22:50.332687 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:50.332316    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld" event={"ID":"68d0b32e-2b8b-47f7-9a4f-9386b218aa9f","Type":"ContainerStarted","Data":"bac73a2d68a39c34fe0592989cd053efaaa6e0b9f1bfeeac440d1777b1d6e79b"}
Jul 16 21:22:50.333789 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:50.333768    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg" event={"ID":"7ab39e48-0068-48e4-8441-d5b4a000e9ff","Type":"ContainerStarted","Data":"cef5ab6df351fff02b22b198797be18c2cdd0d7d389f67faa3554ff94d4e226a"}
Jul 16 21:22:50.333895 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:50.333806    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg" event={"ID":"7ab39e48-0068-48e4-8441-d5b4a000e9ff","Type":"ContainerStarted","Data":"f787ecf4ad4d7ccd44cd17e5f42b0c45c8234ace4777b61851e4dd843807cf80"}
Jul 16 21:22:50.333895 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:50.333826    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"
Jul 16 21:22:50.351906 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:50.351864    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld" podStartSLOduration=1.351850665 podStartE2EDuration="1.351850665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:22:50.350093971 +0000 UTC m=+3008.371681737" watchObservedRunningTime="2026-07-16 21:22:50.351850665 +0000 UTC m=+3008.373438430"
Jul 16 21:22:50.371038 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:50.370988    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg" podStartSLOduration=2.02840672 podStartE2EDuration="2.370971622s" totalImagesPullingTime="342.564902ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:22:50.368586651 +0000 UTC m=+3008.390174418" watchObservedRunningTime="2026-07-16 21:22:50.370971622 +0000 UTC m=+3008.392559388"
Jul 16 21:22:51.339481 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:51.339435    2530 generic.go:356] "Generic (PLEG): container finished" podID="68d0b32e-2b8b-47f7-9a4f-9386b218aa9f" containerID="7790b7460610c37c953826dedf07a6442f3726d9497aada84764138c7e6f51be" exitCode=6
Jul 16 21:22:51.339938 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:51.339487    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld" event={"ID":"68d0b32e-2b8b-47f7-9a4f-9386b218aa9f","Type":"ContainerDied","Data":"7790b7460610c37c953826dedf07a6442f3726d9497aada84764138c7e6f51be"}
Jul 16 21:22:51.339938 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:51.339708    2530 scope.go:117] "RemoveContainer" containerID="7790b7460610c37c953826dedf07a6442f3726d9497aada84764138c7e6f51be"
Jul 16 21:22:52.346006 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:52.345966    2530 generic.go:356] "Generic (PLEG): container finished" podID="68d0b32e-2b8b-47f7-9a4f-9386b218aa9f" containerID="a58ca6e2ce7208d84d498a4f890c6f94e74393bb7b0556ede2da736d522a79e7" exitCode=6
Jul 16 21:22:52.346440 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:52.346048    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld" event={"ID":"68d0b32e-2b8b-47f7-9a4f-9386b218aa9f","Type":"ContainerDied","Data":"a58ca6e2ce7208d84d498a4f890c6f94e74393bb7b0556ede2da736d522a79e7"}
Jul 16 21:22:52.346440 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:52.346132    2530 scope.go:117] "RemoveContainer" containerID="7790b7460610c37c953826dedf07a6442f3726d9497aada84764138c7e6f51be"
Jul 16 21:22:52.346440 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:52.346306    2530 scope.go:117] "RemoveContainer" containerID="a58ca6e2ce7208d84d498a4f890c6f94e74393bb7b0556ede2da736d522a79e7"
Jul 16 21:22:52.346585 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:22:52.346505    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld_odh-ai-gateway-infra(68d0b32e-2b8b-47f7-9a4f-9386b218aa9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld" podUID="68d0b32e-2b8b-47f7-9a4f-9386b218aa9f"
Jul 16 21:22:53.352605 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:53.352569    2530 scope.go:117] "RemoveContainer" containerID="a58ca6e2ce7208d84d498a4f890c6f94e74393bb7b0556ede2da736d522a79e7"
Jul 16 21:22:53.352965 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:22:53.352783    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld_odh-ai-gateway-infra(68d0b32e-2b8b-47f7-9a4f-9386b218aa9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld" podUID="68d0b32e-2b8b-47f7-9a4f-9386b218aa9f"
Jul 16 21:22:56.345014 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:56.344980    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"
Jul 16 21:22:56.502394 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:56.502364    2530 scope.go:117] "RemoveContainer" containerID="c01a371c4ffa794cee47ca3583224feed51a4ae35a7c1dd7a2261f73326fd772"
Jul 16 21:22:57.376676 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:57.376646    2530 generic.go:356] "Generic (PLEG): container finished" podID="80cd16aa-13ab-48ad-ad60-6b8b7d270a9f" containerID="ca61c100ad182d48304d4808f69e8982f393e12891008868d8b342259c35c184" exitCode=6
Jul 16 21:22:57.377030 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:57.376711    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5" event={"ID":"80cd16aa-13ab-48ad-ad60-6b8b7d270a9f","Type":"ContainerDied","Data":"ca61c100ad182d48304d4808f69e8982f393e12891008868d8b342259c35c184"}
Jul 16 21:22:57.377030 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:57.376769    2530 scope.go:117] "RemoveContainer" containerID="c01a371c4ffa794cee47ca3583224feed51a4ae35a7c1dd7a2261f73326fd772"
Jul 16 21:22:57.377030 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:57.377014    2530 scope.go:117] "RemoveContainer" containerID="ca61c100ad182d48304d4808f69e8982f393e12891008868d8b342259c35c184"
Jul 16 21:22:57.377240 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:22:57.377221    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5_odh-ai-gateway-infra(80cd16aa-13ab-48ad-ad60-6b8b7d270a9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5" podUID="80cd16aa-13ab-48ad-ad60-6b8b7d270a9f"
Jul 16 21:22:58.411737 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:58.411703    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5"]
Jul 16 21:22:58.547807 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:58.547783    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5"
Jul 16 21:22:58.607840 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:58.607810    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80cd16aa-13ab-48ad-ad60-6b8b7d270a9f-maas-api-service-ca\") pod \"80cd16aa-13ab-48ad-ad60-6b8b7d270a9f\" (UID: \"80cd16aa-13ab-48ad-ad60-6b8b7d270a9f\") "
Jul 16 21:22:58.608146 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:58.608124    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80cd16aa-13ab-48ad-ad60-6b8b7d270a9f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80cd16aa-13ab-48ad-ad60-6b8b7d270a9f" (UID: "80cd16aa-13ab-48ad-ad60-6b8b7d270a9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:22:58.708722 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:58.708702    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80cd16aa-13ab-48ad-ad60-6b8b7d270a9f-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:22:59.389680 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:59.389645    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5" event={"ID":"80cd16aa-13ab-48ad-ad60-6b8b7d270a9f","Type":"ContainerDied","Data":"b9e864b152cf7d5d9d51ce28d0c03341ea9df680397e10b3aecd34dbea9af020"}
Jul 16 21:22:59.389826 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:59.389706    2530 scope.go:117] "RemoveContainer" containerID="ca61c100ad182d48304d4808f69e8982f393e12891008868d8b342259c35c184"
Jul 16 21:22:59.389826 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:59.389659    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5"
Jul 16 21:22:59.411448 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:59.411428    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5"]
Jul 16 21:22:59.415520 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:22:59.415500    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-vl9d5"]
Jul 16 21:23:00.506495 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:00.506461    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80cd16aa-13ab-48ad-ad60-6b8b7d270a9f" path="/var/lib/kubelet/pods/80cd16aa-13ab-48ad-ad60-6b8b7d270a9f/volumes"
Jul 16 21:23:02.685706 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:02.685677    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb"]
Jul 16 21:23:02.686063 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:02.686047    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80cd16aa-13ab-48ad-ad60-6b8b7d270a9f" containerName="revoke-keys"
Jul 16 21:23:02.686063 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:02.686058    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="80cd16aa-13ab-48ad-ad60-6b8b7d270a9f" containerName="revoke-keys"
Jul 16 21:23:02.686203 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:02.686070    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80cd16aa-13ab-48ad-ad60-6b8b7d270a9f" containerName="revoke-keys"
Jul 16 21:23:02.686203 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:02.686076    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="80cd16aa-13ab-48ad-ad60-6b8b7d270a9f" containerName="revoke-keys"
Jul 16 21:23:02.686203 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:02.686146    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="80cd16aa-13ab-48ad-ad60-6b8b7d270a9f" containerName="revoke-keys"
Jul 16 21:23:02.686203 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:02.686153    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="80cd16aa-13ab-48ad-ad60-6b8b7d270a9f" containerName="revoke-keys"
Jul 16 21:23:02.690529 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:02.690509    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb"
Jul 16 21:23:02.697615 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:02.697554    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb"]
Jul 16 21:23:02.738204 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:02.738179    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cac01ef-303c-4f54-8298-46a27ba3c222-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb\" (UID: \"2cac01ef-303c-4f54-8298-46a27ba3c222\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb"
Jul 16 21:23:02.839013 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:02.838971    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cac01ef-303c-4f54-8298-46a27ba3c222-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb\" (UID: \"2cac01ef-303c-4f54-8298-46a27ba3c222\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb"
Jul 16 21:23:02.839637 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:02.839614    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cac01ef-303c-4f54-8298-46a27ba3c222-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb\" (UID: \"2cac01ef-303c-4f54-8298-46a27ba3c222\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb"
Jul 16 21:23:03.002399 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:03.002371    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb"
Jul 16 21:23:03.338262 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:03.338236    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb"]
Jul 16 21:23:03.340046 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:23:03.340013    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cac01ef_303c_4f54_8298_46a27ba3c222.slice/crio-c3072551e6632211637e36f09c390fca19f33a9ce94839979e6ff3bd60eef0b5 WatchSource:0}: Error finding container c3072551e6632211637e36f09c390fca19f33a9ce94839979e6ff3bd60eef0b5: Status 404 returned error can't find the container with id c3072551e6632211637e36f09c390fca19f33a9ce94839979e6ff3bd60eef0b5
Jul 16 21:23:03.408831 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:03.408804    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb" event={"ID":"2cac01ef-303c-4f54-8298-46a27ba3c222","Type":"ContainerStarted","Data":"c3072551e6632211637e36f09c390fca19f33a9ce94839979e6ff3bd60eef0b5"}
Jul 16 21:23:04.414670 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:04.414636    2530 generic.go:356] "Generic (PLEG): container finished" podID="2cac01ef-303c-4f54-8298-46a27ba3c222" containerID="e28f39b15e142c81ec88507c5fa5d883e54bce0f53bbee2f53bec9d52c0de055" exitCode=6
Jul 16 21:23:04.415154 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:04.414727    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb" event={"ID":"2cac01ef-303c-4f54-8298-46a27ba3c222","Type":"ContainerDied","Data":"e28f39b15e142c81ec88507c5fa5d883e54bce0f53bbee2f53bec9d52c0de055"}
Jul 16 21:23:04.415154 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:04.414911    2530 scope.go:117] "RemoveContainer" containerID="e28f39b15e142c81ec88507c5fa5d883e54bce0f53bbee2f53bec9d52c0de055"
Jul 16 21:23:05.421341 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:05.421269    2530 generic.go:356] "Generic (PLEG): container finished" podID="2cac01ef-303c-4f54-8298-46a27ba3c222" containerID="c91eb8df532ce185a135bbf81f683616d0b67b40756056fd465423c0e80faa9e" exitCode=6
Jul 16 21:23:05.421341 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:05.421325    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb" event={"ID":"2cac01ef-303c-4f54-8298-46a27ba3c222","Type":"ContainerDied","Data":"c91eb8df532ce185a135bbf81f683616d0b67b40756056fd465423c0e80faa9e"}
Jul 16 21:23:05.421799 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:05.421372    2530 scope.go:117] "RemoveContainer" containerID="e28f39b15e142c81ec88507c5fa5d883e54bce0f53bbee2f53bec9d52c0de055"
Jul 16 21:23:05.421799 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:05.421578    2530 scope.go:117] "RemoveContainer" containerID="c91eb8df532ce185a135bbf81f683616d0b67b40756056fd465423c0e80faa9e"
Jul 16 21:23:05.421799 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:23:05.421791    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb_odh-ai-gateway-infra(2cac01ef-303c-4f54-8298-46a27ba3c222)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb" podUID="2cac01ef-303c-4f54-8298-46a27ba3c222"
Jul 16 21:23:06.428728 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:06.428701    2530 scope.go:117] "RemoveContainer" containerID="c91eb8df532ce185a135bbf81f683616d0b67b40756056fd465423c0e80faa9e"
Jul 16 21:23:06.429208 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:23:06.428946    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb_odh-ai-gateway-infra(2cac01ef-303c-4f54-8298-46a27ba3c222)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb" podUID="2cac01ef-303c-4f54-8298-46a27ba3c222"
Jul 16 21:23:06.501864 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:06.501841    2530 scope.go:117] "RemoveContainer" containerID="a58ca6e2ce7208d84d498a4f890c6f94e74393bb7b0556ede2da736d522a79e7"
Jul 16 21:23:07.435148 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:07.435048    2530 generic.go:356] "Generic (PLEG): container finished" podID="68d0b32e-2b8b-47f7-9a4f-9386b218aa9f" containerID="1b2550baf41fff24cf0f2fd391b7b8d5c0347b80bc57995f54c3d2bd16909d46" exitCode=6
Jul 16 21:23:07.435148 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:07.435119    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld" event={"ID":"68d0b32e-2b8b-47f7-9a4f-9386b218aa9f","Type":"ContainerDied","Data":"1b2550baf41fff24cf0f2fd391b7b8d5c0347b80bc57995f54c3d2bd16909d46"}
Jul 16 21:23:07.435628 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:07.435169    2530 scope.go:117] "RemoveContainer" containerID="a58ca6e2ce7208d84d498a4f890c6f94e74393bb7b0556ede2da736d522a79e7"
Jul 16 21:23:07.435628 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:07.435507    2530 scope.go:117] "RemoveContainer" containerID="1b2550baf41fff24cf0f2fd391b7b8d5c0347b80bc57995f54c3d2bd16909d46"
Jul 16 21:23:07.435772 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:23:07.435749    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld_odh-ai-gateway-infra(68d0b32e-2b8b-47f7-9a4f-9386b218aa9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld" podUID="68d0b32e-2b8b-47f7-9a4f-9386b218aa9f"
Jul 16 21:23:08.459537 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:08.459506    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld"]
Jul 16 21:23:08.595972 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:08.595950    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld"
Jul 16 21:23:08.684945 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:08.684923    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d0b32e-2b8b-47f7-9a4f-9386b218aa9f-maas-api-service-ca\") pod \"68d0b32e-2b8b-47f7-9a4f-9386b218aa9f\" (UID: \"68d0b32e-2b8b-47f7-9a4f-9386b218aa9f\") "
Jul 16 21:23:08.685280 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:08.685260    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d0b32e-2b8b-47f7-9a4f-9386b218aa9f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "68d0b32e-2b8b-47f7-9a4f-9386b218aa9f" (UID: "68d0b32e-2b8b-47f7-9a4f-9386b218aa9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:23:08.786310 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:08.786250    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d0b32e-2b8b-47f7-9a4f-9386b218aa9f-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:23:09.446427 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:09.446395    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld" event={"ID":"68d0b32e-2b8b-47f7-9a4f-9386b218aa9f","Type":"ContainerDied","Data":"bac73a2d68a39c34fe0592989cd053efaaa6e0b9f1bfeeac440d1777b1d6e79b"}
Jul 16 21:23:09.446427 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:09.446421    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld"
Jul 16 21:23:09.446636 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:09.446454    2530 scope.go:117] "RemoveContainer" containerID="1b2550baf41fff24cf0f2fd391b7b8d5c0347b80bc57995f54c3d2bd16909d46"
Jul 16 21:23:09.472487 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:09.472455    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld"]
Jul 16 21:23:09.478220 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:09.478201    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7bsld"]
Jul 16 21:23:10.506870 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:10.506837    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68d0b32e-2b8b-47f7-9a4f-9386b218aa9f" path="/var/lib/kubelet/pods/68d0b32e-2b8b-47f7-9a4f-9386b218aa9f/volumes"
Jul 16 21:23:21.502437 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:21.502408    2530 scope.go:117] "RemoveContainer" containerID="c91eb8df532ce185a135bbf81f683616d0b67b40756056fd465423c0e80faa9e"
Jul 16 21:23:22.511495 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:22.511463    2530 generic.go:356] "Generic (PLEG): container finished" podID="2cac01ef-303c-4f54-8298-46a27ba3c222" containerID="39870b99b19edce6503c6186c76029de52166649f372331f73d098235d70c43a" exitCode=6
Jul 16 21:23:22.511890 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:22.511535    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb" event={"ID":"2cac01ef-303c-4f54-8298-46a27ba3c222","Type":"ContainerDied","Data":"39870b99b19edce6503c6186c76029de52166649f372331f73d098235d70c43a"}
Jul 16 21:23:22.511890 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:22.511592    2530 scope.go:117] "RemoveContainer" containerID="c91eb8df532ce185a135bbf81f683616d0b67b40756056fd465423c0e80faa9e"
Jul 16 21:23:22.512009 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:22.511944    2530 scope.go:117] "RemoveContainer" containerID="39870b99b19edce6503c6186c76029de52166649f372331f73d098235d70c43a"
Jul 16 21:23:22.512174 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:23:22.512146    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb_odh-ai-gateway-infra(2cac01ef-303c-4f54-8298-46a27ba3c222)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb" podUID="2cac01ef-303c-4f54-8298-46a27ba3c222"
Jul 16 21:23:23.542484 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:23.537828    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb"]
Jul 16 21:23:23.668521 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:23.668496    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb"
Jul 16 21:23:23.797491 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:23.797424    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cac01ef-303c-4f54-8298-46a27ba3c222-maas-api-service-ca\") pod \"2cac01ef-303c-4f54-8298-46a27ba3c222\" (UID: \"2cac01ef-303c-4f54-8298-46a27ba3c222\") "
Jul 16 21:23:23.797730 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:23.797709    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cac01ef-303c-4f54-8298-46a27ba3c222-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2cac01ef-303c-4f54-8298-46a27ba3c222" (UID: "2cac01ef-303c-4f54-8298-46a27ba3c222"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:23:23.898275 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:23.898246    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cac01ef-303c-4f54-8298-46a27ba3c222-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:23:24.354434 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.354401    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm"]
Jul 16 21:23:24.354749 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.354737    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cac01ef-303c-4f54-8298-46a27ba3c222" containerName="revoke-keys"
Jul 16 21:23:24.354794 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.354752    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cac01ef-303c-4f54-8298-46a27ba3c222" containerName="revoke-keys"
Jul 16 21:23:24.354794 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.354762    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68d0b32e-2b8b-47f7-9a4f-9386b218aa9f" containerName="revoke-keys"
Jul 16 21:23:24.354794 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.354767    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="68d0b32e-2b8b-47f7-9a4f-9386b218aa9f" containerName="revoke-keys"
Jul 16 21:23:24.354794 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.354773    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cac01ef-303c-4f54-8298-46a27ba3c222" containerName="revoke-keys"
Jul 16 21:23:24.354794 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.354779    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cac01ef-303c-4f54-8298-46a27ba3c222" containerName="revoke-keys"
Jul 16 21:23:24.354794 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.354788    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cac01ef-303c-4f54-8298-46a27ba3c222" containerName="revoke-keys"
Jul 16 21:23:24.354794 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.354794    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cac01ef-303c-4f54-8298-46a27ba3c222" containerName="revoke-keys"
Jul 16 21:23:24.355009 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.354803    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68d0b32e-2b8b-47f7-9a4f-9386b218aa9f" containerName="revoke-keys"
Jul 16 21:23:24.355009 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.354809    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="68d0b32e-2b8b-47f7-9a4f-9386b218aa9f" containerName="revoke-keys"
Jul 16 21:23:24.355009 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.354823    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80cd16aa-13ab-48ad-ad60-6b8b7d270a9f" containerName="revoke-keys"
Jul 16 21:23:24.355009 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.354827    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="80cd16aa-13ab-48ad-ad60-6b8b7d270a9f" containerName="revoke-keys"
Jul 16 21:23:24.355009 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.354878    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="68d0b32e-2b8b-47f7-9a4f-9386b218aa9f" containerName="revoke-keys"
Jul 16 21:23:24.355009 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.354885    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cac01ef-303c-4f54-8298-46a27ba3c222" containerName="revoke-keys"
Jul 16 21:23:24.355009 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.354891    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cac01ef-303c-4f54-8298-46a27ba3c222" containerName="revoke-keys"
Jul 16 21:23:24.355009 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.354897    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="80cd16aa-13ab-48ad-ad60-6b8b7d270a9f" containerName="revoke-keys"
Jul 16 21:23:24.355009 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.354903    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="68d0b32e-2b8b-47f7-9a4f-9386b218aa9f" containerName="revoke-keys"
Jul 16 21:23:24.358030 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.358013    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm"
Jul 16 21:23:24.364069 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.364042    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm"]
Jul 16 21:23:24.400896 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.400862    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48be669d-8a80-4475-912d-1157fae67690-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm\" (UID: \"48be669d-8a80-4475-912d-1157fae67690\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm"
Jul 16 21:23:24.502195 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.502168    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48be669d-8a80-4475-912d-1157fae67690-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm\" (UID: \"48be669d-8a80-4475-912d-1157fae67690\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm"
Jul 16 21:23:24.502732 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.502711    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48be669d-8a80-4475-912d-1157fae67690-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm\" (UID: \"48be669d-8a80-4475-912d-1157fae67690\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm"
Jul 16 21:23:24.523125 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.523085    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb" event={"ID":"2cac01ef-303c-4f54-8298-46a27ba3c222","Type":"ContainerDied","Data":"c3072551e6632211637e36f09c390fca19f33a9ce94839979e6ff3bd60eef0b5"}
Jul 16 21:23:24.523224 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.523156    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb"
Jul 16 21:23:24.523284 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.523159    2530 scope.go:117] "RemoveContainer" containerID="39870b99b19edce6503c6186c76029de52166649f372331f73d098235d70c43a"
Jul 16 21:23:24.539400 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.539371    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb"]
Jul 16 21:23:24.546934 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.546915    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-fftlb"]
Jul 16 21:23:24.669575 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.669522    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm"
Jul 16 21:23:24.797535 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:24.797512    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm"]
Jul 16 21:23:24.799438 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:23:24.799410    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48be669d_8a80_4475_912d_1157fae67690.slice/crio-e058f2f885b405bb58a18d7aa321d394608c3cfb2133bc3fa126bf10609c5432 WatchSource:0}: Error finding container e058f2f885b405bb58a18d7aa321d394608c3cfb2133bc3fa126bf10609c5432: Status 404 returned error can't find the container with id e058f2f885b405bb58a18d7aa321d394608c3cfb2133bc3fa126bf10609c5432
Jul 16 21:23:25.530258 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:25.530230    2530 generic.go:356] "Generic (PLEG): container finished" podID="48be669d-8a80-4475-912d-1157fae67690" containerID="d406c704b6594641b29c888e3c3ba11102a3e4bce81c7e7901f352590026d2ad" exitCode=6
Jul 16 21:23:25.530423 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:25.530305    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm" event={"ID":"48be669d-8a80-4475-912d-1157fae67690","Type":"ContainerDied","Data":"d406c704b6594641b29c888e3c3ba11102a3e4bce81c7e7901f352590026d2ad"}
Jul 16 21:23:25.530423 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:25.530341    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm" event={"ID":"48be669d-8a80-4475-912d-1157fae67690","Type":"ContainerStarted","Data":"e058f2f885b405bb58a18d7aa321d394608c3cfb2133bc3fa126bf10609c5432"}
Jul 16 21:23:25.530501 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:25.530426    2530 scope.go:117] "RemoveContainer" containerID="d406c704b6594641b29c888e3c3ba11102a3e4bce81c7e7901f352590026d2ad"
Jul 16 21:23:26.507044 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:26.507010    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cac01ef-303c-4f54-8298-46a27ba3c222" path="/var/lib/kubelet/pods/2cac01ef-303c-4f54-8298-46a27ba3c222/volumes"
Jul 16 21:23:26.535776 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:26.535749    2530 generic.go:356] "Generic (PLEG): container finished" podID="48be669d-8a80-4475-912d-1157fae67690" containerID="e94c3882e244bfad7cc399f0129969102e6fd44597e990dab130bf9b5b13a147" exitCode=6
Jul 16 21:23:26.535915 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:26.535839    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm" event={"ID":"48be669d-8a80-4475-912d-1157fae67690","Type":"ContainerDied","Data":"e94c3882e244bfad7cc399f0129969102e6fd44597e990dab130bf9b5b13a147"}
Jul 16 21:23:26.535915 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:26.535899    2530 scope.go:117] "RemoveContainer" containerID="d406c704b6594641b29c888e3c3ba11102a3e4bce81c7e7901f352590026d2ad"
Jul 16 21:23:26.536025 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:26.535967    2530 scope.go:117] "RemoveContainer" containerID="e94c3882e244bfad7cc399f0129969102e6fd44597e990dab130bf9b5b13a147"
Jul 16 21:23:26.536203 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:23:26.536176    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm_odh-ai-gateway-infra(48be669d-8a80-4475-912d-1157fae67690)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm" podUID="48be669d-8a80-4475-912d-1157fae67690"
Jul 16 21:23:27.542423 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:27.542396    2530 scope.go:117] "RemoveContainer" containerID="e94c3882e244bfad7cc399f0129969102e6fd44597e990dab130bf9b5b13a147"
Jul 16 21:23:27.542885 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:23:27.542651    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm_odh-ai-gateway-infra(48be669d-8a80-4475-912d-1157fae67690)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm" podUID="48be669d-8a80-4475-912d-1157fae67690"
Jul 16 21:23:32.193133 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:32.193087    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59"]
Jul 16 21:23:32.193527 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:32.193461    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68d0b32e-2b8b-47f7-9a4f-9386b218aa9f" containerName="revoke-keys"
Jul 16 21:23:32.193527 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:32.193475    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="68d0b32e-2b8b-47f7-9a4f-9386b218aa9f" containerName="revoke-keys"
Jul 16 21:23:32.193601 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:32.193536    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cac01ef-303c-4f54-8298-46a27ba3c222" containerName="revoke-keys"
Jul 16 21:23:32.193601 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:32.193545    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="68d0b32e-2b8b-47f7-9a4f-9386b218aa9f" containerName="revoke-keys"
Jul 16 21:23:32.197807 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:32.197792    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59"
Jul 16 21:23:32.203374 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:32.203334    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59"]
Jul 16 21:23:32.262884 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:32.262859    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b9d37d-c707-4d02-8e03-e9a2a4db77e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-mmj59\" (UID: \"83b9d37d-c707-4d02-8e03-e9a2a4db77e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59"
Jul 16 21:23:32.363953 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:32.363932    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b9d37d-c707-4d02-8e03-e9a2a4db77e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-mmj59\" (UID: \"83b9d37d-c707-4d02-8e03-e9a2a4db77e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59"
Jul 16 21:23:32.364558 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:32.364540    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b9d37d-c707-4d02-8e03-e9a2a4db77e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-mmj59\" (UID: \"83b9d37d-c707-4d02-8e03-e9a2a4db77e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59"
Jul 16 21:23:32.510956 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:32.510935    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59"
Jul 16 21:23:32.640085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:32.640058    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59"]
Jul 16 21:23:32.642157 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:23:32.642128    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b9d37d_c707_4d02_8e03_e9a2a4db77e1.slice/crio-222e02eef17a7e6e57987a0ad5fbb316a55451a77f11301dc1cb15b794433fb0 WatchSource:0}: Error finding container 222e02eef17a7e6e57987a0ad5fbb316a55451a77f11301dc1cb15b794433fb0: Status 404 returned error can't find the container with id 222e02eef17a7e6e57987a0ad5fbb316a55451a77f11301dc1cb15b794433fb0
Jul 16 21:23:33.573767 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:33.573677    2530 generic.go:356] "Generic (PLEG): container finished" podID="83b9d37d-c707-4d02-8e03-e9a2a4db77e1" containerID="306b144a52d534e390772e1253a042aa529ef3f41a61af78d9fb0dcde80d5df4" exitCode=6
Jul 16 21:23:33.574215 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:33.573761    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59" event={"ID":"83b9d37d-c707-4d02-8e03-e9a2a4db77e1","Type":"ContainerDied","Data":"306b144a52d534e390772e1253a042aa529ef3f41a61af78d9fb0dcde80d5df4"}
Jul 16 21:23:33.574215 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:33.573808    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59" event={"ID":"83b9d37d-c707-4d02-8e03-e9a2a4db77e1","Type":"ContainerStarted","Data":"222e02eef17a7e6e57987a0ad5fbb316a55451a77f11301dc1cb15b794433fb0"}
Jul 16 21:23:33.574215 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:33.573956    2530 scope.go:117] "RemoveContainer" containerID="306b144a52d534e390772e1253a042aa529ef3f41a61af78d9fb0dcde80d5df4"
Jul 16 21:23:34.579716 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:34.579622    2530 generic.go:356] "Generic (PLEG): container finished" podID="83b9d37d-c707-4d02-8e03-e9a2a4db77e1" containerID="97c05280bc449f13dfb0714c83463d5bf570db25a0d44f1c2cb814568ba7df94" exitCode=6
Jul 16 21:23:34.579716 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:34.579692    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59" event={"ID":"83b9d37d-c707-4d02-8e03-e9a2a4db77e1","Type":"ContainerDied","Data":"97c05280bc449f13dfb0714c83463d5bf570db25a0d44f1c2cb814568ba7df94"}
Jul 16 21:23:34.580265 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:34.579745    2530 scope.go:117] "RemoveContainer" containerID="306b144a52d534e390772e1253a042aa529ef3f41a61af78d9fb0dcde80d5df4"
Jul 16 21:23:34.580265 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:34.579916    2530 scope.go:117] "RemoveContainer" containerID="97c05280bc449f13dfb0714c83463d5bf570db25a0d44f1c2cb814568ba7df94"
Jul 16 21:23:34.580265 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:23:34.580155    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-mmj59_odh-ai-gateway-infra(83b9d37d-c707-4d02-8e03-e9a2a4db77e1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59" podUID="83b9d37d-c707-4d02-8e03-e9a2a4db77e1"
Jul 16 21:23:35.586615 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:35.586588    2530 scope.go:117] "RemoveContainer" containerID="97c05280bc449f13dfb0714c83463d5bf570db25a0d44f1c2cb814568ba7df94"
Jul 16 21:23:35.586966 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:23:35.586760    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-mmj59_odh-ai-gateway-infra(83b9d37d-c707-4d02-8e03-e9a2a4db77e1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59" podUID="83b9d37d-c707-4d02-8e03-e9a2a4db77e1"
Jul 16 21:23:39.607029 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:39.607000    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl"]
Jul 16 21:23:39.611497 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:39.611482    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl"
Jul 16 21:23:39.617157 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:39.617065    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl"]
Jul 16 21:23:39.719863 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:39.719834    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c597582e-cf9f-4ef9-bb12-9a4149b869ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl\" (UID: \"c597582e-cf9f-4ef9-bb12-9a4149b869ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl"
Jul 16 21:23:39.820206 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:39.820171    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c597582e-cf9f-4ef9-bb12-9a4149b869ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl\" (UID: \"c597582e-cf9f-4ef9-bb12-9a4149b869ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl"
Jul 16 21:23:39.820742 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:39.820723    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c597582e-cf9f-4ef9-bb12-9a4149b869ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl\" (UID: \"c597582e-cf9f-4ef9-bb12-9a4149b869ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl"
Jul 16 21:23:39.923300 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:39.923230    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl"
Jul 16 21:23:40.051747 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:40.051720    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl"]
Jul 16 21:23:40.053622 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:23:40.053594    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc597582e_cf9f_4ef9_bb12_9a4149b869ea.slice/crio-c2a2f73648c05a63cbeec437c5f5bc675abc47fd603f324bcadab3826b05485a WatchSource:0}: Error finding container c2a2f73648c05a63cbeec437c5f5bc675abc47fd603f324bcadab3826b05485a: Status 404 returned error can't find the container with id c2a2f73648c05a63cbeec437c5f5bc675abc47fd603f324bcadab3826b05485a
Jul 16 21:23:40.611187 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:40.611094    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl" event={"ID":"c597582e-cf9f-4ef9-bb12-9a4149b869ea","Type":"ContainerStarted","Data":"5e0c64ec97606fc117f9612860922bc3c16b1f3e9c3acbf304478bcd3083a933"}
Jul 16 21:23:40.611187 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:40.611156    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl" event={"ID":"c597582e-cf9f-4ef9-bb12-9a4149b869ea","Type":"ContainerStarted","Data":"c2a2f73648c05a63cbeec437c5f5bc675abc47fd603f324bcadab3826b05485a"}
Jul 16 21:23:40.627556 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:40.627462    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl" podStartSLOduration=1.627443142 podStartE2EDuration="1.627443142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:23:40.626521857 +0000 UTC m=+3058.648109625" watchObservedRunningTime="2026-07-16 21:23:40.627443142 +0000 UTC m=+3058.649030910"
Jul 16 21:23:41.501976 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:41.501942    2530 scope.go:117] "RemoveContainer" containerID="e94c3882e244bfad7cc399f0129969102e6fd44597e990dab130bf9b5b13a147"
Jul 16 21:23:41.617894 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:41.617862    2530 generic.go:356] "Generic (PLEG): container finished" podID="c597582e-cf9f-4ef9-bb12-9a4149b869ea" containerID="5e0c64ec97606fc117f9612860922bc3c16b1f3e9c3acbf304478bcd3083a933" exitCode=6
Jul 16 21:23:41.618290 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:41.617947    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl" event={"ID":"c597582e-cf9f-4ef9-bb12-9a4149b869ea","Type":"ContainerDied","Data":"5e0c64ec97606fc117f9612860922bc3c16b1f3e9c3acbf304478bcd3083a933"}
Jul 16 21:23:41.618290 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:41.618151    2530 scope.go:117] "RemoveContainer" containerID="5e0c64ec97606fc117f9612860922bc3c16b1f3e9c3acbf304478bcd3083a933"
Jul 16 21:23:42.624975 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:42.624943    2530 generic.go:356] "Generic (PLEG): container finished" podID="c597582e-cf9f-4ef9-bb12-9a4149b869ea" containerID="74ef7da1c105d43751b6546004f59597c4dcb5c62b8f8dda082de8b64327c8b8" exitCode=6
Jul 16 21:23:42.625429 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:42.625029    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl" event={"ID":"c597582e-cf9f-4ef9-bb12-9a4149b869ea","Type":"ContainerDied","Data":"74ef7da1c105d43751b6546004f59597c4dcb5c62b8f8dda082de8b64327c8b8"}
Jul 16 21:23:42.625429 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:42.625084    2530 scope.go:117] "RemoveContainer" containerID="5e0c64ec97606fc117f9612860922bc3c16b1f3e9c3acbf304478bcd3083a933"
Jul 16 21:23:42.625429 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:42.625240    2530 scope.go:117] "RemoveContainer" containerID="74ef7da1c105d43751b6546004f59597c4dcb5c62b8f8dda082de8b64327c8b8"
Jul 16 21:23:42.625585 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:23:42.625452    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl_odh-ai-gateway-infra(c597582e-cf9f-4ef9-bb12-9a4149b869ea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl" podUID="c597582e-cf9f-4ef9-bb12-9a4149b869ea"
Jul 16 21:23:42.627074 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:42.627054    2530 generic.go:356] "Generic (PLEG): container finished" podID="48be669d-8a80-4475-912d-1157fae67690" containerID="2e9eebf4cc782ab2e268b2ff851be5cdb72b215b60b2d29942db735c73358edf" exitCode=6
Jul 16 21:23:42.627189 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:42.627131    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm" event={"ID":"48be669d-8a80-4475-912d-1157fae67690","Type":"ContainerDied","Data":"2e9eebf4cc782ab2e268b2ff851be5cdb72b215b60b2d29942db735c73358edf"}
Jul 16 21:23:42.627380 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:42.627366    2530 scope.go:117] "RemoveContainer" containerID="2e9eebf4cc782ab2e268b2ff851be5cdb72b215b60b2d29942db735c73358edf"
Jul 16 21:23:42.627527 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:23:42.627510    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm_odh-ai-gateway-infra(48be669d-8a80-4475-912d-1157fae67690)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm" podUID="48be669d-8a80-4475-912d-1157fae67690"
Jul 16 21:23:42.636470 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:42.636445    2530 scope.go:117] "RemoveContainer" containerID="e94c3882e244bfad7cc399f0129969102e6fd44597e990dab130bf9b5b13a147"
Jul 16 21:23:43.633739 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:43.633658    2530 scope.go:117] "RemoveContainer" containerID="74ef7da1c105d43751b6546004f59597c4dcb5c62b8f8dda082de8b64327c8b8"
Jul 16 21:23:43.634162 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:23:43.633862    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl_odh-ai-gateway-infra(c597582e-cf9f-4ef9-bb12-9a4149b869ea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl" podUID="c597582e-cf9f-4ef9-bb12-9a4149b869ea"
Jul 16 21:23:43.669026 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:43.668995    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm"]
Jul 16 21:23:43.816090 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:43.816068    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm"
Jul 16 21:23:43.958392 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:43.958365    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48be669d-8a80-4475-912d-1157fae67690-maas-api-service-ca\") pod \"48be669d-8a80-4475-912d-1157fae67690\" (UID: \"48be669d-8a80-4475-912d-1157fae67690\") "
Jul 16 21:23:43.958698 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:43.958676    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48be669d-8a80-4475-912d-1157fae67690-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48be669d-8a80-4475-912d-1157fae67690" (UID: "48be669d-8a80-4475-912d-1157fae67690"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:23:44.058867 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:44.058843    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48be669d-8a80-4475-912d-1157fae67690-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:23:44.640495 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:44.640465    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm"
Jul 16 21:23:44.640885 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:44.640499    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm" event={"ID":"48be669d-8a80-4475-912d-1157fae67690","Type":"ContainerDied","Data":"e058f2f885b405bb58a18d7aa321d394608c3cfb2133bc3fa126bf10609c5432"}
Jul 16 21:23:44.640885 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:44.640562    2530 scope.go:117] "RemoveContainer" containerID="2e9eebf4cc782ab2e268b2ff851be5cdb72b215b60b2d29942db735c73358edf"
Jul 16 21:23:44.656702 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:44.656679    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm"]
Jul 16 21:23:44.660382 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:44.660359    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-r8xsm"]
Jul 16 21:23:46.501959 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:46.501929    2530 scope.go:117] "RemoveContainer" containerID="97c05280bc449f13dfb0714c83463d5bf570db25a0d44f1c2cb814568ba7df94"
Jul 16 21:23:46.508484 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:46.508454    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48be669d-8a80-4475-912d-1157fae67690" path="/var/lib/kubelet/pods/48be669d-8a80-4475-912d-1157fae67690/volumes"
Jul 16 21:23:47.657672 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:47.657592    2530 generic.go:356] "Generic (PLEG): container finished" podID="83b9d37d-c707-4d02-8e03-e9a2a4db77e1" containerID="21131b32d2118b8d1e4b18bbbdfec3c73818857d249afd356531aa5b00a18924" exitCode=6
Jul 16 21:23:47.658120 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:47.657665    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59" event={"ID":"83b9d37d-c707-4d02-8e03-e9a2a4db77e1","Type":"ContainerDied","Data":"21131b32d2118b8d1e4b18bbbdfec3c73818857d249afd356531aa5b00a18924"}
Jul 16 21:23:47.658120 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:47.657724    2530 scope.go:117] "RemoveContainer" containerID="97c05280bc449f13dfb0714c83463d5bf570db25a0d44f1c2cb814568ba7df94"
Jul 16 21:23:47.658120 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:47.658030    2530 scope.go:117] "RemoveContainer" containerID="21131b32d2118b8d1e4b18bbbdfec3c73818857d249afd356531aa5b00a18924"
Jul 16 21:23:47.658313 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:23:47.658289    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-mmj59_odh-ai-gateway-infra(83b9d37d-c707-4d02-8e03-e9a2a4db77e1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59" podUID="83b9d37d-c707-4d02-8e03-e9a2a4db77e1"
Jul 16 21:23:48.687713 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:48.687684    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59"]
Jul 16 21:23:48.821224 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:48.821201    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59"
Jul 16 21:23:48.899449 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:48.899418    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b9d37d-c707-4d02-8e03-e9a2a4db77e1-maas-api-service-ca\") pod \"83b9d37d-c707-4d02-8e03-e9a2a4db77e1\" (UID: \"83b9d37d-c707-4d02-8e03-e9a2a4db77e1\") "
Jul 16 21:23:48.899758 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:48.899735    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b9d37d-c707-4d02-8e03-e9a2a4db77e1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83b9d37d-c707-4d02-8e03-e9a2a4db77e1" (UID: "83b9d37d-c707-4d02-8e03-e9a2a4db77e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:23:49.000483 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:49.000462    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83b9d37d-c707-4d02-8e03-e9a2a4db77e1-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:23:49.672521 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:49.672486    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59" event={"ID":"83b9d37d-c707-4d02-8e03-e9a2a4db77e1","Type":"ContainerDied","Data":"222e02eef17a7e6e57987a0ad5fbb316a55451a77f11301dc1cb15b794433fb0"}
Jul 16 21:23:49.672521 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:49.672520    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59"
Jul 16 21:23:49.672774 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:49.672588    2530 scope.go:117] "RemoveContainer" containerID="21131b32d2118b8d1e4b18bbbdfec3c73818857d249afd356531aa5b00a18924"
Jul 16 21:23:49.693016 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:49.692991    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59"]
Jul 16 21:23:49.698895 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:49.698875    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-mmj59"]
Jul 16 21:23:50.508197 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:50.508162    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83b9d37d-c707-4d02-8e03-e9a2a4db77e1" path="/var/lib/kubelet/pods/83b9d37d-c707-4d02-8e03-e9a2a4db77e1/volumes"
Jul 16 21:23:55.502423 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:55.502387    2530 scope.go:117] "RemoveContainer" containerID="74ef7da1c105d43751b6546004f59597c4dcb5c62b8f8dda082de8b64327c8b8"
Jul 16 21:23:56.707340 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:56.707301    2530 generic.go:356] "Generic (PLEG): container finished" podID="c597582e-cf9f-4ef9-bb12-9a4149b869ea" containerID="8c393746f1411268cc76dd7e35c6cb6379d43763c8d8e57eab08bc43888c6263" exitCode=6
Jul 16 21:23:56.707803 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:56.707377    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl" event={"ID":"c597582e-cf9f-4ef9-bb12-9a4149b869ea","Type":"ContainerDied","Data":"8c393746f1411268cc76dd7e35c6cb6379d43763c8d8e57eab08bc43888c6263"}
Jul 16 21:23:56.707803 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:56.707435    2530 scope.go:117] "RemoveContainer" containerID="74ef7da1c105d43751b6546004f59597c4dcb5c62b8f8dda082de8b64327c8b8"
Jul 16 21:23:56.707803 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:56.707763    2530 scope.go:117] "RemoveContainer" containerID="8c393746f1411268cc76dd7e35c6cb6379d43763c8d8e57eab08bc43888c6263"
Jul 16 21:23:56.708002 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:23:56.707981    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl_odh-ai-gateway-infra(c597582e-cf9f-4ef9-bb12-9a4149b869ea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl" podUID="c597582e-cf9f-4ef9-bb12-9a4149b869ea"
Jul 16 21:23:57.732166 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:57.732130    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl"]
Jul 16 21:23:57.867293 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:57.867263    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl"
Jul 16 21:23:57.986231 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:57.986161    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c597582e-cf9f-4ef9-bb12-9a4149b869ea-maas-api-service-ca\") pod \"c597582e-cf9f-4ef9-bb12-9a4149b869ea\" (UID: \"c597582e-cf9f-4ef9-bb12-9a4149b869ea\") "
Jul 16 21:23:57.986498 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:57.986474    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c597582e-cf9f-4ef9-bb12-9a4149b869ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c597582e-cf9f-4ef9-bb12-9a4149b869ea" (UID: "c597582e-cf9f-4ef9-bb12-9a4149b869ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:23:58.087154 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:58.087123    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c597582e-cf9f-4ef9-bb12-9a4149b869ea-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:23:58.719971 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:58.719942    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl"
Jul 16 21:23:58.720150 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:58.719943    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl" event={"ID":"c597582e-cf9f-4ef9-bb12-9a4149b869ea","Type":"ContainerDied","Data":"c2a2f73648c05a63cbeec437c5f5bc675abc47fd603f324bcadab3826b05485a"}
Jul 16 21:23:58.720150 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:58.720079    2530 scope.go:117] "RemoveContainer" containerID="8c393746f1411268cc76dd7e35c6cb6379d43763c8d8e57eab08bc43888c6263"
Jul 16 21:23:58.735681 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:58.735657    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl"]
Jul 16 21:23:58.737351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:23:58.737330    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-dwkbl"]
Jul 16 21:24:00.507491 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:00.507456    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c597582e-cf9f-4ef9-bb12-9a4149b869ea" path="/var/lib/kubelet/pods/c597582e-cf9f-4ef9-bb12-9a4149b869ea/volumes"
Jul 16 21:24:02.730736 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.730703    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq"]
Jul 16 21:24:02.731123 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731003    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83b9d37d-c707-4d02-8e03-e9a2a4db77e1" containerName="revoke-keys"
Jul 16 21:24:02.731123 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731013    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="83b9d37d-c707-4d02-8e03-e9a2a4db77e1" containerName="revoke-keys"
Jul 16 21:24:02.731123 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731023    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c597582e-cf9f-4ef9-bb12-9a4149b869ea" containerName="revoke-keys"
Jul 16 21:24:02.731123 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731028    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c597582e-cf9f-4ef9-bb12-9a4149b869ea" containerName="revoke-keys"
Jul 16 21:24:02.731123 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731035    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48be669d-8a80-4475-912d-1157fae67690" containerName="revoke-keys"
Jul 16 21:24:02.731123 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731041    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="48be669d-8a80-4475-912d-1157fae67690" containerName="revoke-keys"
Jul 16 21:24:02.731123 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731050    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48be669d-8a80-4475-912d-1157fae67690" containerName="revoke-keys"
Jul 16 21:24:02.731123 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731055    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="48be669d-8a80-4475-912d-1157fae67690" containerName="revoke-keys"
Jul 16 21:24:02.731123 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731067    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c597582e-cf9f-4ef9-bb12-9a4149b869ea" containerName="revoke-keys"
Jul 16 21:24:02.731123 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731072    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c597582e-cf9f-4ef9-bb12-9a4149b869ea" containerName="revoke-keys"
Jul 16 21:24:02.731123 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731083    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83b9d37d-c707-4d02-8e03-e9a2a4db77e1" containerName="revoke-keys"
Jul 16 21:24:02.731123 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731088    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="83b9d37d-c707-4d02-8e03-e9a2a4db77e1" containerName="revoke-keys"
Jul 16 21:24:02.731495 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731166    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="48be669d-8a80-4475-912d-1157fae67690" containerName="revoke-keys"
Jul 16 21:24:02.731495 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731173    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c597582e-cf9f-4ef9-bb12-9a4149b869ea" containerName="revoke-keys"
Jul 16 21:24:02.731495 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731178    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="48be669d-8a80-4475-912d-1157fae67690" containerName="revoke-keys"
Jul 16 21:24:02.731495 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731184    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c597582e-cf9f-4ef9-bb12-9a4149b869ea" containerName="revoke-keys"
Jul 16 21:24:02.731495 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731191    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="83b9d37d-c707-4d02-8e03-e9a2a4db77e1" containerName="revoke-keys"
Jul 16 21:24:02.731495 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731196    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="83b9d37d-c707-4d02-8e03-e9a2a4db77e1" containerName="revoke-keys"
Jul 16 21:24:02.731495 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.731201    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="83b9d37d-c707-4d02-8e03-e9a2a4db77e1" containerName="revoke-keys"
Jul 16 21:24:02.735477 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.735462    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq"
Jul 16 21:24:02.737576 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.737554    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:24:02.741261 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.741239    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq"]
Jul 16 21:24:02.820286 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.820258    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c46bab2e-4a63-43d0-8600-7bc183506cbf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq\" (UID: \"c46bab2e-4a63-43d0-8600-7bc183506cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq"
Jul 16 21:24:02.921021 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.920989    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c46bab2e-4a63-43d0-8600-7bc183506cbf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq\" (UID: \"c46bab2e-4a63-43d0-8600-7bc183506cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq"
Jul 16 21:24:02.921621 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:02.921605    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c46bab2e-4a63-43d0-8600-7bc183506cbf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq\" (UID: \"c46bab2e-4a63-43d0-8600-7bc183506cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq"
Jul 16 21:24:03.046883 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:03.046817    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq"
Jul 16 21:24:03.385170 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:03.385145    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq"]
Jul 16 21:24:03.387626 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:24:03.387599    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc46bab2e_4a63_43d0_8600_7bc183506cbf.slice/crio-1fd6fe0bfc17bb3684c4067bcfc4b268e1f3cad84dbc6c706bb51f1b8bb985e5 WatchSource:0}: Error finding container 1fd6fe0bfc17bb3684c4067bcfc4b268e1f3cad84dbc6c706bb51f1b8bb985e5: Status 404 returned error can't find the container with id 1fd6fe0bfc17bb3684c4067bcfc4b268e1f3cad84dbc6c706bb51f1b8bb985e5
Jul 16 21:24:03.745820 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:03.745785    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq" event={"ID":"c46bab2e-4a63-43d0-8600-7bc183506cbf","Type":"ContainerStarted","Data":"0698a9771b95f960547aecc263bd314817c5580e91c03de0f1ee2aa29490cef0"}
Jul 16 21:24:03.746177 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:03.745876    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq" event={"ID":"c46bab2e-4a63-43d0-8600-7bc183506cbf","Type":"ContainerStarted","Data":"1fd6fe0bfc17bb3684c4067bcfc4b268e1f3cad84dbc6c706bb51f1b8bb985e5"}
Jul 16 21:24:03.761572 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:03.761524    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq" podStartSLOduration=1.7615095219999999 podStartE2EDuration="1.761509522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:24:03.760647668 +0000 UTC m=+3081.782235435" watchObservedRunningTime="2026-07-16 21:24:03.761509522 +0000 UTC m=+3081.783097333"
Jul 16 21:24:04.751404 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:04.751373    2530 generic.go:356] "Generic (PLEG): container finished" podID="c46bab2e-4a63-43d0-8600-7bc183506cbf" containerID="0698a9771b95f960547aecc263bd314817c5580e91c03de0f1ee2aa29490cef0" exitCode=6
Jul 16 21:24:04.751803 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:04.751456    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq" event={"ID":"c46bab2e-4a63-43d0-8600-7bc183506cbf","Type":"ContainerDied","Data":"0698a9771b95f960547aecc263bd314817c5580e91c03de0f1ee2aa29490cef0"}
Jul 16 21:24:04.751803 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:04.751626    2530 scope.go:117] "RemoveContainer" containerID="0698a9771b95f960547aecc263bd314817c5580e91c03de0f1ee2aa29490cef0"
Jul 16 21:24:05.757948 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:05.757914    2530 generic.go:356] "Generic (PLEG): container finished" podID="c46bab2e-4a63-43d0-8600-7bc183506cbf" containerID="6b9462cd332e72e61c9c52217c880361dd60210608af65eca234835decd6097b" exitCode=6
Jul 16 21:24:05.758329 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:05.757997    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq" event={"ID":"c46bab2e-4a63-43d0-8600-7bc183506cbf","Type":"ContainerDied","Data":"6b9462cd332e72e61c9c52217c880361dd60210608af65eca234835decd6097b"}
Jul 16 21:24:05.758329 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:05.758058    2530 scope.go:117] "RemoveContainer" containerID="0698a9771b95f960547aecc263bd314817c5580e91c03de0f1ee2aa29490cef0"
Jul 16 21:24:05.758329 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:05.758218    2530 scope.go:117] "RemoveContainer" containerID="6b9462cd332e72e61c9c52217c880361dd60210608af65eca234835decd6097b"
Jul 16 21:24:05.758452 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:24:05.758388    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq_odh-ai-gateway-infra(c46bab2e-4a63-43d0-8600-7bc183506cbf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq" podUID="c46bab2e-4a63-43d0-8600-7bc183506cbf"
Jul 16 21:24:06.764437 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:06.764409    2530 scope.go:117] "RemoveContainer" containerID="6b9462cd332e72e61c9c52217c880361dd60210608af65eca234835decd6097b"
Jul 16 21:24:06.764814 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:24:06.764592    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq_odh-ai-gateway-infra(c46bab2e-4a63-43d0-8600-7bc183506cbf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq" podUID="c46bab2e-4a63-43d0-8600-7bc183506cbf"
Jul 16 21:24:19.502057 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:19.502022    2530 scope.go:117] "RemoveContainer" containerID="6b9462cd332e72e61c9c52217c880361dd60210608af65eca234835decd6097b"
Jul 16 21:24:19.828764 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:19.828709    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq" event={"ID":"c46bab2e-4a63-43d0-8600-7bc183506cbf","Type":"ContainerStarted","Data":"7f038cf2615a96237016dd0b05604c11b0b8efa118f8d48115a15018e3af9cc9"}
Jul 16 21:24:20.835716 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:20.835680    2530 generic.go:356] "Generic (PLEG): container finished" podID="c46bab2e-4a63-43d0-8600-7bc183506cbf" containerID="7f038cf2615a96237016dd0b05604c11b0b8efa118f8d48115a15018e3af9cc9" exitCode=6
Jul 16 21:24:20.836148 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:20.835755    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq" event={"ID":"c46bab2e-4a63-43d0-8600-7bc183506cbf","Type":"ContainerDied","Data":"7f038cf2615a96237016dd0b05604c11b0b8efa118f8d48115a15018e3af9cc9"}
Jul 16 21:24:20.836148 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:20.835819    2530 scope.go:117] "RemoveContainer" containerID="6b9462cd332e72e61c9c52217c880361dd60210608af65eca234835decd6097b"
Jul 16 21:24:20.836148 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:20.836128    2530 scope.go:117] "RemoveContainer" containerID="7f038cf2615a96237016dd0b05604c11b0b8efa118f8d48115a15018e3af9cc9"
Jul 16 21:24:20.836385 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:24:20.836360    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq_odh-ai-gateway-infra(c46bab2e-4a63-43d0-8600-7bc183506cbf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq" podUID="c46bab2e-4a63-43d0-8600-7bc183506cbf"
Jul 16 21:24:20.853014 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:20.852991    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq"]
Jul 16 21:24:21.977524 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:21.977502    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq"
Jul 16 21:24:22.071689 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.071666    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c46bab2e-4a63-43d0-8600-7bc183506cbf-maas-api-service-ca\") pod \"c46bab2e-4a63-43d0-8600-7bc183506cbf\" (UID: \"c46bab2e-4a63-43d0-8600-7bc183506cbf\") "
Jul 16 21:24:22.071986 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.071967    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c46bab2e-4a63-43d0-8600-7bc183506cbf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c46bab2e-4a63-43d0-8600-7bc183506cbf" (UID: "c46bab2e-4a63-43d0-8600-7bc183506cbf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:24:22.172858 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.172782    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c46bab2e-4a63-43d0-8600-7bc183506cbf-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:24:22.238734 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.238704    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn"]
Jul 16 21:24:22.239046 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.239033    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c46bab2e-4a63-43d0-8600-7bc183506cbf" containerName="revoke-keys"
Jul 16 21:24:22.239092 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.239048    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c46bab2e-4a63-43d0-8600-7bc183506cbf" containerName="revoke-keys"
Jul 16 21:24:22.239092 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.239058    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48be669d-8a80-4475-912d-1157fae67690" containerName="revoke-keys"
Jul 16 21:24:22.239092 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.239065    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="48be669d-8a80-4475-912d-1157fae67690" containerName="revoke-keys"
Jul 16 21:24:22.239092 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.239075    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c46bab2e-4a63-43d0-8600-7bc183506cbf" containerName="revoke-keys"
Jul 16 21:24:22.239092 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.239081    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c46bab2e-4a63-43d0-8600-7bc183506cbf" containerName="revoke-keys"
Jul 16 21:24:22.239092 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.239094    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c46bab2e-4a63-43d0-8600-7bc183506cbf" containerName="revoke-keys"
Jul 16 21:24:22.239299 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.239115    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c46bab2e-4a63-43d0-8600-7bc183506cbf" containerName="revoke-keys"
Jul 16 21:24:22.239299 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.239127    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83b9d37d-c707-4d02-8e03-e9a2a4db77e1" containerName="revoke-keys"
Jul 16 21:24:22.239299 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.239134    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="83b9d37d-c707-4d02-8e03-e9a2a4db77e1" containerName="revoke-keys"
Jul 16 21:24:22.239299 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.239143    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c597582e-cf9f-4ef9-bb12-9a4149b869ea" containerName="revoke-keys"
Jul 16 21:24:22.239299 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.239148    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="c597582e-cf9f-4ef9-bb12-9a4149b869ea" containerName="revoke-keys"
Jul 16 21:24:22.239299 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.239214    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c46bab2e-4a63-43d0-8600-7bc183506cbf" containerName="revoke-keys"
Jul 16 21:24:22.239299 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.239224    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c46bab2e-4a63-43d0-8600-7bc183506cbf" containerName="revoke-keys"
Jul 16 21:24:22.239299 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.239231    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="48be669d-8a80-4475-912d-1157fae67690" containerName="revoke-keys"
Jul 16 21:24:22.239299 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.239236    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c597582e-cf9f-4ef9-bb12-9a4149b869ea" containerName="revoke-keys"
Jul 16 21:24:22.242172 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.242156    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn"
Jul 16 21:24:22.251274 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.250356    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn"]
Jul 16 21:24:22.273174 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.273148    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d90b91-556e-4081-b51b-b8a63d2e4638-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn\" (UID: \"b9d90b91-556e-4081-b51b-b8a63d2e4638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn"
Jul 16 21:24:22.374417 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.374387    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d90b91-556e-4081-b51b-b8a63d2e4638-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn\" (UID: \"b9d90b91-556e-4081-b51b-b8a63d2e4638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn"
Jul 16 21:24:22.375042 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.375023    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d90b91-556e-4081-b51b-b8a63d2e4638-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn\" (UID: \"b9d90b91-556e-4081-b51b-b8a63d2e4638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn"
Jul 16 21:24:22.558978 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.558946    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn"
Jul 16 21:24:22.688820 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.688796    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn"]
Jul 16 21:24:22.690400 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:24:22.690379    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d90b91_556e_4081_b51b_b8a63d2e4638.slice/crio-27c722f3ea1cb17ae3c2d34107384fa261af576da1414875fceb782373ad0162 WatchSource:0}: Error finding container 27c722f3ea1cb17ae3c2d34107384fa261af576da1414875fceb782373ad0162: Status 404 returned error can't find the container with id 27c722f3ea1cb17ae3c2d34107384fa261af576da1414875fceb782373ad0162
Jul 16 21:24:22.849321 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.849255    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq"
Jul 16 21:24:22.849321 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.849264    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq" event={"ID":"c46bab2e-4a63-43d0-8600-7bc183506cbf","Type":"ContainerDied","Data":"1fd6fe0bfc17bb3684c4067bcfc4b268e1f3cad84dbc6c706bb51f1b8bb985e5"}
Jul 16 21:24:22.849497 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.849334    2530 scope.go:117] "RemoveContainer" containerID="7f038cf2615a96237016dd0b05604c11b0b8efa118f8d48115a15018e3af9cc9"
Jul 16 21:24:22.850509 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.850485    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn" event={"ID":"b9d90b91-556e-4081-b51b-b8a63d2e4638","Type":"ContainerStarted","Data":"27c722f3ea1cb17ae3c2d34107384fa261af576da1414875fceb782373ad0162"}
Jul 16 21:24:22.864969 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.864947    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq"]
Jul 16 21:24:22.871783 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:22.871764    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-46ngq"]
Jul 16 21:24:23.858133 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:23.858080    2530 generic.go:356] "Generic (PLEG): container finished" podID="b9d90b91-556e-4081-b51b-b8a63d2e4638" containerID="8cc5f46223a99d7807b773420e0935a305b0bcf1cc6981542b5415641db9448d" exitCode=6
Jul 16 21:24:23.858549 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:23.858157    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn" event={"ID":"b9d90b91-556e-4081-b51b-b8a63d2e4638","Type":"ContainerDied","Data":"8cc5f46223a99d7807b773420e0935a305b0bcf1cc6981542b5415641db9448d"}
Jul 16 21:24:23.858549 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:23.858328    2530 scope.go:117] "RemoveContainer" containerID="8cc5f46223a99d7807b773420e0935a305b0bcf1cc6981542b5415641db9448d"
Jul 16 21:24:24.393352 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:24.393320    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq"]
Jul 16 21:24:24.393703 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:24.393691    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="c46bab2e-4a63-43d0-8600-7bc183506cbf" containerName="revoke-keys"
Jul 16 21:24:24.396573 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:24.396558    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq"
Jul 16 21:24:24.404657 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:24.404626    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq"]
Jul 16 21:24:24.491496 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:24.491469    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e19b254-58cc-4ff4-9684-e4ec2d84caa7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq\" (UID: \"2e19b254-58cc-4ff4-9684-e4ec2d84caa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq"
Jul 16 21:24:24.507575 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:24.507545    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c46bab2e-4a63-43d0-8600-7bc183506cbf" path="/var/lib/kubelet/pods/c46bab2e-4a63-43d0-8600-7bc183506cbf/volumes"
Jul 16 21:24:24.592216 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:24.592179    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e19b254-58cc-4ff4-9684-e4ec2d84caa7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq\" (UID: \"2e19b254-58cc-4ff4-9684-e4ec2d84caa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq"
Jul 16 21:24:24.592774 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:24.592756    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e19b254-58cc-4ff4-9684-e4ec2d84caa7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq\" (UID: \"2e19b254-58cc-4ff4-9684-e4ec2d84caa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq"
Jul 16 21:24:24.707615 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:24.707586    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq"
Jul 16 21:24:24.835578 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:24.835536    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq"]
Jul 16 21:24:24.838389 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:24:24.838355    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e19b254_58cc_4ff4_9684_e4ec2d84caa7.slice/crio-86dc3967ffbb9f1239163c9439b0304218153cef2c5ae9b316dfc678ab57a6d4 WatchSource:0}: Error finding container 86dc3967ffbb9f1239163c9439b0304218153cef2c5ae9b316dfc678ab57a6d4: Status 404 returned error can't find the container with id 86dc3967ffbb9f1239163c9439b0304218153cef2c5ae9b316dfc678ab57a6d4
Jul 16 21:24:24.863666 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:24.863639    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq" event={"ID":"2e19b254-58cc-4ff4-9684-e4ec2d84caa7","Type":"ContainerStarted","Data":"86dc3967ffbb9f1239163c9439b0304218153cef2c5ae9b316dfc678ab57a6d4"}
Jul 16 21:24:24.865494 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:24.865468    2530 generic.go:356] "Generic (PLEG): container finished" podID="b9d90b91-556e-4081-b51b-b8a63d2e4638" containerID="badd5e41d0fcdf8be40ee9c7ff27c7b47b479fcbc11be17fa8b61954e03dea95" exitCode=6
Jul 16 21:24:24.865600 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:24.865531    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn" event={"ID":"b9d90b91-556e-4081-b51b-b8a63d2e4638","Type":"ContainerDied","Data":"badd5e41d0fcdf8be40ee9c7ff27c7b47b479fcbc11be17fa8b61954e03dea95"}
Jul 16 21:24:24.865600 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:24.865574    2530 scope.go:117] "RemoveContainer" containerID="8cc5f46223a99d7807b773420e0935a305b0bcf1cc6981542b5415641db9448d"
Jul 16 21:24:24.865750 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:24.865732    2530 scope.go:117] "RemoveContainer" containerID="badd5e41d0fcdf8be40ee9c7ff27c7b47b479fcbc11be17fa8b61954e03dea95"
Jul 16 21:24:24.865972 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:24:24.865948    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn_odh-ai-gateway-infra(b9d90b91-556e-4081-b51b-b8a63d2e4638)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn" podUID="b9d90b91-556e-4081-b51b-b8a63d2e4638"
Jul 16 21:24:25.872403 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:25.872366    2530 generic.go:356] "Generic (PLEG): container finished" podID="2e19b254-58cc-4ff4-9684-e4ec2d84caa7" containerID="6046111e110f7f9fdc2fb1db18d9a3be3abae6cf04902c90e1716306ead898d4" exitCode=6
Jul 16 21:24:25.872888 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:25.872422    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq" event={"ID":"2e19b254-58cc-4ff4-9684-e4ec2d84caa7","Type":"ContainerDied","Data":"6046111e110f7f9fdc2fb1db18d9a3be3abae6cf04902c90e1716306ead898d4"}
Jul 16 21:24:25.872888 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:25.872684    2530 scope.go:117] "RemoveContainer" containerID="6046111e110f7f9fdc2fb1db18d9a3be3abae6cf04902c90e1716306ead898d4"
Jul 16 21:24:25.874543 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:25.874525    2530 scope.go:117] "RemoveContainer" containerID="badd5e41d0fcdf8be40ee9c7ff27c7b47b479fcbc11be17fa8b61954e03dea95"
Jul 16 21:24:25.874709 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:24:25.874691    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn_odh-ai-gateway-infra(b9d90b91-556e-4081-b51b-b8a63d2e4638)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn" podUID="b9d90b91-556e-4081-b51b-b8a63d2e4638"
Jul 16 21:24:26.881160 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:26.881132    2530 generic.go:356] "Generic (PLEG): container finished" podID="2e19b254-58cc-4ff4-9684-e4ec2d84caa7" containerID="4657d06329329d59e23371438953aadbf645033a773be92e1cf4d801ffb8617e" exitCode=6
Jul 16 21:24:26.881589 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:26.881220    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq" event={"ID":"2e19b254-58cc-4ff4-9684-e4ec2d84caa7","Type":"ContainerDied","Data":"4657d06329329d59e23371438953aadbf645033a773be92e1cf4d801ffb8617e"}
Jul 16 21:24:26.881589 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:26.881282    2530 scope.go:117] "RemoveContainer" containerID="6046111e110f7f9fdc2fb1db18d9a3be3abae6cf04902c90e1716306ead898d4"
Jul 16 21:24:26.881589 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:26.881394    2530 scope.go:117] "RemoveContainer" containerID="4657d06329329d59e23371438953aadbf645033a773be92e1cf4d801ffb8617e"
Jul 16 21:24:26.881589 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:24:26.881584    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq_odh-ai-gateway-infra(2e19b254-58cc-4ff4-9684-e4ec2d84caa7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq" podUID="2e19b254-58cc-4ff4-9684-e4ec2d84caa7"
Jul 16 21:24:27.888119 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:27.888073    2530 scope.go:117] "RemoveContainer" containerID="4657d06329329d59e23371438953aadbf645033a773be92e1cf4d801ffb8617e"
Jul 16 21:24:27.888561 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:24:27.888266    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq_odh-ai-gateway-infra(2e19b254-58cc-4ff4-9684-e4ec2d84caa7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq" podUID="2e19b254-58cc-4ff4-9684-e4ec2d84caa7"
Jul 16 21:24:29.643277 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:29.643246    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw"]
Jul 16 21:24:29.647857 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:29.647837    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw"
Jul 16 21:24:29.655767 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:29.655741    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw"]
Jul 16 21:24:29.728976 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:29.728948    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94050be-ef6c-4f83-b047-3739e14fde4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw\" (UID: \"e94050be-ef6c-4f83-b047-3739e14fde4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw"
Jul 16 21:24:29.830094 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:29.830065    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94050be-ef6c-4f83-b047-3739e14fde4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw\" (UID: \"e94050be-ef6c-4f83-b047-3739e14fde4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw"
Jul 16 21:24:29.830668 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:29.830650    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94050be-ef6c-4f83-b047-3739e14fde4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw\" (UID: \"e94050be-ef6c-4f83-b047-3739e14fde4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw"
Jul 16 21:24:29.959562 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:29.959537    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw"
Jul 16 21:24:30.088193 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:30.088167    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw"]
Jul 16 21:24:30.090453 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:24:30.090426    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94050be_ef6c_4f83_b047_3739e14fde4f.slice/crio-c1b58562f4a54b1150fcbffdb6fafd413abae3e3cc6cb75e0b7eb280ee94c75b WatchSource:0}: Error finding container c1b58562f4a54b1150fcbffdb6fafd413abae3e3cc6cb75e0b7eb280ee94c75b: Status 404 returned error can't find the container with id c1b58562f4a54b1150fcbffdb6fafd413abae3e3cc6cb75e0b7eb280ee94c75b
Jul 16 21:24:30.904094 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:30.904057    2530 generic.go:356] "Generic (PLEG): container finished" podID="e94050be-ef6c-4f83-b047-3739e14fde4f" containerID="2a387599af40bcc4db18502df4f811a03e184d637a7b57e6c8cd2179a968d345" exitCode=6
Jul 16 21:24:30.904454 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:30.904147    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw" event={"ID":"e94050be-ef6c-4f83-b047-3739e14fde4f","Type":"ContainerDied","Data":"2a387599af40bcc4db18502df4f811a03e184d637a7b57e6c8cd2179a968d345"}
Jul 16 21:24:30.904454 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:30.904209    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw" event={"ID":"e94050be-ef6c-4f83-b047-3739e14fde4f","Type":"ContainerStarted","Data":"c1b58562f4a54b1150fcbffdb6fafd413abae3e3cc6cb75e0b7eb280ee94c75b"}
Jul 16 21:24:30.904454 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:30.904317    2530 scope.go:117] "RemoveContainer" containerID="2a387599af40bcc4db18502df4f811a03e184d637a7b57e6c8cd2179a968d345"
Jul 16 21:24:31.910992 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:31.910914    2530 generic.go:356] "Generic (PLEG): container finished" podID="e94050be-ef6c-4f83-b047-3739e14fde4f" containerID="b2f8d4a726ab37353194255f5e5f31c00079d53e44fb216de4fc342f8dc3368b" exitCode=6
Jul 16 21:24:31.910992 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:31.910967    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw" event={"ID":"e94050be-ef6c-4f83-b047-3739e14fde4f","Type":"ContainerDied","Data":"b2f8d4a726ab37353194255f5e5f31c00079d53e44fb216de4fc342f8dc3368b"}
Jul 16 21:24:31.911444 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:31.911026    2530 scope.go:117] "RemoveContainer" containerID="2a387599af40bcc4db18502df4f811a03e184d637a7b57e6c8cd2179a968d345"
Jul 16 21:24:31.911444 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:31.911223    2530 scope.go:117] "RemoveContainer" containerID="b2f8d4a726ab37353194255f5e5f31c00079d53e44fb216de4fc342f8dc3368b"
Jul 16 21:24:31.911534 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:24:31.911464    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw_odh-ai-gateway-infra(e94050be-ef6c-4f83-b047-3739e14fde4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw" podUID="e94050be-ef6c-4f83-b047-3739e14fde4f"
Jul 16 21:24:32.917454 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:32.917425    2530 scope.go:117] "RemoveContainer" containerID="b2f8d4a726ab37353194255f5e5f31c00079d53e44fb216de4fc342f8dc3368b"
Jul 16 21:24:32.917817 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:24:32.917588    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw_odh-ai-gateway-infra(e94050be-ef6c-4f83-b047-3739e14fde4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw" podUID="e94050be-ef6c-4f83-b047-3739e14fde4f"
Jul 16 21:24:36.502921 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:36.502884    2530 scope.go:117] "RemoveContainer" containerID="badd5e41d0fcdf8be40ee9c7ff27c7b47b479fcbc11be17fa8b61954e03dea95"
Jul 16 21:24:36.937961 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:36.937886    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn" event={"ID":"b9d90b91-556e-4081-b51b-b8a63d2e4638","Type":"ContainerStarted","Data":"70dc9ea709e0f8c1d4c4b405a7e1028c7d5410c4dbd2253023545c78f5d1c2e8"}
Jul 16 21:24:36.955884 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:36.955842    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn" podStartSLOduration=14.95582674 podStartE2EDuration="14.95582674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:24:36.954266362 +0000 UTC m=+3114.975854130" watchObservedRunningTime="2026-07-16 21:24:36.95582674 +0000 UTC m=+3114.977414505"
Jul 16 21:24:37.945879 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:37.945842    2530 generic.go:356] "Generic (PLEG): container finished" podID="b9d90b91-556e-4081-b51b-b8a63d2e4638" containerID="70dc9ea709e0f8c1d4c4b405a7e1028c7d5410c4dbd2253023545c78f5d1c2e8" exitCode=6
Jul 16 21:24:37.946391 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:37.945893    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn" event={"ID":"b9d90b91-556e-4081-b51b-b8a63d2e4638","Type":"ContainerDied","Data":"70dc9ea709e0f8c1d4c4b405a7e1028c7d5410c4dbd2253023545c78f5d1c2e8"}
Jul 16 21:24:37.946391 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:37.945953    2530 scope.go:117] "RemoveContainer" containerID="badd5e41d0fcdf8be40ee9c7ff27c7b47b479fcbc11be17fa8b61954e03dea95"
Jul 16 21:24:37.946391 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:37.946249    2530 scope.go:117] "RemoveContainer" containerID="70dc9ea709e0f8c1d4c4b405a7e1028c7d5410c4dbd2253023545c78f5d1c2e8"
Jul 16 21:24:37.946520 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:24:37.946447    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn_odh-ai-gateway-infra(b9d90b91-556e-4081-b51b-b8a63d2e4638)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn" podUID="b9d90b91-556e-4081-b51b-b8a63d2e4638"
Jul 16 21:24:37.971534 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:37.971503    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn"]
Jul 16 21:24:39.092604 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:39.092578    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn"
Jul 16 21:24:39.207486 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:39.207406    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d90b91-556e-4081-b51b-b8a63d2e4638-maas-api-service-ca\") pod \"b9d90b91-556e-4081-b51b-b8a63d2e4638\" (UID: \"b9d90b91-556e-4081-b51b-b8a63d2e4638\") "
Jul 16 21:24:39.207736 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:39.207717    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d90b91-556e-4081-b51b-b8a63d2e4638-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b9d90b91-556e-4081-b51b-b8a63d2e4638" (UID: "b9d90b91-556e-4081-b51b-b8a63d2e4638"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:24:39.308038 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:39.308010    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d90b91-556e-4081-b51b-b8a63d2e4638-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:24:39.961793 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:39.961748    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn" event={"ID":"b9d90b91-556e-4081-b51b-b8a63d2e4638","Type":"ContainerDied","Data":"27c722f3ea1cb17ae3c2d34107384fa261af576da1414875fceb782373ad0162"}
Jul 16 21:24:39.961974 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:39.961770    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn"
Jul 16 21:24:39.961974 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:39.961837    2530 scope.go:117] "RemoveContainer" containerID="70dc9ea709e0f8c1d4c4b405a7e1028c7d5410c4dbd2253023545c78f5d1c2e8"
Jul 16 21:24:39.985124 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:39.985081    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn"]
Jul 16 21:24:39.991547 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:39.991526    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-8rbvn"]
Jul 16 21:24:40.507950 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:40.507916    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9d90b91-556e-4081-b51b-b8a63d2e4638" path="/var/lib/kubelet/pods/b9d90b91-556e-4081-b51b-b8a63d2e4638/volumes"
Jul 16 21:24:41.502438 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:41.502412    2530 scope.go:117] "RemoveContainer" containerID="4657d06329329d59e23371438953aadbf645033a773be92e1cf4d801ffb8617e"
Jul 16 21:24:41.973693 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:41.973657    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq" event={"ID":"2e19b254-58cc-4ff4-9684-e4ec2d84caa7","Type":"ContainerStarted","Data":"b0691af025f30e6c4d07cc91759e65e94362c99db9b29e97b0c575a76c44a078"}
Jul 16 21:24:41.990510 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:41.990466    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq" podStartSLOduration=17.990451077 podStartE2EDuration="17.990451077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:24:41.989256941 +0000 UTC m=+3120.010844707" watchObservedRunningTime="2026-07-16 21:24:41.990451077 +0000 UTC m=+3120.012038947"
Jul 16 21:24:42.980949 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:42.980911    2530 generic.go:356] "Generic (PLEG): container finished" podID="2e19b254-58cc-4ff4-9684-e4ec2d84caa7" containerID="b0691af025f30e6c4d07cc91759e65e94362c99db9b29e97b0c575a76c44a078" exitCode=6
Jul 16 21:24:42.981387 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:42.980988    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq" event={"ID":"2e19b254-58cc-4ff4-9684-e4ec2d84caa7","Type":"ContainerDied","Data":"b0691af025f30e6c4d07cc91759e65e94362c99db9b29e97b0c575a76c44a078"}
Jul 16 21:24:42.981387 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:42.981061    2530 scope.go:117] "RemoveContainer" containerID="4657d06329329d59e23371438953aadbf645033a773be92e1cf4d801ffb8617e"
Jul 16 21:24:42.981500 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:42.981387    2530 scope.go:117] "RemoveContainer" containerID="b0691af025f30e6c4d07cc91759e65e94362c99db9b29e97b0c575a76c44a078"
Jul 16 21:24:42.981602 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:24:42.981580    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq_odh-ai-gateway-infra(2e19b254-58cc-4ff4-9684-e4ec2d84caa7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq" podUID="2e19b254-58cc-4ff4-9684-e4ec2d84caa7"
Jul 16 21:24:43.005942 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:43.005914    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq"]
Jul 16 21:24:44.118720 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:44.118698    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq"
Jul 16 21:24:44.247410 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:44.247329    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e19b254-58cc-4ff4-9684-e4ec2d84caa7-maas-api-service-ca\") pod \"2e19b254-58cc-4ff4-9684-e4ec2d84caa7\" (UID: \"2e19b254-58cc-4ff4-9684-e4ec2d84caa7\") "
Jul 16 21:24:44.247678 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:44.247657    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e19b254-58cc-4ff4-9684-e4ec2d84caa7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2e19b254-58cc-4ff4-9684-e4ec2d84caa7" (UID: "2e19b254-58cc-4ff4-9684-e4ec2d84caa7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:24:44.348786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:44.348755    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e19b254-58cc-4ff4-9684-e4ec2d84caa7-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:24:44.993266 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:44.993229    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq"
Jul 16 21:24:44.993460 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:44.993257    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq" event={"ID":"2e19b254-58cc-4ff4-9684-e4ec2d84caa7","Type":"ContainerDied","Data":"86dc3967ffbb9f1239163c9439b0304218153cef2c5ae9b316dfc678ab57a6d4"}
Jul 16 21:24:44.993460 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:44.993336    2530 scope.go:117] "RemoveContainer" containerID="b0691af025f30e6c4d07cc91759e65e94362c99db9b29e97b0c575a76c44a078"
Jul 16 21:24:45.010837 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:45.010812    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq"]
Jul 16 21:24:45.017717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:45.017694    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9q5tq"]
Jul 16 21:24:46.508514 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:46.508478    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e19b254-58cc-4ff4-9684-e4ec2d84caa7" path="/var/lib/kubelet/pods/2e19b254-58cc-4ff4-9684-e4ec2d84caa7/volumes"
Jul 16 21:24:47.502737 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:47.502708    2530 scope.go:117] "RemoveContainer" containerID="b2f8d4a726ab37353194255f5e5f31c00079d53e44fb216de4fc342f8dc3368b"
Jul 16 21:24:48.011909 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:48.011884    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw" event={"ID":"e94050be-ef6c-4f83-b047-3739e14fde4f","Type":"ContainerStarted","Data":"bb648a3eb02a6261bf86c41b4784b3c82d647114e3d8afdf19e9e9ef7b1f6ef1"}
Jul 16 21:24:48.029936 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:48.029898    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw" podStartSLOduration=19.029884776 podStartE2EDuration="19.029884776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:24:48.028012145 +0000 UTC m=+3126.049599917" watchObservedRunningTime="2026-07-16 21:24:48.029884776 +0000 UTC m=+3126.051472576"
Jul 16 21:24:49.018073 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:49.018038    2530 generic.go:356] "Generic (PLEG): container finished" podID="e94050be-ef6c-4f83-b047-3739e14fde4f" containerID="bb648a3eb02a6261bf86c41b4784b3c82d647114e3d8afdf19e9e9ef7b1f6ef1" exitCode=6
Jul 16 21:24:49.018564 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:49.018126    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw" event={"ID":"e94050be-ef6c-4f83-b047-3739e14fde4f","Type":"ContainerDied","Data":"bb648a3eb02a6261bf86c41b4784b3c82d647114e3d8afdf19e9e9ef7b1f6ef1"}
Jul 16 21:24:49.018564 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:49.018185    2530 scope.go:117] "RemoveContainer" containerID="b2f8d4a726ab37353194255f5e5f31c00079d53e44fb216de4fc342f8dc3368b"
Jul 16 21:24:49.018564 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:49.018449    2530 scope.go:117] "RemoveContainer" containerID="bb648a3eb02a6261bf86c41b4784b3c82d647114e3d8afdf19e9e9ef7b1f6ef1"
Jul 16 21:24:49.018732 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:24:49.018691    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw_odh-ai-gateway-infra(e94050be-ef6c-4f83-b047-3739e14fde4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw" podUID="e94050be-ef6c-4f83-b047-3739e14fde4f"
Jul 16 21:24:49.044471 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:49.044450    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw"]
Jul 16 21:24:50.168973 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:50.168951    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw"
Jul 16 21:24:50.300406 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:50.300329    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94050be-ef6c-4f83-b047-3739e14fde4f-maas-api-service-ca\") pod \"e94050be-ef6c-4f83-b047-3739e14fde4f\" (UID: \"e94050be-ef6c-4f83-b047-3739e14fde4f\") "
Jul 16 21:24:50.300675 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:50.300652    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94050be-ef6c-4f83-b047-3739e14fde4f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e94050be-ef6c-4f83-b047-3739e14fde4f" (UID: "e94050be-ef6c-4f83-b047-3739e14fde4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:24:50.401724 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:50.401688    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94050be-ef6c-4f83-b047-3739e14fde4f-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:24:51.030288 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.030256    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw"
Jul 16 21:24:51.030493 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.030294    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw" event={"ID":"e94050be-ef6c-4f83-b047-3739e14fde4f","Type":"ContainerDied","Data":"c1b58562f4a54b1150fcbffdb6fafd413abae3e3cc6cb75e0b7eb280ee94c75b"}
Jul 16 21:24:51.030493 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.030381    2530 scope.go:117] "RemoveContainer" containerID="bb648a3eb02a6261bf86c41b4784b3c82d647114e3d8afdf19e9e9ef7b1f6ef1"
Jul 16 21:24:51.045899 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.045875    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw"]
Jul 16 21:24:51.050333 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.050308    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-bclnw"]
Jul 16 21:24:51.156045 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156008    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2gtt9"]
Jul 16 21:24:51.156453 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156435    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d90b91-556e-4081-b51b-b8a63d2e4638" containerName="revoke-keys"
Jul 16 21:24:51.156550 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156456    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d90b91-556e-4081-b51b-b8a63d2e4638" containerName="revoke-keys"
Jul 16 21:24:51.156550 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156471    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d90b91-556e-4081-b51b-b8a63d2e4638" containerName="revoke-keys"
Jul 16 21:24:51.156550 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156479    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d90b91-556e-4081-b51b-b8a63d2e4638" containerName="revoke-keys"
Jul 16 21:24:51.156550 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156497    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e19b254-58cc-4ff4-9684-e4ec2d84caa7" containerName="revoke-keys"
Jul 16 21:24:51.156550 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156506    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e19b254-58cc-4ff4-9684-e4ec2d84caa7" containerName="revoke-keys"
Jul 16 21:24:51.156550 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156523    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94050be-ef6c-4f83-b047-3739e14fde4f" containerName="revoke-keys"
Jul 16 21:24:51.156550 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156531    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94050be-ef6c-4f83-b047-3739e14fde4f" containerName="revoke-keys"
Jul 16 21:24:51.156550 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156540    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d90b91-556e-4081-b51b-b8a63d2e4638" containerName="revoke-keys"
Jul 16 21:24:51.156550 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156547    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d90b91-556e-4081-b51b-b8a63d2e4638" containerName="revoke-keys"
Jul 16 21:24:51.157002 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156556    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e19b254-58cc-4ff4-9684-e4ec2d84caa7" containerName="revoke-keys"
Jul 16 21:24:51.157002 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156564    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e19b254-58cc-4ff4-9684-e4ec2d84caa7" containerName="revoke-keys"
Jul 16 21:24:51.157002 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156581    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94050be-ef6c-4f83-b047-3739e14fde4f" containerName="revoke-keys"
Jul 16 21:24:51.157002 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156590    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94050be-ef6c-4f83-b047-3739e14fde4f" containerName="revoke-keys"
Jul 16 21:24:51.157002 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156602    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94050be-ef6c-4f83-b047-3739e14fde4f" containerName="revoke-keys"
Jul 16 21:24:51.157002 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156610    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94050be-ef6c-4f83-b047-3739e14fde4f" containerName="revoke-keys"
Jul 16 21:24:51.157002 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156623    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e19b254-58cc-4ff4-9684-e4ec2d84caa7" containerName="revoke-keys"
Jul 16 21:24:51.157002 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156631    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e19b254-58cc-4ff4-9684-e4ec2d84caa7" containerName="revoke-keys"
Jul 16 21:24:51.157002 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156717    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e19b254-58cc-4ff4-9684-e4ec2d84caa7" containerName="revoke-keys"
Jul 16 21:24:51.157002 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156727    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d90b91-556e-4081-b51b-b8a63d2e4638" containerName="revoke-keys"
Jul 16 21:24:51.157002 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156737    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="e94050be-ef6c-4f83-b047-3739e14fde4f" containerName="revoke-keys"
Jul 16 21:24:51.157002 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156749    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d90b91-556e-4081-b51b-b8a63d2e4638" containerName="revoke-keys"
Jul 16 21:24:51.157002 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156758    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="e94050be-ef6c-4f83-b047-3739e14fde4f" containerName="revoke-keys"
Jul 16 21:24:51.157002 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156767    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e19b254-58cc-4ff4-9684-e4ec2d84caa7" containerName="revoke-keys"
Jul 16 21:24:51.157002 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.156776    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e19b254-58cc-4ff4-9684-e4ec2d84caa7" containerName="revoke-keys"
Jul 16 21:24:51.161301 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.161271    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2gtt9"
Jul 16 21:24:51.163899 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.163881    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-cd8xg\""
Jul 16 21:24:51.167981 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.167958    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2gtt9"]
Jul 16 21:24:51.309424 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.309362    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqjqh\" (UniqueName: \"kubernetes.io/projected/bf3527ad-d923-4139-a21f-c9852e940c7b-kube-api-access-dqjqh\") pod \"kuadrant-operator-catalog-2gtt9\" (UID: \"bf3527ad-d923-4139-a21f-c9852e940c7b\") " pod="kuadrant-system/kuadrant-operator-catalog-2gtt9"
Jul 16 21:24:51.410513 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.410486    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqjqh\" (UniqueName: \"kubernetes.io/projected/bf3527ad-d923-4139-a21f-c9852e940c7b-kube-api-access-dqjqh\") pod \"kuadrant-operator-catalog-2gtt9\" (UID: \"bf3527ad-d923-4139-a21f-c9852e940c7b\") " pod="kuadrant-system/kuadrant-operator-catalog-2gtt9"
Jul 16 21:24:51.418708 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.418688    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqjqh\" (UniqueName: \"kubernetes.io/projected/bf3527ad-d923-4139-a21f-c9852e940c7b-kube-api-access-dqjqh\") pod \"kuadrant-operator-catalog-2gtt9\" (UID: \"bf3527ad-d923-4139-a21f-c9852e940c7b\") " pod="kuadrant-system/kuadrant-operator-catalog-2gtt9"
Jul 16 21:24:51.472340 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.472315    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2gtt9"
Jul 16 21:24:51.596426 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:51.596341    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2gtt9"]
Jul 16 21:24:51.598711 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:24:51.598681    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3527ad_d923_4139_a21f_c9852e940c7b.slice/crio-fedef19a5fe3fd2a74a4959914bd859db653a709262da555d2ccdd0372ed328c WatchSource:0}: Error finding container fedef19a5fe3fd2a74a4959914bd859db653a709262da555d2ccdd0372ed328c: Status 404 returned error can't find the container with id fedef19a5fe3fd2a74a4959914bd859db653a709262da555d2ccdd0372ed328c
Jul 16 21:24:52.037738 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:52.037695    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2gtt9" event={"ID":"bf3527ad-d923-4139-a21f-c9852e940c7b","Type":"ContainerStarted","Data":"fedef19a5fe3fd2a74a4959914bd859db653a709262da555d2ccdd0372ed328c"}
Jul 16 21:24:52.510076 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:52.510041    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e94050be-ef6c-4f83-b047-3739e14fde4f" path="/var/lib/kubelet/pods/e94050be-ef6c-4f83-b047-3739e14fde4f/volumes"
Jul 16 21:24:54.050713 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:54.050674    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2gtt9" event={"ID":"bf3527ad-d923-4139-a21f-c9852e940c7b","Type":"ContainerStarted","Data":"6c69e47c85838e495baeb810ad79f098df0545cb61433a559668c887b7f7d14a"}
Jul 16 21:24:54.067111 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:24:54.067041    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-2gtt9" podStartSLOduration=1.3066795199999999 podStartE2EDuration="3.067025094s" totalImagesPullingTime="1.760345574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:24:54.066679837 +0000 UTC m=+3132.088267604" watchObservedRunningTime="2026-07-16 21:24:54.067025094 +0000 UTC m=+3132.088612859"
Jul 16 21:25:01.472433 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:01.472399    2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-2gtt9"
Jul 16 21:25:01.472983 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:01.472500    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-2gtt9"
Jul 16 21:25:01.496477 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:01.496449    2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-2gtt9"
Jul 16 21:25:02.114122 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:02.114078    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-2gtt9"
Jul 16 21:25:02.150238 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:02.150199    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2gtt9"]
Jul 16 21:25:02.782685 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:02.782648    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9"]
Jul 16 21:25:02.783069 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:02.783044    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d90b91-556e-4081-b51b-b8a63d2e4638" containerName="revoke-keys"
Jul 16 21:25:02.783069 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:02.783057    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="e94050be-ef6c-4f83-b047-3739e14fde4f" containerName="revoke-keys"
Jul 16 21:25:02.786526 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:02.786503    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9"
Jul 16 21:25:02.789024 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:02.789000    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:25:02.794609 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:02.794586    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9"]
Jul 16 21:25:02.805340 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:02.805289    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe65226-301f-4d1a-8647-7c763a2b438a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9\" (UID: \"4fe65226-301f-4d1a-8647-7c763a2b438a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9"
Jul 16 21:25:02.905707 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:02.905682    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe65226-301f-4d1a-8647-7c763a2b438a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9\" (UID: \"4fe65226-301f-4d1a-8647-7c763a2b438a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9"
Jul 16 21:25:02.906343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:02.906327    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe65226-301f-4d1a-8647-7c763a2b438a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9\" (UID: \"4fe65226-301f-4d1a-8647-7c763a2b438a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9"
Jul 16 21:25:03.097039 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:03.096968    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9"
Jul 16 21:25:03.228650 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:03.228628    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9"]
Jul 16 21:25:03.230827 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:25:03.230799    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe65226_301f_4d1a_8647_7c763a2b438a.slice/crio-f8183945511c6c08533964e1d826201bf908fbc07cb0ffaa3d3f1514e7ec22ba WatchSource:0}: Error finding container f8183945511c6c08533964e1d826201bf908fbc07cb0ffaa3d3f1514e7ec22ba: Status 404 returned error can't find the container with id f8183945511c6c08533964e1d826201bf908fbc07cb0ffaa3d3f1514e7ec22ba
Jul 16 21:25:04.102636 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:04.102550    2530 generic.go:356] "Generic (PLEG): container finished" podID="4fe65226-301f-4d1a-8647-7c763a2b438a" containerID="d51df01a705206030aa0105d167fd7babd566cc82d07e8a9d14d73121a3f31d8" exitCode=6
Jul 16 21:25:04.103074 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:04.102635    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9" event={"ID":"4fe65226-301f-4d1a-8647-7c763a2b438a","Type":"ContainerDied","Data":"d51df01a705206030aa0105d167fd7babd566cc82d07e8a9d14d73121a3f31d8"}
Jul 16 21:25:04.103074 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:04.102690    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9" event={"ID":"4fe65226-301f-4d1a-8647-7c763a2b438a","Type":"ContainerStarted","Data":"f8183945511c6c08533964e1d826201bf908fbc07cb0ffaa3d3f1514e7ec22ba"}
Jul 16 21:25:04.103074 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:04.102781    2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-2gtt9" podUID="bf3527ad-d923-4139-a21f-c9852e940c7b" containerName="registry-server" containerID="cri-o://6c69e47c85838e495baeb810ad79f098df0545cb61433a559668c887b7f7d14a" gracePeriod=2
Jul 16 21:25:04.103074 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:04.102842    2530 scope.go:117] "RemoveContainer" containerID="d51df01a705206030aa0105d167fd7babd566cc82d07e8a9d14d73121a3f31d8"
Jul 16 21:25:04.350484 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:04.350459    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2gtt9"
Jul 16 21:25:04.415136 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:04.415061    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqjqh\" (UniqueName: \"kubernetes.io/projected/bf3527ad-d923-4139-a21f-c9852e940c7b-kube-api-access-dqjqh\") pod \"bf3527ad-d923-4139-a21f-c9852e940c7b\" (UID: \"bf3527ad-d923-4139-a21f-c9852e940c7b\") "
Jul 16 21:25:04.417125 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:04.417090    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3527ad-d923-4139-a21f-c9852e940c7b-kube-api-access-dqjqh" (OuterVolumeSpecName: "kube-api-access-dqjqh") pod "bf3527ad-d923-4139-a21f-c9852e940c7b" (UID: "bf3527ad-d923-4139-a21f-c9852e940c7b"). InnerVolumeSpecName "kube-api-access-dqjqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:25:04.515764 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:04.515743    2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqjqh\" (UniqueName: \"kubernetes.io/projected/bf3527ad-d923-4139-a21f-c9852e940c7b-kube-api-access-dqjqh\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:25:05.109653 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:05.109564    2530 generic.go:356] "Generic (PLEG): container finished" podID="4fe65226-301f-4d1a-8647-7c763a2b438a" containerID="a67598cb7698e23937703ae041b881a081ac71599149a65ad987e3cbf8af9307" exitCode=6
Jul 16 21:25:05.110076 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:05.109656    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9" event={"ID":"4fe65226-301f-4d1a-8647-7c763a2b438a","Type":"ContainerDied","Data":"a67598cb7698e23937703ae041b881a081ac71599149a65ad987e3cbf8af9307"}
Jul 16 21:25:05.110076 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:05.109741    2530 scope.go:117] "RemoveContainer" containerID="d51df01a705206030aa0105d167fd7babd566cc82d07e8a9d14d73121a3f31d8"
Jul 16 21:25:05.110076 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:05.109862    2530 scope.go:117] "RemoveContainer" containerID="a67598cb7698e23937703ae041b881a081ac71599149a65ad987e3cbf8af9307"
Jul 16 21:25:05.110249 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:25:05.110142    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9_odh-ai-gateway-infra(4fe65226-301f-4d1a-8647-7c763a2b438a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9" podUID="4fe65226-301f-4d1a-8647-7c763a2b438a"
Jul 16 21:25:05.111326 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:05.111297    2530 generic.go:356] "Generic (PLEG): container finished" podID="bf3527ad-d923-4139-a21f-c9852e940c7b" containerID="6c69e47c85838e495baeb810ad79f098df0545cb61433a559668c887b7f7d14a" exitCode=0
Jul 16 21:25:05.111447 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:05.111331    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2gtt9"
Jul 16 21:25:05.111447 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:05.111382    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2gtt9" event={"ID":"bf3527ad-d923-4139-a21f-c9852e940c7b","Type":"ContainerDied","Data":"6c69e47c85838e495baeb810ad79f098df0545cb61433a559668c887b7f7d14a"}
Jul 16 21:25:05.111447 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:05.111421    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2gtt9" event={"ID":"bf3527ad-d923-4139-a21f-c9852e940c7b","Type":"ContainerDied","Data":"fedef19a5fe3fd2a74a4959914bd859db653a709262da555d2ccdd0372ed328c"}
Jul 16 21:25:05.123974 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:05.123959    2530 scope.go:117] "RemoveContainer" containerID="6c69e47c85838e495baeb810ad79f098df0545cb61433a559668c887b7f7d14a"
Jul 16 21:25:05.133956 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:05.133925    2530 scope.go:117] "RemoveContainer" containerID="6c69e47c85838e495baeb810ad79f098df0545cb61433a559668c887b7f7d14a"
Jul 16 21:25:05.134227 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:25:05.134208    2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c69e47c85838e495baeb810ad79f098df0545cb61433a559668c887b7f7d14a\": container with ID starting with 6c69e47c85838e495baeb810ad79f098df0545cb61433a559668c887b7f7d14a not found: ID does not exist" containerID="6c69e47c85838e495baeb810ad79f098df0545cb61433a559668c887b7f7d14a"
Jul 16 21:25:05.134290 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:05.134235    2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c69e47c85838e495baeb810ad79f098df0545cb61433a559668c887b7f7d14a"} err="failed to get container status \"6c69e47c85838e495baeb810ad79f098df0545cb61433a559668c887b7f7d14a\": rpc error: code = NotFound desc = could not find container \"6c69e47c85838e495baeb810ad79f098df0545cb61433a559668c887b7f7d14a\": container with ID starting with 6c69e47c85838e495baeb810ad79f098df0545cb61433a559668c887b7f7d14a not found: ID does not exist"
Jul 16 21:25:05.138402 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:05.138379    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2gtt9"]
Jul 16 21:25:05.142718 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:05.142694    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2gtt9"]
Jul 16 21:25:06.118325 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:06.118300    2530 scope.go:117] "RemoveContainer" containerID="a67598cb7698e23937703ae041b881a081ac71599149a65ad987e3cbf8af9307"
Jul 16 21:25:06.118786 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:25:06.118485    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9_odh-ai-gateway-infra(4fe65226-301f-4d1a-8647-7c763a2b438a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9" podUID="4fe65226-301f-4d1a-8647-7c763a2b438a"
Jul 16 21:25:06.507386 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:06.507356    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf3527ad-d923-4139-a21f-c9852e940c7b" path="/var/lib/kubelet/pods/bf3527ad-d923-4139-a21f-c9852e940c7b/volumes"
Jul 16 21:25:12.278774 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:12.278733    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv"]
Jul 16 21:25:12.279286 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:12.279267    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf3527ad-d923-4139-a21f-c9852e940c7b" containerName="registry-server"
Jul 16 21:25:12.279369 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:12.279289    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf3527ad-d923-4139-a21f-c9852e940c7b" containerName="registry-server"
Jul 16 21:25:12.279369 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:12.279366    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf3527ad-d923-4139-a21f-c9852e940c7b" containerName="registry-server"
Jul 16 21:25:12.283895 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:12.283872    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv"
Jul 16 21:25:12.290094 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:12.290056    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv"]
Jul 16 21:25:12.375633 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:12.375602    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6790dc74-39e3-4897-a00d-f38cfc530a92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv\" (UID: \"6790dc74-39e3-4897-a00d-f38cfc530a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv"
Jul 16 21:25:12.476666 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:12.476634    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6790dc74-39e3-4897-a00d-f38cfc530a92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv\" (UID: \"6790dc74-39e3-4897-a00d-f38cfc530a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv"
Jul 16 21:25:12.477336 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:12.477318    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6790dc74-39e3-4897-a00d-f38cfc530a92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv\" (UID: \"6790dc74-39e3-4897-a00d-f38cfc530a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv"
Jul 16 21:25:12.596743 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:12.596668    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv"
Jul 16 21:25:12.728280 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:12.728258    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv"]
Jul 16 21:25:12.730127 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:25:12.730079    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6790dc74_39e3_4897_a00d_f38cfc530a92.slice/crio-733e9f8fa92a672e1196872b6b01e190e644903d409964cb36ee44e29b45312f WatchSource:0}: Error finding container 733e9f8fa92a672e1196872b6b01e190e644903d409964cb36ee44e29b45312f: Status 404 returned error can't find the container with id 733e9f8fa92a672e1196872b6b01e190e644903d409964cb36ee44e29b45312f
Jul 16 21:25:13.153069 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:13.153039    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv" event={"ID":"6790dc74-39e3-4897-a00d-f38cfc530a92","Type":"ContainerStarted","Data":"f3fd4d0cfc12e8a507d0e3a2efedaea7802e5bb3ffd26f31a8d42ee8c69f6f26"}
Jul 16 21:25:13.153236 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:13.153089    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv" event={"ID":"6790dc74-39e3-4897-a00d-f38cfc530a92","Type":"ContainerStarted","Data":"733e9f8fa92a672e1196872b6b01e190e644903d409964cb36ee44e29b45312f"}
Jul 16 21:25:13.169207 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:13.169162    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv" podStartSLOduration=1.169147548 podStartE2EDuration="1.169147548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:25:13.167606418 +0000 UTC m=+3151.189194183" watchObservedRunningTime="2026-07-16 21:25:13.169147548 +0000 UTC m=+3151.190735314"
Jul 16 21:25:14.159372 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:14.159334    2530 generic.go:356] "Generic (PLEG): container finished" podID="6790dc74-39e3-4897-a00d-f38cfc530a92" containerID="f3fd4d0cfc12e8a507d0e3a2efedaea7802e5bb3ffd26f31a8d42ee8c69f6f26" exitCode=6
Jul 16 21:25:14.159794 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:14.159422    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv" event={"ID":"6790dc74-39e3-4897-a00d-f38cfc530a92","Type":"ContainerDied","Data":"f3fd4d0cfc12e8a507d0e3a2efedaea7802e5bb3ffd26f31a8d42ee8c69f6f26"}
Jul 16 21:25:14.159794 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:14.159583    2530 scope.go:117] "RemoveContainer" containerID="f3fd4d0cfc12e8a507d0e3a2efedaea7802e5bb3ffd26f31a8d42ee8c69f6f26"
Jul 16 21:25:15.166072 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:15.165999    2530 generic.go:356] "Generic (PLEG): container finished" podID="6790dc74-39e3-4897-a00d-f38cfc530a92" containerID="08ae722f5971746558e21eff027b0500b7e782b424171fcb2c64ed9b56fe8c25" exitCode=6
Jul 16 21:25:15.166508 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:15.166089    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv" event={"ID":"6790dc74-39e3-4897-a00d-f38cfc530a92","Type":"ContainerDied","Data":"08ae722f5971746558e21eff027b0500b7e782b424171fcb2c64ed9b56fe8c25"}
Jul 16 21:25:15.166508 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:15.166179    2530 scope.go:117] "RemoveContainer" containerID="f3fd4d0cfc12e8a507d0e3a2efedaea7802e5bb3ffd26f31a8d42ee8c69f6f26"
Jul 16 21:25:15.166508 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:15.166263    2530 scope.go:117] "RemoveContainer" containerID="08ae722f5971746558e21eff027b0500b7e782b424171fcb2c64ed9b56fe8c25"
Jul 16 21:25:15.166508 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:25:15.166468    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv_odh-ai-gateway-infra(6790dc74-39e3-4897-a00d-f38cfc530a92)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv" podUID="6790dc74-39e3-4897-a00d-f38cfc530a92"
Jul 16 21:25:16.173563 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:16.173535    2530 scope.go:117] "RemoveContainer" containerID="08ae722f5971746558e21eff027b0500b7e782b424171fcb2c64ed9b56fe8c25"
Jul 16 21:25:16.173968 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:25:16.173707    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv_odh-ai-gateway-infra(6790dc74-39e3-4897-a00d-f38cfc530a92)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv" podUID="6790dc74-39e3-4897-a00d-f38cfc530a92"
Jul 16 21:25:19.502685 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:19.502656    2530 scope.go:117] "RemoveContainer" containerID="a67598cb7698e23937703ae041b881a081ac71599149a65ad987e3cbf8af9307"
Jul 16 21:25:20.198151 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:20.198052    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9" event={"ID":"4fe65226-301f-4d1a-8647-7c763a2b438a","Type":"ContainerStarted","Data":"25538755662a790342d159076d3041f961d89b2b5c8bf6eddefd9177cb3e3ada"}
Jul 16 21:25:20.214172 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:20.214126    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9" podStartSLOduration=18.21409221 podStartE2EDuration="18.21409221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:25:20.212800347 +0000 UTC m=+3158.234388112" watchObservedRunningTime="2026-07-16 21:25:20.21409221 +0000 UTC m=+3158.235679976"
Jul 16 21:25:21.204948 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:21.204918    2530 generic.go:356] "Generic (PLEG): container finished" podID="4fe65226-301f-4d1a-8647-7c763a2b438a" containerID="25538755662a790342d159076d3041f961d89b2b5c8bf6eddefd9177cb3e3ada" exitCode=6
Jul 16 21:25:21.205375 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:21.204993    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9" event={"ID":"4fe65226-301f-4d1a-8647-7c763a2b438a","Type":"ContainerDied","Data":"25538755662a790342d159076d3041f961d89b2b5c8bf6eddefd9177cb3e3ada"}
Jul 16 21:25:21.205375 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:21.205051    2530 scope.go:117] "RemoveContainer" containerID="a67598cb7698e23937703ae041b881a081ac71599149a65ad987e3cbf8af9307"
Jul 16 21:25:21.205375 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:21.205318    2530 scope.go:117] "RemoveContainer" containerID="25538755662a790342d159076d3041f961d89b2b5c8bf6eddefd9177cb3e3ada"
Jul 16 21:25:21.205565 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:25:21.205546    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9_odh-ai-gateway-infra(4fe65226-301f-4d1a-8647-7c763a2b438a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9" podUID="4fe65226-301f-4d1a-8647-7c763a2b438a"
Jul 16 21:25:21.230617 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:21.230596    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9"]
Jul 16 21:25:22.344183 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:22.344153    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9"
Jul 16 21:25:22.454649 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:22.454612    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe65226-301f-4d1a-8647-7c763a2b438a-maas-api-service-ca\") pod \"4fe65226-301f-4d1a-8647-7c763a2b438a\" (UID: \"4fe65226-301f-4d1a-8647-7c763a2b438a\") "
Jul 16 21:25:22.454925 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:22.454904    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe65226-301f-4d1a-8647-7c763a2b438a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4fe65226-301f-4d1a-8647-7c763a2b438a" (UID: "4fe65226-301f-4d1a-8647-7c763a2b438a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:25:22.555491 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:22.555464    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fe65226-301f-4d1a-8647-7c763a2b438a-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:25:23.217068 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:23.217035    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9"
Jul 16 21:25:23.217265 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:23.217074    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9" event={"ID":"4fe65226-301f-4d1a-8647-7c763a2b438a","Type":"ContainerDied","Data":"f8183945511c6c08533964e1d826201bf908fbc07cb0ffaa3d3f1514e7ec22ba"}
Jul 16 21:25:23.217265 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:23.217169    2530 scope.go:117] "RemoveContainer" containerID="25538755662a790342d159076d3041f961d89b2b5c8bf6eddefd9177cb3e3ada"
Jul 16 21:25:23.234660 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:23.234639    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9"]
Jul 16 21:25:23.239012 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:23.238992    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-ns4m9"]
Jul 16 21:25:24.438013 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:24.437976    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85"]
Jul 16 21:25:24.438395 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:24.438324    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe65226-301f-4d1a-8647-7c763a2b438a" containerName="revoke-keys"
Jul 16 21:25:24.438395 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:24.438335    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe65226-301f-4d1a-8647-7c763a2b438a" containerName="revoke-keys"
Jul 16 21:25:24.438395 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:24.438356    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe65226-301f-4d1a-8647-7c763a2b438a" containerName="revoke-keys"
Jul 16 21:25:24.438395 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:24.438361    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe65226-301f-4d1a-8647-7c763a2b438a" containerName="revoke-keys"
Jul 16 21:25:24.438542 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:24.438417    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fe65226-301f-4d1a-8647-7c763a2b438a" containerName="revoke-keys"
Jul 16 21:25:24.438542 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:24.438426    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fe65226-301f-4d1a-8647-7c763a2b438a" containerName="revoke-keys"
Jul 16 21:25:24.443830 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:24.443811    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85"
Jul 16 21:25:24.448410 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:24.448381    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85"]
Jul 16 21:25:24.508262 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:24.508231    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fe65226-301f-4d1a-8647-7c763a2b438a" path="/var/lib/kubelet/pods/4fe65226-301f-4d1a-8647-7c763a2b438a/volumes"
Jul 16 21:25:24.571462 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:24.571431    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b3241b-5ac4-488b-9337-a2d6a4e565c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85\" (UID: \"81b3241b-5ac4-488b-9337-a2d6a4e565c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85"
Jul 16 21:25:24.672818 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:24.672791    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b3241b-5ac4-488b-9337-a2d6a4e565c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85\" (UID: \"81b3241b-5ac4-488b-9337-a2d6a4e565c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85"
Jul 16 21:25:24.673414 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:24.673396    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b3241b-5ac4-488b-9337-a2d6a4e565c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85\" (UID: \"81b3241b-5ac4-488b-9337-a2d6a4e565c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85"
Jul 16 21:25:24.755596 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:24.755568    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85"
Jul 16 21:25:24.886081 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:24.885912    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85"]
Jul 16 21:25:24.888379 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:25:24.888351    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b3241b_5ac4_488b_9337_a2d6a4e565c1.slice/crio-96d2cc6166730196190b18bf6251d373c02f00dd1dbdb500d29fc65f6be1289f WatchSource:0}: Error finding container 96d2cc6166730196190b18bf6251d373c02f00dd1dbdb500d29fc65f6be1289f: Status 404 returned error can't find the container with id 96d2cc6166730196190b18bf6251d373c02f00dd1dbdb500d29fc65f6be1289f
Jul 16 21:25:25.230465 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:25.230436    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85" event={"ID":"81b3241b-5ac4-488b-9337-a2d6a4e565c1","Type":"ContainerStarted","Data":"96d2cc6166730196190b18bf6251d373c02f00dd1dbdb500d29fc65f6be1289f"}
Jul 16 21:25:26.236487 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:26.236448    2530 generic.go:356] "Generic (PLEG): container finished" podID="81b3241b-5ac4-488b-9337-a2d6a4e565c1" containerID="c10e61f99d0a4ba727af1b167e5ba0117b0466f3ecd4ae45e40a62430c44b1c9" exitCode=6
Jul 16 21:25:26.236869 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:26.236534    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85" event={"ID":"81b3241b-5ac4-488b-9337-a2d6a4e565c1","Type":"ContainerDied","Data":"c10e61f99d0a4ba727af1b167e5ba0117b0466f3ecd4ae45e40a62430c44b1c9"}
Jul 16 21:25:26.236869 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:26.236680    2530 scope.go:117] "RemoveContainer" containerID="c10e61f99d0a4ba727af1b167e5ba0117b0466f3ecd4ae45e40a62430c44b1c9"
Jul 16 21:25:27.242600 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:27.242563    2530 generic.go:356] "Generic (PLEG): container finished" podID="81b3241b-5ac4-488b-9337-a2d6a4e565c1" containerID="8dbdd9fd27298e20223b1656fef400d3071a49daad6acce710fe28f9a6490bc8" exitCode=6
Jul 16 21:25:27.243045 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:27.242656    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85" event={"ID":"81b3241b-5ac4-488b-9337-a2d6a4e565c1","Type":"ContainerDied","Data":"8dbdd9fd27298e20223b1656fef400d3071a49daad6acce710fe28f9a6490bc8"}
Jul 16 21:25:27.243045 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:27.242712    2530 scope.go:117] "RemoveContainer" containerID="c10e61f99d0a4ba727af1b167e5ba0117b0466f3ecd4ae45e40a62430c44b1c9"
Jul 16 21:25:27.243045 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:27.242861    2530 scope.go:117] "RemoveContainer" containerID="8dbdd9fd27298e20223b1656fef400d3071a49daad6acce710fe28f9a6490bc8"
Jul 16 21:25:27.243244 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:25:27.243074    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85_odh-ai-gateway-infra(81b3241b-5ac4-488b-9337-a2d6a4e565c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85" podUID="81b3241b-5ac4-488b-9337-a2d6a4e565c1"
Jul 16 21:25:28.249196 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:28.249163    2530 scope.go:117] "RemoveContainer" containerID="8dbdd9fd27298e20223b1656fef400d3071a49daad6acce710fe28f9a6490bc8"
Jul 16 21:25:28.249540 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:25:28.249347    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85_odh-ai-gateway-infra(81b3241b-5ac4-488b-9337-a2d6a4e565c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85" podUID="81b3241b-5ac4-488b-9337-a2d6a4e565c1"
Jul 16 21:25:29.696052 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:29.696019    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz"]
Jul 16 21:25:29.696583 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:29.696563    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe65226-301f-4d1a-8647-7c763a2b438a" containerName="revoke-keys"
Jul 16 21:25:29.696667 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:29.696586    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe65226-301f-4d1a-8647-7c763a2b438a" containerName="revoke-keys"
Jul 16 21:25:29.696717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:29.696684    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fe65226-301f-4d1a-8647-7c763a2b438a" containerName="revoke-keys"
Jul 16 21:25:29.701151 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:29.701130    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz"
Jul 16 21:25:29.707252 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:29.707229    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz"]
Jul 16 21:25:29.817985 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:29.817955    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60ab0a4-6fdb-466c-a027-828c25250ef5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz\" (UID: \"b60ab0a4-6fdb-466c-a027-828c25250ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz"
Jul 16 21:25:29.918538 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:29.918506    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60ab0a4-6fdb-466c-a027-828c25250ef5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz\" (UID: \"b60ab0a4-6fdb-466c-a027-828c25250ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz"
Jul 16 21:25:29.919119 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:29.919068    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60ab0a4-6fdb-466c-a027-828c25250ef5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz\" (UID: \"b60ab0a4-6fdb-466c-a027-828c25250ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz"
Jul 16 21:25:30.012717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:30.012697    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz"
Jul 16 21:25:30.145395 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:30.145371    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz"]
Jul 16 21:25:30.147326 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:25:30.147296    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60ab0a4_6fdb_466c_a027_828c25250ef5.slice/crio-ff0b0355b9f951f1cae43ba1db67ea05d23aaa7cb5429e4476707e4ae9b5a0f3 WatchSource:0}: Error finding container ff0b0355b9f951f1cae43ba1db67ea05d23aaa7cb5429e4476707e4ae9b5a0f3: Status 404 returned error can't find the container with id ff0b0355b9f951f1cae43ba1db67ea05d23aaa7cb5429e4476707e4ae9b5a0f3
Jul 16 21:25:30.259322 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:30.259291    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz" event={"ID":"b60ab0a4-6fdb-466c-a027-828c25250ef5","Type":"ContainerStarted","Data":"ff0b0355b9f951f1cae43ba1db67ea05d23aaa7cb5429e4476707e4ae9b5a0f3"}
Jul 16 21:25:30.502821 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:30.502795    2530 scope.go:117] "RemoveContainer" containerID="08ae722f5971746558e21eff027b0500b7e782b424171fcb2c64ed9b56fe8c25"
Jul 16 21:25:31.265571 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:31.265537    2530 generic.go:356] "Generic (PLEG): container finished" podID="6790dc74-39e3-4897-a00d-f38cfc530a92" containerID="cd6d2b8f70479938e4a0cc4e74d516ac0fa305669007f289d5ea81d5b1ed53d0" exitCode=6
Jul 16 21:25:31.265929 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:31.265608    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv" event={"ID":"6790dc74-39e3-4897-a00d-f38cfc530a92","Type":"ContainerDied","Data":"cd6d2b8f70479938e4a0cc4e74d516ac0fa305669007f289d5ea81d5b1ed53d0"}
Jul 16 21:25:31.265929 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:31.265668    2530 scope.go:117] "RemoveContainer" containerID="08ae722f5971746558e21eff027b0500b7e782b424171fcb2c64ed9b56fe8c25"
Jul 16 21:25:31.265929 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:31.265912    2530 scope.go:117] "RemoveContainer" containerID="cd6d2b8f70479938e4a0cc4e74d516ac0fa305669007f289d5ea81d5b1ed53d0"
Jul 16 21:25:31.266157 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:25:31.266133    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv_odh-ai-gateway-infra(6790dc74-39e3-4897-a00d-f38cfc530a92)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv" podUID="6790dc74-39e3-4897-a00d-f38cfc530a92"
Jul 16 21:25:31.267373 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:31.267351    2530 generic.go:356] "Generic (PLEG): container finished" podID="b60ab0a4-6fdb-466c-a027-828c25250ef5" containerID="6308f47ed96f92d3eaa076052675a6ce60b7fdbe8e0b6bfe5ee39ee58fd25e99" exitCode=6
Jul 16 21:25:31.267459 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:31.267404    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz" event={"ID":"b60ab0a4-6fdb-466c-a027-828c25250ef5","Type":"ContainerDied","Data":"6308f47ed96f92d3eaa076052675a6ce60b7fdbe8e0b6bfe5ee39ee58fd25e99"}
Jul 16 21:25:31.267574 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:31.267563    2530 scope.go:117] "RemoveContainer" containerID="6308f47ed96f92d3eaa076052675a6ce60b7fdbe8e0b6bfe5ee39ee58fd25e99"
Jul 16 21:25:32.274232 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.274201    2530 generic.go:356] "Generic (PLEG): container finished" podID="b60ab0a4-6fdb-466c-a027-828c25250ef5" containerID="eb1988ec793f991c193802f2d18051cc5ce845aa8ccc8eedf1e309f1e8218523" exitCode=6
Jul 16 21:25:32.274613 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.274298    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz" event={"ID":"b60ab0a4-6fdb-466c-a027-828c25250ef5","Type":"ContainerDied","Data":"eb1988ec793f991c193802f2d18051cc5ce845aa8ccc8eedf1e309f1e8218523"}
Jul 16 21:25:32.274613 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.274356    2530 scope.go:117] "RemoveContainer" containerID="6308f47ed96f92d3eaa076052675a6ce60b7fdbe8e0b6bfe5ee39ee58fd25e99"
Jul 16 21:25:32.274613 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.274470    2530 scope.go:117] "RemoveContainer" containerID="eb1988ec793f991c193802f2d18051cc5ce845aa8ccc8eedf1e309f1e8218523"
Jul 16 21:25:32.274787 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:25:32.274666    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz_odh-ai-gateway-infra(b60ab0a4-6fdb-466c-a027-828c25250ef5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz" podUID="b60ab0a4-6fdb-466c-a027-828c25250ef5"
Jul 16 21:25:32.303786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.303763    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv"]
Jul 16 21:25:32.441541 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.441520    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv"
Jul 16 21:25:32.539745 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.539681    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6790dc74-39e3-4897-a00d-f38cfc530a92-maas-api-service-ca\") pod \"6790dc74-39e3-4897-a00d-f38cfc530a92\" (UID: \"6790dc74-39e3-4897-a00d-f38cfc530a92\") "
Jul 16 21:25:32.540002 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.539981    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6790dc74-39e3-4897-a00d-f38cfc530a92-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6790dc74-39e3-4897-a00d-f38cfc530a92" (UID: "6790dc74-39e3-4897-a00d-f38cfc530a92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:25:32.641055 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.641030    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6790dc74-39e3-4897-a00d-f38cfc530a92-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:25:32.983134 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.983078    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"]
Jul 16 21:25:32.983657 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.983637    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6790dc74-39e3-4897-a00d-f38cfc530a92" containerName="revoke-keys"
Jul 16 21:25:32.983730 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.983664    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="6790dc74-39e3-4897-a00d-f38cfc530a92" containerName="revoke-keys"
Jul 16 21:25:32.983730 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.983716    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6790dc74-39e3-4897-a00d-f38cfc530a92" containerName="revoke-keys"
Jul 16 21:25:32.983730 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.983725    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="6790dc74-39e3-4897-a00d-f38cfc530a92" containerName="revoke-keys"
Jul 16 21:25:32.983895 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.983806    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="6790dc74-39e3-4897-a00d-f38cfc530a92" containerName="revoke-keys"
Jul 16 21:25:32.983895 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.983817    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="6790dc74-39e3-4897-a00d-f38cfc530a92" containerName="revoke-keys"
Jul 16 21:25:32.986915 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.986894    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:32.990259 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.990235    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9331e3\"/\"openshift-service-ca.crt\""
Jul 16 21:25:32.990259 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.990253    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9331e3\"/\"sub-test-model-9331e3-kserve-self-signed-certs\""
Jul 16 21:25:32.990432 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.990240    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9331e3\"/\"default-dockercfg-b5mjn\""
Jul 16 21:25:32.990432 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.990253    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9331e3\"/\"kube-root-ca.crt\""
Jul 16 21:25:32.994124 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:32.994076    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"]
Jul 16 21:25:33.044063 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.044039    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-home\") pod \"sub-test-model-9331e3-kserve-84c4d9948c-5flfs\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") " pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.044188 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.044071    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dmvr\" (UniqueName: \"kubernetes.io/projected/43866392-5c48-42e3-bec1-55ee518ff72b-kube-api-access-4dmvr\") pod \"sub-test-model-9331e3-kserve-84c4d9948c-5flfs\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") " pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.044188 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.044128    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-model-cache\") pod \"sub-test-model-9331e3-kserve-84c4d9948c-5flfs\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") " pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.044188 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.044151    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43866392-5c48-42e3-bec1-55ee518ff72b-tls-certs\") pod \"sub-test-model-9331e3-kserve-84c4d9948c-5flfs\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") " pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.044307 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.044228    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-tmp-dir\") pod \"sub-test-model-9331e3-kserve-84c4d9948c-5flfs\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") " pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.044307 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.044257    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-dshm\") pod \"sub-test-model-9331e3-kserve-84c4d9948c-5flfs\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") " pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.144991 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.144966    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-dshm\") pod \"sub-test-model-9331e3-kserve-84c4d9948c-5flfs\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") " pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.145136 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.145022    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-home\") pod \"sub-test-model-9331e3-kserve-84c4d9948c-5flfs\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") " pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.145136 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.145054    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dmvr\" (UniqueName: \"kubernetes.io/projected/43866392-5c48-42e3-bec1-55ee518ff72b-kube-api-access-4dmvr\") pod \"sub-test-model-9331e3-kserve-84c4d9948c-5flfs\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") " pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.145136 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.145085    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-model-cache\") pod \"sub-test-model-9331e3-kserve-84c4d9948c-5flfs\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") " pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.145302 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.145140    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43866392-5c48-42e3-bec1-55ee518ff72b-tls-certs\") pod \"sub-test-model-9331e3-kserve-84c4d9948c-5flfs\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") " pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.145302 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.145189    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-tmp-dir\") pod \"sub-test-model-9331e3-kserve-84c4d9948c-5flfs\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") " pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.145516 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.145498    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-model-cache\") pod \"sub-test-model-9331e3-kserve-84c4d9948c-5flfs\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") " pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.145583 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.145561    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-home\") pod \"sub-test-model-9331e3-kserve-84c4d9948c-5flfs\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") " pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.145633 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.145570    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-tmp-dir\") pod \"sub-test-model-9331e3-kserve-84c4d9948c-5flfs\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") " pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.147485 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.147465    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-dshm\") pod \"sub-test-model-9331e3-kserve-84c4d9948c-5flfs\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") " pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.147753 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.147735    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43866392-5c48-42e3-bec1-55ee518ff72b-tls-certs\") pod \"sub-test-model-9331e3-kserve-84c4d9948c-5flfs\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") " pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.156769 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.156740    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dmvr\" (UniqueName: \"kubernetes.io/projected/43866392-5c48-42e3-bec1-55ee518ff72b-kube-api-access-4dmvr\") pod \"sub-test-model-9331e3-kserve-84c4d9948c-5flfs\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") " pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.283018 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.282948    2530 scope.go:117] "RemoveContainer" containerID="eb1988ec793f991c193802f2d18051cc5ce845aa8ccc8eedf1e309f1e8218523"
Jul 16 21:25:33.283375 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:25:33.283213    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz_odh-ai-gateway-infra(b60ab0a4-6fdb-466c-a027-828c25250ef5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz" podUID="b60ab0a4-6fdb-466c-a027-828c25250ef5"
Jul 16 21:25:33.283971 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.283951    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv" event={"ID":"6790dc74-39e3-4897-a00d-f38cfc530a92","Type":"ContainerDied","Data":"733e9f8fa92a672e1196872b6b01e190e644903d409964cb36ee44e29b45312f"}
Jul 16 21:25:33.284044 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.283996    2530 scope.go:117] "RemoveContainer" containerID="cd6d2b8f70479938e4a0cc4e74d516ac0fa305669007f289d5ea81d5b1ed53d0"
Jul 16 21:25:33.284090 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.283958    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv"
Jul 16 21:25:33.298249 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.298232    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:33.314024 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.314002    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv"]
Jul 16 21:25:33.319764 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.319738    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gsvpv"]
Jul 16 21:25:33.431820 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:33.431797    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"]
Jul 16 21:25:33.433536 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:25:33.433510    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43866392_5c48_42e3_bec1_55ee518ff72b.slice/crio-9e8a86dc8a50d4af7181b3d1fb75383f4b0d538f4c0b1ddbdad138b1bc12eafd WatchSource:0}: Error finding container 9e8a86dc8a50d4af7181b3d1fb75383f4b0d538f4c0b1ddbdad138b1bc12eafd: Status 404 returned error can't find the container with id 9e8a86dc8a50d4af7181b3d1fb75383f4b0d538f4c0b1ddbdad138b1bc12eafd
Jul 16 21:25:34.291238 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:34.291200    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs" event={"ID":"43866392-5c48-42e3-bec1-55ee518ff72b","Type":"ContainerStarted","Data":"d30b6cc9b5003fdd34980e6b0b068bea098529068da3e8c18f123e7403d670dc"}
Jul 16 21:25:34.291238 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:34.291239    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs" event={"ID":"43866392-5c48-42e3-bec1-55ee518ff72b","Type":"ContainerStarted","Data":"9e8a86dc8a50d4af7181b3d1fb75383f4b0d538f4c0b1ddbdad138b1bc12eafd"}
Jul 16 21:25:34.311664 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:34.311618    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs" podStartSLOduration=2.311602787 podStartE2EDuration="2.311602787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:25:34.310714478 +0000 UTC m=+3172.332302246" watchObservedRunningTime="2026-07-16 21:25:34.311602787 +0000 UTC m=+3172.333190556"
Jul 16 21:25:34.507691 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:34.507662    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6790dc74-39e3-4897-a00d-f38cfc530a92" path="/var/lib/kubelet/pods/6790dc74-39e3-4897-a00d-f38cfc530a92/volumes"
Jul 16 21:25:41.502823 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:41.502788    2530 scope.go:117] "RemoveContainer" containerID="8dbdd9fd27298e20223b1656fef400d3071a49daad6acce710fe28f9a6490bc8"
Jul 16 21:25:42.335213 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:42.335179    2530 generic.go:356] "Generic (PLEG): container finished" podID="81b3241b-5ac4-488b-9337-a2d6a4e565c1" containerID="81a2c17e1d51acd54cb0b337039d8830342ffeacc1180272d923b7d3328798de" exitCode=6
Jul 16 21:25:42.335395 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:42.335222    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85" event={"ID":"81b3241b-5ac4-488b-9337-a2d6a4e565c1","Type":"ContainerDied","Data":"81a2c17e1d51acd54cb0b337039d8830342ffeacc1180272d923b7d3328798de"}
Jul 16 21:25:42.335395 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:42.335267    2530 scope.go:117] "RemoveContainer" containerID="8dbdd9fd27298e20223b1656fef400d3071a49daad6acce710fe28f9a6490bc8"
Jul 16 21:25:42.335616 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:42.335595    2530 scope.go:117] "RemoveContainer" containerID="81a2c17e1d51acd54cb0b337039d8830342ffeacc1180272d923b7d3328798de"
Jul 16 21:25:42.335838 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:25:42.335813    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85_odh-ai-gateway-infra(81b3241b-5ac4-488b-9337-a2d6a4e565c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85" podUID="81b3241b-5ac4-488b-9337-a2d6a4e565c1"
Jul 16 21:25:43.298427 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:43.298399    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:43.298427 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:43.298438    2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:43.311248 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:43.311228    2530 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:43.352599 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:43.352576    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:25:43.360986 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:43.360963    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85"]
Jul 16 21:25:43.507853 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:43.507832    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85"
Jul 16 21:25:43.632210 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:43.632133    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b3241b-5ac4-488b-9337-a2d6a4e565c1-maas-api-service-ca\") pod \"81b3241b-5ac4-488b-9337-a2d6a4e565c1\" (UID: \"81b3241b-5ac4-488b-9337-a2d6a4e565c1\") "
Jul 16 21:25:43.632456 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:43.632437    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b3241b-5ac4-488b-9337-a2d6a4e565c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "81b3241b-5ac4-488b-9337-a2d6a4e565c1" (UID: "81b3241b-5ac4-488b-9337-a2d6a4e565c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:25:43.732825 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:43.732803    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81b3241b-5ac4-488b-9337-a2d6a4e565c1-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:25:44.347438 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:44.347409    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85"
Jul 16 21:25:44.347875 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:44.347448    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85" event={"ID":"81b3241b-5ac4-488b-9337-a2d6a4e565c1","Type":"ContainerDied","Data":"96d2cc6166730196190b18bf6251d373c02f00dd1dbdb500d29fc65f6be1289f"}
Jul 16 21:25:44.347875 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:44.347531    2530 scope.go:117] "RemoveContainer" containerID="81a2c17e1d51acd54cb0b337039d8830342ffeacc1180272d923b7d3328798de"
Jul 16 21:25:44.367735 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:44.367710    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85"]
Jul 16 21:25:44.374450 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:44.374426    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-t6v85"]
Jul 16 21:25:44.507439 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:44.507416    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81b3241b-5ac4-488b-9337-a2d6a4e565c1" path="/var/lib/kubelet/pods/81b3241b-5ac4-488b-9337-a2d6a4e565c1/volumes"
Jul 16 21:25:47.502073 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:47.502037    2530 scope.go:117] "RemoveContainer" containerID="eb1988ec793f991c193802f2d18051cc5ce845aa8ccc8eedf1e309f1e8218523"
Jul 16 21:25:48.368324 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:48.368290    2530 generic.go:356] "Generic (PLEG): container finished" podID="b60ab0a4-6fdb-466c-a027-828c25250ef5" containerID="58f6edf7d9bead1e93358e90ac99009034c904b157747f4ce2f5e86f9d84cf14" exitCode=6
Jul 16 21:25:48.368495 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:48.368364    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz" event={"ID":"b60ab0a4-6fdb-466c-a027-828c25250ef5","Type":"ContainerDied","Data":"58f6edf7d9bead1e93358e90ac99009034c904b157747f4ce2f5e86f9d84cf14"}
Jul 16 21:25:48.368495 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:48.368432    2530 scope.go:117] "RemoveContainer" containerID="eb1988ec793f991c193802f2d18051cc5ce845aa8ccc8eedf1e309f1e8218523"
Jul 16 21:25:48.368717 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:48.368700    2530 scope.go:117] "RemoveContainer" containerID="58f6edf7d9bead1e93358e90ac99009034c904b157747f4ce2f5e86f9d84cf14"
Jul 16 21:25:48.368904 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:25:48.368882    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz_odh-ai-gateway-infra(b60ab0a4-6fdb-466c-a027-828c25250ef5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz" podUID="b60ab0a4-6fdb-466c-a027-828c25250ef5"
Jul 16 21:25:49.396639 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:49.396606    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz"]
Jul 16 21:25:49.525765 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:49.525740    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz"
Jul 16 21:25:49.680969 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:49.680901    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60ab0a4-6fdb-466c-a027-828c25250ef5-maas-api-service-ca\") pod \"b60ab0a4-6fdb-466c-a027-828c25250ef5\" (UID: \"b60ab0a4-6fdb-466c-a027-828c25250ef5\") "
Jul 16 21:25:49.681246 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:49.681226    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b60ab0a4-6fdb-466c-a027-828c25250ef5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b60ab0a4-6fdb-466c-a027-828c25250ef5" (UID: "b60ab0a4-6fdb-466c-a027-828c25250ef5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:25:49.782329 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:49.782298    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b60ab0a4-6fdb-466c-a027-828c25250ef5-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:25:50.381258 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:50.381219    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz"
Jul 16 21:25:50.381479 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:50.381229    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz" event={"ID":"b60ab0a4-6fdb-466c-a027-828c25250ef5","Type":"ContainerDied","Data":"ff0b0355b9f951f1cae43ba1db67ea05d23aaa7cb5429e4476707e4ae9b5a0f3"}
Jul 16 21:25:50.381479 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:50.381444    2530 scope.go:117] "RemoveContainer" containerID="58f6edf7d9bead1e93358e90ac99009034c904b157747f4ce2f5e86f9d84cf14"
Jul 16 21:25:50.409666 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:50.409636    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz"]
Jul 16 21:25:50.416065 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:50.416036    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-pm8lz"]
Jul 16 21:25:50.508437 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:25:50.508403    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b60ab0a4-6fdb-466c-a027-828c25250ef5" path="/var/lib/kubelet/pods/b60ab0a4-6fdb-466c-a027-828c25250ef5/volumes"
Jul 16 21:26:02.831245 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831208    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr"]
Jul 16 21:26:02.831652 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831534    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b60ab0a4-6fdb-466c-a027-828c25250ef5" containerName="revoke-keys"
Jul 16 21:26:02.831652 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831546    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b60ab0a4-6fdb-466c-a027-828c25250ef5" containerName="revoke-keys"
Jul 16 21:26:02.831652 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831563    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6790dc74-39e3-4897-a00d-f38cfc530a92" containerName="revoke-keys"
Jul 16 21:26:02.831652 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831568    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="6790dc74-39e3-4897-a00d-f38cfc530a92" containerName="revoke-keys"
Jul 16 21:26:02.831652 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831575    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b3241b-5ac4-488b-9337-a2d6a4e565c1" containerName="revoke-keys"
Jul 16 21:26:02.831652 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831581    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b3241b-5ac4-488b-9337-a2d6a4e565c1" containerName="revoke-keys"
Jul 16 21:26:02.831652 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831588    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b3241b-5ac4-488b-9337-a2d6a4e565c1" containerName="revoke-keys"
Jul 16 21:26:02.831652 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831593    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b3241b-5ac4-488b-9337-a2d6a4e565c1" containerName="revoke-keys"
Jul 16 21:26:02.831652 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831602    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b3241b-5ac4-488b-9337-a2d6a4e565c1" containerName="revoke-keys"
Jul 16 21:26:02.831652 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831607    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b3241b-5ac4-488b-9337-a2d6a4e565c1" containerName="revoke-keys"
Jul 16 21:26:02.831652 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831614    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b60ab0a4-6fdb-466c-a027-828c25250ef5" containerName="revoke-keys"
Jul 16 21:26:02.831652 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831619    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b60ab0a4-6fdb-466c-a027-828c25250ef5" containerName="revoke-keys"
Jul 16 21:26:02.832012 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831686    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b3241b-5ac4-488b-9337-a2d6a4e565c1" containerName="revoke-keys"
Jul 16 21:26:02.832012 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831695    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b3241b-5ac4-488b-9337-a2d6a4e565c1" containerName="revoke-keys"
Jul 16 21:26:02.832012 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831700    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b60ab0a4-6fdb-466c-a027-828c25250ef5" containerName="revoke-keys"
Jul 16 21:26:02.832012 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831706    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b3241b-5ac4-488b-9337-a2d6a4e565c1" containerName="revoke-keys"
Jul 16 21:26:02.832012 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831712    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="6790dc74-39e3-4897-a00d-f38cfc530a92" containerName="revoke-keys"
Jul 16 21:26:02.832012 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.831716    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b60ab0a4-6fdb-466c-a027-828c25250ef5" containerName="revoke-keys"
Jul 16 21:26:02.834677 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.834660    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr"
Jul 16 21:26:02.837197 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.837176    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:26:02.844017 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.843993    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr"]
Jul 16 21:26:02.900017 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:02.899985    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96172d37-c0bb-4c82-a466-d5bd5f53aad9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr\" (UID: \"96172d37-c0bb-4c82-a466-d5bd5f53aad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr"
Jul 16 21:26:03.000661 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:03.000622    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96172d37-c0bb-4c82-a466-d5bd5f53aad9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr\" (UID: \"96172d37-c0bb-4c82-a466-d5bd5f53aad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr"
Jul 16 21:26:03.001329 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:03.001305    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96172d37-c0bb-4c82-a466-d5bd5f53aad9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr\" (UID: \"96172d37-c0bb-4c82-a466-d5bd5f53aad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr"
Jul 16 21:26:03.144837 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:03.144746    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr"
Jul 16 21:26:03.282574 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:03.282538    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr"]
Jul 16 21:26:03.283662 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:26:03.283631    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96172d37_c0bb_4c82_a466_d5bd5f53aad9.slice/crio-07787d558ab4045fb50c7a544863937b55e4ce486573c6224c2f14c9cd9e9a73 WatchSource:0}: Error finding container 07787d558ab4045fb50c7a544863937b55e4ce486573c6224c2f14c9cd9e9a73: Status 404 returned error can't find the container with id 07787d558ab4045fb50c7a544863937b55e4ce486573c6224c2f14c9cd9e9a73
Jul 16 21:26:03.444933 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:03.444902    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr" event={"ID":"96172d37-c0bb-4c82-a466-d5bd5f53aad9","Type":"ContainerStarted","Data":"07787d558ab4045fb50c7a544863937b55e4ce486573c6224c2f14c9cd9e9a73"}
Jul 16 21:26:04.450967 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:04.450928    2530 generic.go:356] "Generic (PLEG): container finished" podID="96172d37-c0bb-4c82-a466-d5bd5f53aad9" containerID="ec094d4970ba3a83728f43da043c0649af0d1bc78b1ae24f0e62a22022a00c12" exitCode=6
Jul 16 21:26:04.451376 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:04.451014    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr" event={"ID":"96172d37-c0bb-4c82-a466-d5bd5f53aad9","Type":"ContainerDied","Data":"ec094d4970ba3a83728f43da043c0649af0d1bc78b1ae24f0e62a22022a00c12"}
Jul 16 21:26:04.451376 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:04.451199    2530 scope.go:117] "RemoveContainer" containerID="ec094d4970ba3a83728f43da043c0649af0d1bc78b1ae24f0e62a22022a00c12"
Jul 16 21:26:05.457802 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:05.457771    2530 generic.go:356] "Generic (PLEG): container finished" podID="96172d37-c0bb-4c82-a466-d5bd5f53aad9" containerID="8e06ab02b4f03ef9acd39fce41e6909ab7416f00aeff9a90ce446f63f2bf6d03" exitCode=6
Jul 16 21:26:05.458299 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:05.457864    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr" event={"ID":"96172d37-c0bb-4c82-a466-d5bd5f53aad9","Type":"ContainerDied","Data":"8e06ab02b4f03ef9acd39fce41e6909ab7416f00aeff9a90ce446f63f2bf6d03"}
Jul 16 21:26:05.458299 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:05.457924    2530 scope.go:117] "RemoveContainer" containerID="ec094d4970ba3a83728f43da043c0649af0d1bc78b1ae24f0e62a22022a00c12"
Jul 16 21:26:05.458299 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:05.458074    2530 scope.go:117] "RemoveContainer" containerID="8e06ab02b4f03ef9acd39fce41e6909ab7416f00aeff9a90ce446f63f2bf6d03"
Jul 16 21:26:05.458414 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:26:05.458331    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr_odh-ai-gateway-infra(96172d37-c0bb-4c82-a466-d5bd5f53aad9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr" podUID="96172d37-c0bb-4c82-a466-d5bd5f53aad9"
Jul 16 21:26:06.464784 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:06.464757    2530 scope.go:117] "RemoveContainer" containerID="8e06ab02b4f03ef9acd39fce41e6909ab7416f00aeff9a90ce446f63f2bf6d03"
Jul 16 21:26:06.465190 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:26:06.464967    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr_odh-ai-gateway-infra(96172d37-c0bb-4c82-a466-d5bd5f53aad9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr" podUID="96172d37-c0bb-4c82-a466-d5bd5f53aad9"
Jul 16 21:26:12.334383 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:12.334348    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5"]
Jul 16 21:26:12.334753 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:12.334711    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b60ab0a4-6fdb-466c-a027-828c25250ef5" containerName="revoke-keys"
Jul 16 21:26:12.334753 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:12.334722    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="b60ab0a4-6fdb-466c-a027-828c25250ef5" containerName="revoke-keys"
Jul 16 21:26:12.334830 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:12.334779    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="b60ab0a4-6fdb-466c-a027-828c25250ef5" containerName="revoke-keys"
Jul 16 21:26:12.339159 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:12.339133    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5"
Jul 16 21:26:12.345150 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:12.345125    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5"]
Jul 16 21:26:12.379419 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:12.379392    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e6df58-9578-494f-bbcc-b1b6091f8b6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-x78f5\" (UID: \"f0e6df58-9578-494f-bbcc-b1b6091f8b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5"
Jul 16 21:26:12.480368 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:12.480336    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e6df58-9578-494f-bbcc-b1b6091f8b6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-x78f5\" (UID: \"f0e6df58-9578-494f-bbcc-b1b6091f8b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5"
Jul 16 21:26:12.480946 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:12.480925    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e6df58-9578-494f-bbcc-b1b6091f8b6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-x78f5\" (UID: \"f0e6df58-9578-494f-bbcc-b1b6091f8b6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5"
Jul 16 21:26:12.651488 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:12.651393    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5"
Jul 16 21:26:12.797758 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:12.797735    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5"]
Jul 16 21:26:12.799859 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:26:12.799834    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e6df58_9578_494f_bbcc_b1b6091f8b6f.slice/crio-a28dc1e2e38c8f82d673c9ffe5fb77c4167b1fa5e4a6d19d5616dded63cd8359 WatchSource:0}: Error finding container a28dc1e2e38c8f82d673c9ffe5fb77c4167b1fa5e4a6d19d5616dded63cd8359: Status 404 returned error can't find the container with id a28dc1e2e38c8f82d673c9ffe5fb77c4167b1fa5e4a6d19d5616dded63cd8359
Jul 16 21:26:13.498524 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:13.498494    2530 generic.go:356] "Generic (PLEG): container finished" podID="f0e6df58-9578-494f-bbcc-b1b6091f8b6f" containerID="6ec3e6a4a15998f2b514c93ce39c9a611ca86c3b8049d3f76e55176f38aff6bf" exitCode=6
Jul 16 21:26:13.498870 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:13.498587    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5" event={"ID":"f0e6df58-9578-494f-bbcc-b1b6091f8b6f","Type":"ContainerDied","Data":"6ec3e6a4a15998f2b514c93ce39c9a611ca86c3b8049d3f76e55176f38aff6bf"}
Jul 16 21:26:13.498870 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:13.498647    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5" event={"ID":"f0e6df58-9578-494f-bbcc-b1b6091f8b6f","Type":"ContainerStarted","Data":"a28dc1e2e38c8f82d673c9ffe5fb77c4167b1fa5e4a6d19d5616dded63cd8359"}
Jul 16 21:26:13.498870 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:13.498759    2530 scope.go:117] "RemoveContainer" containerID="6ec3e6a4a15998f2b514c93ce39c9a611ca86c3b8049d3f76e55176f38aff6bf"
Jul 16 21:26:14.475231 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:14.475195    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8"]
Jul 16 21:26:14.478587 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:14.478570    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8"
Jul 16 21:26:14.485394 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:14.485361    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8"]
Jul 16 21:26:14.505202 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:14.505171    2530 generic.go:356] "Generic (PLEG): container finished" podID="f0e6df58-9578-494f-bbcc-b1b6091f8b6f" containerID="07926e6668e39cad4ed554ab3879f3d6f0b70696e65d2fa6cf95b07f97a28620" exitCode=6
Jul 16 21:26:14.505563 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:14.505480    2530 scope.go:117] "RemoveContainer" containerID="07926e6668e39cad4ed554ab3879f3d6f0b70696e65d2fa6cf95b07f97a28620"
Jul 16 21:26:14.505674 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:26:14.505656    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-x78f5_odh-ai-gateway-infra(f0e6df58-9578-494f-bbcc-b1b6091f8b6f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5" podUID="f0e6df58-9578-494f-bbcc-b1b6091f8b6f"
Jul 16 21:26:14.508524 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:14.508504    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5" event={"ID":"f0e6df58-9578-494f-bbcc-b1b6091f8b6f","Type":"ContainerDied","Data":"07926e6668e39cad4ed554ab3879f3d6f0b70696e65d2fa6cf95b07f97a28620"}
Jul 16 21:26:14.508621 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:14.508554    2530 scope.go:117] "RemoveContainer" containerID="6ec3e6a4a15998f2b514c93ce39c9a611ca86c3b8049d3f76e55176f38aff6bf"
Jul 16 21:26:14.602027 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:14.601998    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c17f775-ce95-48d1-a07e-856cdd3f3c4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8\" (UID: \"1c17f775-ce95-48d1-a07e-856cdd3f3c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8"
Jul 16 21:26:14.703212 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:14.703182    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c17f775-ce95-48d1-a07e-856cdd3f3c4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8\" (UID: \"1c17f775-ce95-48d1-a07e-856cdd3f3c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8"
Jul 16 21:26:14.703768 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:14.703749    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c17f775-ce95-48d1-a07e-856cdd3f3c4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8\" (UID: \"1c17f775-ce95-48d1-a07e-856cdd3f3c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8"
Jul 16 21:26:14.789499 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:14.789431    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8"
Jul 16 21:26:14.923072 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:14.923041    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8"]
Jul 16 21:26:14.926523 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:26:14.926496    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c17f775_ce95_48d1_a07e_856cdd3f3c4c.slice/crio-accdbe39e5c744d42130a342243c9aba2b28e6e12a57530280f590b8b4fce133 WatchSource:0}: Error finding container accdbe39e5c744d42130a342243c9aba2b28e6e12a57530280f590b8b4fce133: Status 404 returned error can't find the container with id accdbe39e5c744d42130a342243c9aba2b28e6e12a57530280f590b8b4fce133
Jul 16 21:26:15.512009 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:15.511972    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8" event={"ID":"1c17f775-ce95-48d1-a07e-856cdd3f3c4c","Type":"ContainerStarted","Data":"da2cb35d0ed48895970a0ab48d039678ee943f5527cea5742698755eff86fc0d"}
Jul 16 21:26:15.512438 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:15.512245    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8" event={"ID":"1c17f775-ce95-48d1-a07e-856cdd3f3c4c","Type":"ContainerStarted","Data":"accdbe39e5c744d42130a342243c9aba2b28e6e12a57530280f590b8b4fce133"}
Jul 16 21:26:15.520256 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:15.520232    2530 scope.go:117] "RemoveContainer" containerID="07926e6668e39cad4ed554ab3879f3d6f0b70696e65d2fa6cf95b07f97a28620"
Jul 16 21:26:15.520426 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:26:15.520402    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-x78f5_odh-ai-gateway-infra(f0e6df58-9578-494f-bbcc-b1b6091f8b6f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5" podUID="f0e6df58-9578-494f-bbcc-b1b6091f8b6f"
Jul 16 21:26:15.532604 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:15.532535    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8" podStartSLOduration=1.53252137 podStartE2EDuration="1.53252137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:26:15.531257529 +0000 UTC m=+3213.552845297" watchObservedRunningTime="2026-07-16 21:26:15.53252137 +0000 UTC m=+3213.554109136"
Jul 16 21:26:16.526730 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:16.526690    2530 generic.go:356] "Generic (PLEG): container finished" podID="1c17f775-ce95-48d1-a07e-856cdd3f3c4c" containerID="da2cb35d0ed48895970a0ab48d039678ee943f5527cea5742698755eff86fc0d" exitCode=6
Jul 16 21:26:16.527125 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:16.526757    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8" event={"ID":"1c17f775-ce95-48d1-a07e-856cdd3f3c4c","Type":"ContainerDied","Data":"da2cb35d0ed48895970a0ab48d039678ee943f5527cea5742698755eff86fc0d"}
Jul 16 21:26:16.527125 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:16.526911    2530 scope.go:117] "RemoveContainer" containerID="da2cb35d0ed48895970a0ab48d039678ee943f5527cea5742698755eff86fc0d"
Jul 16 21:26:17.502423 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:17.502393    2530 scope.go:117] "RemoveContainer" containerID="8e06ab02b4f03ef9acd39fce41e6909ab7416f00aeff9a90ce446f63f2bf6d03"
Jul 16 21:26:17.533954 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:17.533925    2530 generic.go:356] "Generic (PLEG): container finished" podID="1c17f775-ce95-48d1-a07e-856cdd3f3c4c" containerID="0fd5b8bf003f791855117a5a760948ee5eb21d709facf707aea8f834ab196ea1" exitCode=6
Jul 16 21:26:17.534298 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:17.534002    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8" event={"ID":"1c17f775-ce95-48d1-a07e-856cdd3f3c4c","Type":"ContainerDied","Data":"0fd5b8bf003f791855117a5a760948ee5eb21d709facf707aea8f834ab196ea1"}
Jul 16 21:26:17.534298 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:17.534065    2530 scope.go:117] "RemoveContainer" containerID="da2cb35d0ed48895970a0ab48d039678ee943f5527cea5742698755eff86fc0d"
Jul 16 21:26:17.534298 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:17.534258    2530 scope.go:117] "RemoveContainer" containerID="0fd5b8bf003f791855117a5a760948ee5eb21d709facf707aea8f834ab196ea1"
Jul 16 21:26:17.534463 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:26:17.534443    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8_odh-ai-gateway-infra(1c17f775-ce95-48d1-a07e-856cdd3f3c4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8" podUID="1c17f775-ce95-48d1-a07e-856cdd3f3c4c"
Jul 16 21:26:18.540831 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:18.540798    2530 scope.go:117] "RemoveContainer" containerID="0fd5b8bf003f791855117a5a760948ee5eb21d709facf707aea8f834ab196ea1"
Jul 16 21:26:18.541337 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:26:18.541047    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8_odh-ai-gateway-infra(1c17f775-ce95-48d1-a07e-856cdd3f3c4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8" podUID="1c17f775-ce95-48d1-a07e-856cdd3f3c4c"
Jul 16 21:26:18.542380 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:18.542358    2530 generic.go:356] "Generic (PLEG): container finished" podID="96172d37-c0bb-4c82-a466-d5bd5f53aad9" containerID="e6d417193227eddd26b625b9cb67035c160912856f93c0614bf34d7d8c3e8a37" exitCode=6
Jul 16 21:26:18.542480 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:18.542407    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr" event={"ID":"96172d37-c0bb-4c82-a466-d5bd5f53aad9","Type":"ContainerDied","Data":"e6d417193227eddd26b625b9cb67035c160912856f93c0614bf34d7d8c3e8a37"}
Jul 16 21:26:18.542480 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:18.542455    2530 scope.go:117] "RemoveContainer" containerID="8e06ab02b4f03ef9acd39fce41e6909ab7416f00aeff9a90ce446f63f2bf6d03"
Jul 16 21:26:18.542738 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:18.542721    2530 scope.go:117] "RemoveContainer" containerID="e6d417193227eddd26b625b9cb67035c160912856f93c0614bf34d7d8c3e8a37"
Jul 16 21:26:18.542918 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:26:18.542898    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr_odh-ai-gateway-infra(96172d37-c0bb-4c82-a466-d5bd5f53aad9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr" podUID="96172d37-c0bb-4c82-a466-d5bd5f53aad9"
Jul 16 21:26:19.582750 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:19.582712    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr"]
Jul 16 21:26:19.715721 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:19.715698    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr"
Jul 16 21:26:19.853589 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:19.853511    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96172d37-c0bb-4c82-a466-d5bd5f53aad9-maas-api-service-ca\") pod \"96172d37-c0bb-4c82-a466-d5bd5f53aad9\" (UID: \"96172d37-c0bb-4c82-a466-d5bd5f53aad9\") "
Jul 16 21:26:19.853865 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:19.853843    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96172d37-c0bb-4c82-a466-d5bd5f53aad9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96172d37-c0bb-4c82-a466-d5bd5f53aad9" (UID: "96172d37-c0bb-4c82-a466-d5bd5f53aad9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:26:19.954422 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:19.954394    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96172d37-c0bb-4c82-a466-d5bd5f53aad9-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:26:20.554213 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:20.554185    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr"
Jul 16 21:26:20.554347 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:20.554216    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr" event={"ID":"96172d37-c0bb-4c82-a466-d5bd5f53aad9","Type":"ContainerDied","Data":"07787d558ab4045fb50c7a544863937b55e4ce486573c6224c2f14c9cd9e9a73"}
Jul 16 21:26:20.554347 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:20.554273    2530 scope.go:117] "RemoveContainer" containerID="e6d417193227eddd26b625b9cb67035c160912856f93c0614bf34d7d8c3e8a37"
Jul 16 21:26:20.571461 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:20.571433    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr"]
Jul 16 21:26:20.577559 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:20.577538    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-54ctr"]
Jul 16 21:26:22.507757 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:22.507726    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96172d37-c0bb-4c82-a466-d5bd5f53aad9" path="/var/lib/kubelet/pods/96172d37-c0bb-4c82-a466-d5bd5f53aad9/volumes"
Jul 16 21:26:27.502649 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:27.502618    2530 scope.go:117] "RemoveContainer" containerID="07926e6668e39cad4ed554ab3879f3d6f0b70696e65d2fa6cf95b07f97a28620"
Jul 16 21:26:28.594398 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:28.594365    2530 generic.go:356] "Generic (PLEG): container finished" podID="f0e6df58-9578-494f-bbcc-b1b6091f8b6f" containerID="e4da0d5553a1a8bc82d2658b31c27818e348e0c1297d0202d428f8c208ed53b0" exitCode=6
Jul 16 21:26:28.594811 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:28.594426    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5" event={"ID":"f0e6df58-9578-494f-bbcc-b1b6091f8b6f","Type":"ContainerDied","Data":"e4da0d5553a1a8bc82d2658b31c27818e348e0c1297d0202d428f8c208ed53b0"}
Jul 16 21:26:28.594811 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:28.594472    2530 scope.go:117] "RemoveContainer" containerID="07926e6668e39cad4ed554ab3879f3d6f0b70696e65d2fa6cf95b07f97a28620"
Jul 16 21:26:28.594811 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:28.594709    2530 scope.go:117] "RemoveContainer" containerID="e4da0d5553a1a8bc82d2658b31c27818e348e0c1297d0202d428f8c208ed53b0"
Jul 16 21:26:28.594949 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:26:28.594895    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-x78f5_odh-ai-gateway-infra(f0e6df58-9578-494f-bbcc-b1b6091f8b6f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5" podUID="f0e6df58-9578-494f-bbcc-b1b6091f8b6f"
Jul 16 21:26:29.621773 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.621734    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5"]
Jul 16 21:26:29.747947 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.747921    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47"]
Jul 16 21:26:29.748342 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.748315    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96172d37-c0bb-4c82-a466-d5bd5f53aad9" containerName="revoke-keys"
Jul 16 21:26:29.748342 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.748338    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="96172d37-c0bb-4c82-a466-d5bd5f53aad9" containerName="revoke-keys"
Jul 16 21:26:29.748463 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.748360    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96172d37-c0bb-4c82-a466-d5bd5f53aad9" containerName="revoke-keys"
Jul 16 21:26:29.748463 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.748367    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="96172d37-c0bb-4c82-a466-d5bd5f53aad9" containerName="revoke-keys"
Jul 16 21:26:29.748463 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.748381    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96172d37-c0bb-4c82-a466-d5bd5f53aad9" containerName="revoke-keys"
Jul 16 21:26:29.748463 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.748389    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="96172d37-c0bb-4c82-a466-d5bd5f53aad9" containerName="revoke-keys"
Jul 16 21:26:29.748594 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.748485    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="96172d37-c0bb-4c82-a466-d5bd5f53aad9" containerName="revoke-keys"
Jul 16 21:26:29.748594 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.748494    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="96172d37-c0bb-4c82-a466-d5bd5f53aad9" containerName="revoke-keys"
Jul 16 21:26:29.748594 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.748499    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="96172d37-c0bb-4c82-a466-d5bd5f53aad9" containerName="revoke-keys"
Jul 16 21:26:29.753023 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.753006    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47"
Jul 16 21:26:29.753683 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.753664    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5"
Jul 16 21:26:29.760534 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.760510    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47"]
Jul 16 21:26:29.833646 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.833619    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e6df58-9578-494f-bbcc-b1b6091f8b6f-maas-api-service-ca\") pod \"f0e6df58-9578-494f-bbcc-b1b6091f8b6f\" (UID: \"f0e6df58-9578-494f-bbcc-b1b6091f8b6f\") "
Jul 16 21:26:29.833805 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.833785    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59c13b0f-d2e2-4ed7-9526-011a4e6546e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47\" (UID: \"59c13b0f-d2e2-4ed7-9526-011a4e6546e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47"
Jul 16 21:26:29.833956 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.833933    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e6df58-9578-494f-bbcc-b1b6091f8b6f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f0e6df58-9578-494f-bbcc-b1b6091f8b6f" (UID: "f0e6df58-9578-494f-bbcc-b1b6091f8b6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:26:29.934362 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.934298    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59c13b0f-d2e2-4ed7-9526-011a4e6546e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47\" (UID: \"59c13b0f-d2e2-4ed7-9526-011a4e6546e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47"
Jul 16 21:26:29.934476 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.934413    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e6df58-9578-494f-bbcc-b1b6091f8b6f-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:26:29.934875 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:29.934857    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59c13b0f-d2e2-4ed7-9526-011a4e6546e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47\" (UID: \"59c13b0f-d2e2-4ed7-9526-011a4e6546e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47"
Jul 16 21:26:30.064589 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:30.064566    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47"
Jul 16 21:26:30.194085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:30.194062    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47"]
Jul 16 21:26:30.196174 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:26:30.196147    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c13b0f_d2e2_4ed7_9526_011a4e6546e4.slice/crio-4abafa4c9f35bc8b546ee7eb4403be7d14b76e6aca17e69e67080dce30c56c53 WatchSource:0}: Error finding container 4abafa4c9f35bc8b546ee7eb4403be7d14b76e6aca17e69e67080dce30c56c53: Status 404 returned error can't find the container with id 4abafa4c9f35bc8b546ee7eb4403be7d14b76e6aca17e69e67080dce30c56c53
Jul 16 21:26:30.607537 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:30.607500    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47" event={"ID":"59c13b0f-d2e2-4ed7-9526-011a4e6546e4","Type":"ContainerStarted","Data":"76aa501c855a7d519f6eec762f87e48da8bdb94e8bad1cb493da2a57da078c5e"}
Jul 16 21:26:30.607665 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:30.607554    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47" event={"ID":"59c13b0f-d2e2-4ed7-9526-011a4e6546e4","Type":"ContainerStarted","Data":"4abafa4c9f35bc8b546ee7eb4403be7d14b76e6aca17e69e67080dce30c56c53"}
Jul 16 21:26:30.608781 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:30.608764    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5"
Jul 16 21:26:30.608861 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:30.608798    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5" event={"ID":"f0e6df58-9578-494f-bbcc-b1b6091f8b6f","Type":"ContainerDied","Data":"a28dc1e2e38c8f82d673c9ffe5fb77c4167b1fa5e4a6d19d5616dded63cd8359"}
Jul 16 21:26:30.608861 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:30.608841    2530 scope.go:117] "RemoveContainer" containerID="e4da0d5553a1a8bc82d2658b31c27818e348e0c1297d0202d428f8c208ed53b0"
Jul 16 21:26:30.624586 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:30.624548    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47" podStartSLOduration=1.624533618 podStartE2EDuration="1.624533618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:26:30.623325163 +0000 UTC m=+3228.644912929" watchObservedRunningTime="2026-07-16 21:26:30.624533618 +0000 UTC m=+3228.646121383"
Jul 16 21:26:30.640581 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:30.640557    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5"]
Jul 16 21:26:30.644381 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:30.644362    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-x78f5"]
Jul 16 21:26:31.615836 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:31.615795    2530 generic.go:356] "Generic (PLEG): container finished" podID="59c13b0f-d2e2-4ed7-9526-011a4e6546e4" containerID="76aa501c855a7d519f6eec762f87e48da8bdb94e8bad1cb493da2a57da078c5e" exitCode=6
Jul 16 21:26:31.616019 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:31.615881    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47" event={"ID":"59c13b0f-d2e2-4ed7-9526-011a4e6546e4","Type":"ContainerDied","Data":"76aa501c855a7d519f6eec762f87e48da8bdb94e8bad1cb493da2a57da078c5e"}
Jul 16 21:26:31.616061 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:31.616017    2530 scope.go:117] "RemoveContainer" containerID="76aa501c855a7d519f6eec762f87e48da8bdb94e8bad1cb493da2a57da078c5e"
Jul 16 21:26:32.504704 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:32.504622    2530 scope.go:117] "RemoveContainer" containerID="0fd5b8bf003f791855117a5a760948ee5eb21d709facf707aea8f834ab196ea1"
Jul 16 21:26:32.508916 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:32.508895    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0e6df58-9578-494f-bbcc-b1b6091f8b6f" path="/var/lib/kubelet/pods/f0e6df58-9578-494f-bbcc-b1b6091f8b6f/volumes"
Jul 16 21:26:32.622469 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:32.622444    2530 generic.go:356] "Generic (PLEG): container finished" podID="59c13b0f-d2e2-4ed7-9526-011a4e6546e4" containerID="a83d46858243db5cb11fd7157a1db2537f4c248c391df4a513ba67667fca3793" exitCode=6
Jul 16 21:26:32.622567 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:32.622526    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47" event={"ID":"59c13b0f-d2e2-4ed7-9526-011a4e6546e4","Type":"ContainerDied","Data":"a83d46858243db5cb11fd7157a1db2537f4c248c391df4a513ba67667fca3793"}
Jul 16 21:26:32.622613 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:32.622586    2530 scope.go:117] "RemoveContainer" containerID="76aa501c855a7d519f6eec762f87e48da8bdb94e8bad1cb493da2a57da078c5e"
Jul 16 21:26:32.622687 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:32.622671    2530 scope.go:117] "RemoveContainer" containerID="a83d46858243db5cb11fd7157a1db2537f4c248c391df4a513ba67667fca3793"
Jul 16 21:26:32.622883 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:26:32.622860    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47_odh-ai-gateway-infra(59c13b0f-d2e2-4ed7-9526-011a4e6546e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47" podUID="59c13b0f-d2e2-4ed7-9526-011a4e6546e4"
Jul 16 21:26:33.628727 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:33.628693    2530 generic.go:356] "Generic (PLEG): container finished" podID="1c17f775-ce95-48d1-a07e-856cdd3f3c4c" containerID="809a521037f9269b106611dc5b1cca9e3813820f7d804a498f34aa568b98939b" exitCode=6
Jul 16 21:26:33.629185 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:33.628772    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8" event={"ID":"1c17f775-ce95-48d1-a07e-856cdd3f3c4c","Type":"ContainerDied","Data":"809a521037f9269b106611dc5b1cca9e3813820f7d804a498f34aa568b98939b"}
Jul 16 21:26:33.629185 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:33.628837    2530 scope.go:117] "RemoveContainer" containerID="0fd5b8bf003f791855117a5a760948ee5eb21d709facf707aea8f834ab196ea1"
Jul 16 21:26:33.629185 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:33.629080    2530 scope.go:117] "RemoveContainer" containerID="809a521037f9269b106611dc5b1cca9e3813820f7d804a498f34aa568b98939b"
Jul 16 21:26:33.629357 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:26:33.629303    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8_odh-ai-gateway-infra(1c17f775-ce95-48d1-a07e-856cdd3f3c4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8" podUID="1c17f775-ce95-48d1-a07e-856cdd3f3c4c"
Jul 16 21:26:33.630679 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:33.630664    2530 scope.go:117] "RemoveContainer" containerID="a83d46858243db5cb11fd7157a1db2537f4c248c391df4a513ba67667fca3793"
Jul 16 21:26:33.630825 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:26:33.630807    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47_odh-ai-gateway-infra(59c13b0f-d2e2-4ed7-9526-011a4e6546e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47" podUID="59c13b0f-d2e2-4ed7-9526-011a4e6546e4"
Jul 16 21:26:34.662277 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:34.662250    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8"]
Jul 16 21:26:34.793758 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:34.793735    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8"
Jul 16 21:26:34.883151 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:34.883120    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c17f775-ce95-48d1-a07e-856cdd3f3c4c-maas-api-service-ca\") pod \"1c17f775-ce95-48d1-a07e-856cdd3f3c4c\" (UID: \"1c17f775-ce95-48d1-a07e-856cdd3f3c4c\") "
Jul 16 21:26:34.883434 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:34.883415    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c17f775-ce95-48d1-a07e-856cdd3f3c4c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c17f775-ce95-48d1-a07e-856cdd3f3c4c" (UID: "1c17f775-ce95-48d1-a07e-856cdd3f3c4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:26:34.984414 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:34.984387    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c17f775-ce95-48d1-a07e-856cdd3f3c4c-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:26:35.642579 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:35.642546    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8"
Jul 16 21:26:35.642579 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:35.642557    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8" event={"ID":"1c17f775-ce95-48d1-a07e-856cdd3f3c4c","Type":"ContainerDied","Data":"accdbe39e5c744d42130a342243c9aba2b28e6e12a57530280f590b8b4fce133"}
Jul 16 21:26:35.642817 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:35.642617    2530 scope.go:117] "RemoveContainer" containerID="809a521037f9269b106611dc5b1cca9e3813820f7d804a498f34aa568b98939b"
Jul 16 21:26:35.671543 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:35.671512    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8"]
Jul 16 21:26:35.679742 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:35.679723    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-dzzg8"]
Jul 16 21:26:36.507284 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:36.507254    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c17f775-ce95-48d1-a07e-856cdd3f3c4c" path="/var/lib/kubelet/pods/1c17f775-ce95-48d1-a07e-856cdd3f3c4c/volumes"
Jul 16 21:26:46.502506 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:46.502475    2530 scope.go:117] "RemoveContainer" containerID="a83d46858243db5cb11fd7157a1db2537f4c248c391df4a513ba67667fca3793"
Jul 16 21:26:47.700172 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:47.700141    2530 generic.go:356] "Generic (PLEG): container finished" podID="59c13b0f-d2e2-4ed7-9526-011a4e6546e4" containerID="12346ecc69d8660e147d9f897ca7dbe640e53499eb663ac51fa24f06e3a985fd" exitCode=6
Jul 16 21:26:47.700629 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:47.700221    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47" event={"ID":"59c13b0f-d2e2-4ed7-9526-011a4e6546e4","Type":"ContainerDied","Data":"12346ecc69d8660e147d9f897ca7dbe640e53499eb663ac51fa24f06e3a985fd"}
Jul 16 21:26:47.700629 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:47.700283    2530 scope.go:117] "RemoveContainer" containerID="a83d46858243db5cb11fd7157a1db2537f4c248c391df4a513ba67667fca3793"
Jul 16 21:26:47.700756 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:47.700632    2530 scope.go:117] "RemoveContainer" containerID="12346ecc69d8660e147d9f897ca7dbe640e53499eb663ac51fa24f06e3a985fd"
Jul 16 21:26:47.700855 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:26:47.700830    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47_odh-ai-gateway-infra(59c13b0f-d2e2-4ed7-9526-011a4e6546e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47" podUID="59c13b0f-d2e2-4ed7-9526-011a4e6546e4"
Jul 16 21:26:48.734580 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:48.734548    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47"]
Jul 16 21:26:48.864131 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:48.864091    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47"
Jul 16 21:26:48.894682 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:48.894659    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59c13b0f-d2e2-4ed7-9526-011a4e6546e4-maas-api-service-ca\") pod \"59c13b0f-d2e2-4ed7-9526-011a4e6546e4\" (UID: \"59c13b0f-d2e2-4ed7-9526-011a4e6546e4\") "
Jul 16 21:26:48.894969 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:48.894947    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c13b0f-d2e2-4ed7-9526-011a4e6546e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59c13b0f-d2e2-4ed7-9526-011a4e6546e4" (UID: "59c13b0f-d2e2-4ed7-9526-011a4e6546e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:26:48.996143 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:48.996068    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59c13b0f-d2e2-4ed7-9526-011a4e6546e4-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:26:49.711518 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:49.711495    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47"
Jul 16 21:26:49.711669 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:49.711495    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47" event={"ID":"59c13b0f-d2e2-4ed7-9526-011a4e6546e4","Type":"ContainerDied","Data":"4abafa4c9f35bc8b546ee7eb4403be7d14b76e6aca17e69e67080dce30c56c53"}
Jul 16 21:26:49.711669 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:49.711623    2530 scope.go:117] "RemoveContainer" containerID="12346ecc69d8660e147d9f897ca7dbe640e53499eb663ac51fa24f06e3a985fd"
Jul 16 21:26:49.745056 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:49.745021    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47"]
Jul 16 21:26:49.746969 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:49.746946    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-s2q47"]
Jul 16 21:26:50.508175 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:50.508141    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59c13b0f-d2e2-4ed7-9526-011a4e6546e4" path="/var/lib/kubelet/pods/59c13b0f-d2e2-4ed7-9526-011a4e6546e4/volumes"
Jul 16 21:26:52.875951 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.875920    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg"]
Jul 16 21:26:52.876359 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876289    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e6df58-9578-494f-bbcc-b1b6091f8b6f" containerName="revoke-keys"
Jul 16 21:26:52.876359 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876301    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e6df58-9578-494f-bbcc-b1b6091f8b6f" containerName="revoke-keys"
Jul 16 21:26:52.876359 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876315    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c13b0f-d2e2-4ed7-9526-011a4e6546e4" containerName="revoke-keys"
Jul 16 21:26:52.876359 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876321    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c13b0f-d2e2-4ed7-9526-011a4e6546e4" containerName="revoke-keys"
Jul 16 21:26:52.876359 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876329    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e6df58-9578-494f-bbcc-b1b6091f8b6f" containerName="revoke-keys"
Jul 16 21:26:52.876359 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876336    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e6df58-9578-494f-bbcc-b1b6091f8b6f" containerName="revoke-keys"
Jul 16 21:26:52.876359 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876344    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c17f775-ce95-48d1-a07e-856cdd3f3c4c" containerName="revoke-keys"
Jul 16 21:26:52.876359 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876349    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c17f775-ce95-48d1-a07e-856cdd3f3c4c" containerName="revoke-keys"
Jul 16 21:26:52.876359 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876361    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c13b0f-d2e2-4ed7-9526-011a4e6546e4" containerName="revoke-keys"
Jul 16 21:26:52.876641 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876370    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c13b0f-d2e2-4ed7-9526-011a4e6546e4" containerName="revoke-keys"
Jul 16 21:26:52.876641 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876381    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c17f775-ce95-48d1-a07e-856cdd3f3c4c" containerName="revoke-keys"
Jul 16 21:26:52.876641 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876388    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c17f775-ce95-48d1-a07e-856cdd3f3c4c" containerName="revoke-keys"
Jul 16 21:26:52.876641 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876394    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c17f775-ce95-48d1-a07e-856cdd3f3c4c" containerName="revoke-keys"
Jul 16 21:26:52.876641 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876399    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c17f775-ce95-48d1-a07e-856cdd3f3c4c" containerName="revoke-keys"
Jul 16 21:26:52.876641 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876457    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c17f775-ce95-48d1-a07e-856cdd3f3c4c" containerName="revoke-keys"
Jul 16 21:26:52.876641 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876466    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c13b0f-d2e2-4ed7-9526-011a4e6546e4" containerName="revoke-keys"
Jul 16 21:26:52.876641 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876471    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0e6df58-9578-494f-bbcc-b1b6091f8b6f" containerName="revoke-keys"
Jul 16 21:26:52.876641 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876477    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0e6df58-9578-494f-bbcc-b1b6091f8b6f" containerName="revoke-keys"
Jul 16 21:26:52.876641 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876482    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0e6df58-9578-494f-bbcc-b1b6091f8b6f" containerName="revoke-keys"
Jul 16 21:26:52.876641 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876489    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c13b0f-d2e2-4ed7-9526-011a4e6546e4" containerName="revoke-keys"
Jul 16 21:26:52.876641 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.876496    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c17f775-ce95-48d1-a07e-856cdd3f3c4c" containerName="revoke-keys"
Jul 16 21:26:52.879401 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.879386    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg"
Jul 16 21:26:52.881906 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.881880    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:26:52.887758 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.887737    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg"]
Jul 16 21:26:52.933322 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:52.933297    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12923c49-ca79-44b0-98f2-bd91015bff50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg\" (UID: \"12923c49-ca79-44b0-98f2-bd91015bff50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg"
Jul 16 21:26:53.034014 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:53.033984    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12923c49-ca79-44b0-98f2-bd91015bff50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg\" (UID: \"12923c49-ca79-44b0-98f2-bd91015bff50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg"
Jul 16 21:26:53.034601 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:53.034583    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12923c49-ca79-44b0-98f2-bd91015bff50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg\" (UID: \"12923c49-ca79-44b0-98f2-bd91015bff50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg"
Jul 16 21:26:53.191320 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:53.191256    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg"
Jul 16 21:26:53.323623 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:53.323541    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg"]
Jul 16 21:26:53.326690 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:26:53.326662    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12923c49_ca79_44b0_98f2_bd91015bff50.slice/crio-8a1209f6ef8a9cd4fc23bbfa439f6a76302e97570ca9a43f707951f1eb1f2ff1 WatchSource:0}: Error finding container 8a1209f6ef8a9cd4fc23bbfa439f6a76302e97570ca9a43f707951f1eb1f2ff1: Status 404 returned error can't find the container with id 8a1209f6ef8a9cd4fc23bbfa439f6a76302e97570ca9a43f707951f1eb1f2ff1
Jul 16 21:26:53.733343 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:53.733306    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg" event={"ID":"12923c49-ca79-44b0-98f2-bd91015bff50","Type":"ContainerStarted","Data":"9bb9468b1cb4d5dc88bd8d40134d5a98a04798e751caa6d6128b0842e9baec1c"}
Jul 16 21:26:53.733512 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:53.733365    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg" event={"ID":"12923c49-ca79-44b0-98f2-bd91015bff50","Type":"ContainerStarted","Data":"8a1209f6ef8a9cd4fc23bbfa439f6a76302e97570ca9a43f707951f1eb1f2ff1"}
Jul 16 21:26:53.750894 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:53.750847    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg" podStartSLOduration=1.750831743 podStartE2EDuration="1.750831743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:26:53.749449529 +0000 UTC m=+3251.771037295" watchObservedRunningTime="2026-07-16 21:26:53.750831743 +0000 UTC m=+3251.772419515"
Jul 16 21:26:54.740055 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:54.740013    2530 generic.go:356] "Generic (PLEG): container finished" podID="12923c49-ca79-44b0-98f2-bd91015bff50" containerID="9bb9468b1cb4d5dc88bd8d40134d5a98a04798e751caa6d6128b0842e9baec1c" exitCode=6
Jul 16 21:26:54.740456 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:54.740120    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg" event={"ID":"12923c49-ca79-44b0-98f2-bd91015bff50","Type":"ContainerDied","Data":"9bb9468b1cb4d5dc88bd8d40134d5a98a04798e751caa6d6128b0842e9baec1c"}
Jul 16 21:26:54.740456 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:54.740289    2530 scope.go:117] "RemoveContainer" containerID="9bb9468b1cb4d5dc88bd8d40134d5a98a04798e751caa6d6128b0842e9baec1c"
Jul 16 21:26:55.746867 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:55.746834    2530 generic.go:356] "Generic (PLEG): container finished" podID="12923c49-ca79-44b0-98f2-bd91015bff50" containerID="61c23a8ada4453d7ace27a62cc500792ad77e31b83fbf7ab28d0427442b14b18" exitCode=6
Jul 16 21:26:55.747312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:55.746884    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg" event={"ID":"12923c49-ca79-44b0-98f2-bd91015bff50","Type":"ContainerDied","Data":"61c23a8ada4453d7ace27a62cc500792ad77e31b83fbf7ab28d0427442b14b18"}
Jul 16 21:26:55.747312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:55.746944    2530 scope.go:117] "RemoveContainer" containerID="9bb9468b1cb4d5dc88bd8d40134d5a98a04798e751caa6d6128b0842e9baec1c"
Jul 16 21:26:55.747312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:55.747159    2530 scope.go:117] "RemoveContainer" containerID="61c23a8ada4453d7ace27a62cc500792ad77e31b83fbf7ab28d0427442b14b18"
Jul 16 21:26:55.747444 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:26:55.747369    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg_odh-ai-gateway-infra(12923c49-ca79-44b0-98f2-bd91015bff50)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg" podUID="12923c49-ca79-44b0-98f2-bd91015bff50"
Jul 16 21:26:56.754794 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:26:56.754769    2530 scope.go:117] "RemoveContainer" containerID="61c23a8ada4453d7ace27a62cc500792ad77e31b83fbf7ab28d0427442b14b18"
Jul 16 21:26:56.755192 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:26:56.754918    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg_odh-ai-gateway-infra(12923c49-ca79-44b0-98f2-bd91015bff50)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg" podUID="12923c49-ca79-44b0-98f2-bd91015bff50"
Jul 16 21:27:02.379628 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:02.379586    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm"]
Jul 16 21:27:02.380160 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:02.380142    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c13b0f-d2e2-4ed7-9526-011a4e6546e4" containerName="revoke-keys"
Jul 16 21:27:02.380259 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:02.380163    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c13b0f-d2e2-4ed7-9526-011a4e6546e4" containerName="revoke-keys"
Jul 16 21:27:02.380259 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:02.380194    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e6df58-9578-494f-bbcc-b1b6091f8b6f" containerName="revoke-keys"
Jul 16 21:27:02.380259 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:02.380202    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e6df58-9578-494f-bbcc-b1b6091f8b6f" containerName="revoke-keys"
Jul 16 21:27:02.380417 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:02.380293    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c17f775-ce95-48d1-a07e-856cdd3f3c4c" containerName="revoke-keys"
Jul 16 21:27:02.380417 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:02.380305    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c13b0f-d2e2-4ed7-9526-011a4e6546e4" containerName="revoke-keys"
Jul 16 21:27:02.384607 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:02.384587    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm"
Jul 16 21:27:02.391121 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:02.391078    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm"]
Jul 16 21:27:02.516312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:02.516281    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5aa09c-97f4-437b-93e2-9aa6155b441f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm\" (UID: \"5f5aa09c-97f4-437b-93e2-9aa6155b441f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm"
Jul 16 21:27:02.617565 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:02.617536    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5aa09c-97f4-437b-93e2-9aa6155b441f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm\" (UID: \"5f5aa09c-97f4-437b-93e2-9aa6155b441f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm"
Jul 16 21:27:02.618083 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:02.618066    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5aa09c-97f4-437b-93e2-9aa6155b441f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm\" (UID: \"5f5aa09c-97f4-437b-93e2-9aa6155b441f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm"
Jul 16 21:27:02.696562 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:02.696491    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm"
Jul 16 21:27:02.828867 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:02.828848    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm"]
Jul 16 21:27:02.831124 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:27:02.831082    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5aa09c_97f4_437b_93e2_9aa6155b441f.slice/crio-49fde930c75550b3c693be79e26e9089a3118e6aec21096bf39e253f43fc6916 WatchSource:0}: Error finding container 49fde930c75550b3c693be79e26e9089a3118e6aec21096bf39e253f43fc6916: Status 404 returned error can't find the container with id 49fde930c75550b3c693be79e26e9089a3118e6aec21096bf39e253f43fc6916
Jul 16 21:27:03.105859 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:03.105825    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"]
Jul 16 21:27:03.788354 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:03.788318    2530 generic.go:356] "Generic (PLEG): container finished" podID="5f5aa09c-97f4-437b-93e2-9aa6155b441f" containerID="13f44f34f18790c2ae1f61060088a594772b81663aa324db8096bd7e7f536780" exitCode=6
Jul 16 21:27:03.788731 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:03.788376    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm" event={"ID":"5f5aa09c-97f4-437b-93e2-9aa6155b441f","Type":"ContainerDied","Data":"13f44f34f18790c2ae1f61060088a594772b81663aa324db8096bd7e7f536780"}
Jul 16 21:27:03.788731 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:03.788417    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm" event={"ID":"5f5aa09c-97f4-437b-93e2-9aa6155b441f","Type":"ContainerStarted","Data":"49fde930c75550b3c693be79e26e9089a3118e6aec21096bf39e253f43fc6916"}
Jul 16 21:27:03.788731 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:03.788610    2530 scope.go:117] "RemoveContainer" containerID="13f44f34f18790c2ae1f61060088a594772b81663aa324db8096bd7e7f536780"
Jul 16 21:27:04.795908 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:04.795877    2530 generic.go:356] "Generic (PLEG): container finished" podID="5f5aa09c-97f4-437b-93e2-9aa6155b441f" containerID="ca28375d5e10903e86f3aa4367307cbc874b25aa7a2b6a598a4ada4d083bdd0f" exitCode=6
Jul 16 21:27:04.796342 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:04.795915    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm" event={"ID":"5f5aa09c-97f4-437b-93e2-9aa6155b441f","Type":"ContainerDied","Data":"ca28375d5e10903e86f3aa4367307cbc874b25aa7a2b6a598a4ada4d083bdd0f"}
Jul 16 21:27:04.796342 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:04.795974    2530 scope.go:117] "RemoveContainer" containerID="13f44f34f18790c2ae1f61060088a594772b81663aa324db8096bd7e7f536780"
Jul 16 21:27:04.796342 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:04.796162    2530 scope.go:117] "RemoveContainer" containerID="ca28375d5e10903e86f3aa4367307cbc874b25aa7a2b6a598a4ada4d083bdd0f"
Jul 16 21:27:04.796482 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:27:04.796363    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm_odh-ai-gateway-infra(5f5aa09c-97f4-437b-93e2-9aa6155b441f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm" podUID="5f5aa09c-97f4-437b-93e2-9aa6155b441f"
Jul 16 21:27:05.802351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:05.802326    2530 scope.go:117] "RemoveContainer" containerID="ca28375d5e10903e86f3aa4367307cbc874b25aa7a2b6a598a4ada4d083bdd0f"
Jul 16 21:27:05.802728 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:27:05.802466    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm_odh-ai-gateway-infra(5f5aa09c-97f4-437b-93e2-9aa6155b441f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm" podUID="5f5aa09c-97f4-437b-93e2-9aa6155b441f"
Jul 16 21:27:11.502734 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:11.502698    2530 scope.go:117] "RemoveContainer" containerID="61c23a8ada4453d7ace27a62cc500792ad77e31b83fbf7ab28d0427442b14b18"
Jul 16 21:27:12.837907 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:12.837870    2530 generic.go:356] "Generic (PLEG): container finished" podID="12923c49-ca79-44b0-98f2-bd91015bff50" containerID="6cb1f3fc72ad91c3ed1d35e5a0de4f161f8c89fe5fc0649bc9475b597148b333" exitCode=6
Jul 16 21:27:12.838385 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:12.837944    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg" event={"ID":"12923c49-ca79-44b0-98f2-bd91015bff50","Type":"ContainerDied","Data":"6cb1f3fc72ad91c3ed1d35e5a0de4f161f8c89fe5fc0649bc9475b597148b333"}
Jul 16 21:27:12.838385 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:12.838005    2530 scope.go:117] "RemoveContainer" containerID="61c23a8ada4453d7ace27a62cc500792ad77e31b83fbf7ab28d0427442b14b18"
Jul 16 21:27:12.838385 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:12.838292    2530 scope.go:117] "RemoveContainer" containerID="6cb1f3fc72ad91c3ed1d35e5a0de4f161f8c89fe5fc0649bc9475b597148b333"
Jul 16 21:27:12.838516 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:27:12.838492    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg_odh-ai-gateway-infra(12923c49-ca79-44b0-98f2-bd91015bff50)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg" podUID="12923c49-ca79-44b0-98f2-bd91015bff50"
Jul 16 21:27:13.863475 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:13.863436    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg"]
Jul 16 21:27:14.001742 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.001717    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg"
Jul 16 21:27:14.114094 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.114017    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12923c49-ca79-44b0-98f2-bd91015bff50-maas-api-service-ca\") pod \"12923c49-ca79-44b0-98f2-bd91015bff50\" (UID: \"12923c49-ca79-44b0-98f2-bd91015bff50\") "
Jul 16 21:27:14.114381 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.114359    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12923c49-ca79-44b0-98f2-bd91015bff50-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "12923c49-ca79-44b0-98f2-bd91015bff50" (UID: "12923c49-ca79-44b0-98f2-bd91015bff50"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:27:14.215010 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.214959    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12923c49-ca79-44b0-98f2-bd91015bff50-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:27:14.604175 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.604143    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj"]
Jul 16 21:27:14.604476 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.604464    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12923c49-ca79-44b0-98f2-bd91015bff50" containerName="revoke-keys"
Jul 16 21:27:14.604519 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.604478    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="12923c49-ca79-44b0-98f2-bd91015bff50" containerName="revoke-keys"
Jul 16 21:27:14.604519 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.604486    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12923c49-ca79-44b0-98f2-bd91015bff50" containerName="revoke-keys"
Jul 16 21:27:14.604519 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.604491    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="12923c49-ca79-44b0-98f2-bd91015bff50" containerName="revoke-keys"
Jul 16 21:27:14.604519 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.604502    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12923c49-ca79-44b0-98f2-bd91015bff50" containerName="revoke-keys"
Jul 16 21:27:14.604519 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.604508    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="12923c49-ca79-44b0-98f2-bd91015bff50" containerName="revoke-keys"
Jul 16 21:27:14.604675 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.604556    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="12923c49-ca79-44b0-98f2-bd91015bff50" containerName="revoke-keys"
Jul 16 21:27:14.604675 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.604564    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="12923c49-ca79-44b0-98f2-bd91015bff50" containerName="revoke-keys"
Jul 16 21:27:14.604675 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.604571    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="12923c49-ca79-44b0-98f2-bd91015bff50" containerName="revoke-keys"
Jul 16 21:27:14.607495 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.607480    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj"
Jul 16 21:27:14.616455 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.616435    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj"]
Jul 16 21:27:14.719064 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.719031    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d648b0-964b-4052-a848-6c8c9040534b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj\" (UID: \"08d648b0-964b-4052-a848-6c8c9040534b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj"
Jul 16 21:27:14.819876 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.819800    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d648b0-964b-4052-a848-6c8c9040534b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj\" (UID: \"08d648b0-964b-4052-a848-6c8c9040534b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj"
Jul 16 21:27:14.820390 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.820371    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d648b0-964b-4052-a848-6c8c9040534b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj\" (UID: \"08d648b0-964b-4052-a848-6c8c9040534b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj"
Jul 16 21:27:14.848964 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.848941    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg"
Jul 16 21:27:14.848964 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.848944    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg" event={"ID":"12923c49-ca79-44b0-98f2-bd91015bff50","Type":"ContainerDied","Data":"8a1209f6ef8a9cd4fc23bbfa439f6a76302e97570ca9a43f707951f1eb1f2ff1"}
Jul 16 21:27:14.849165 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.849008    2530 scope.go:117] "RemoveContainer" containerID="6cb1f3fc72ad91c3ed1d35e5a0de4f161f8c89fe5fc0649bc9475b597148b333"
Jul 16 21:27:14.865501 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.865474    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg"]
Jul 16 21:27:14.869650 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.869629    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-m2svg"]
Jul 16 21:27:14.918006 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:14.917987    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj"
Jul 16 21:27:15.048169 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:15.048134    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj"]
Jul 16 21:27:15.051247 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:27:15.051220    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d648b0_964b_4052_a848_6c8c9040534b.slice/crio-d05f7ecba5cbcb24a996ebb1ae925c6bf9b8630ce26da62c6ee2f8aff8895ef2 WatchSource:0}: Error finding container d05f7ecba5cbcb24a996ebb1ae925c6bf9b8630ce26da62c6ee2f8aff8895ef2: Status 404 returned error can't find the container with id d05f7ecba5cbcb24a996ebb1ae925c6bf9b8630ce26da62c6ee2f8aff8895ef2
Jul 16 21:27:15.855961 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:15.855926    2530 generic.go:356] "Generic (PLEG): container finished" podID="08d648b0-964b-4052-a848-6c8c9040534b" containerID="a395b1a61b525a8a4f55cd47f4902b555c46e2ccf2f89743ecce6dffc1f69e10" exitCode=6
Jul 16 21:27:15.856161 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:15.856010    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj" event={"ID":"08d648b0-964b-4052-a848-6c8c9040534b","Type":"ContainerDied","Data":"a395b1a61b525a8a4f55cd47f4902b555c46e2ccf2f89743ecce6dffc1f69e10"}
Jul 16 21:27:15.856161 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:15.856059    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj" event={"ID":"08d648b0-964b-4052-a848-6c8c9040534b","Type":"ContainerStarted","Data":"d05f7ecba5cbcb24a996ebb1ae925c6bf9b8630ce26da62c6ee2f8aff8895ef2"}
Jul 16 21:27:15.856296 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:15.856231    2530 scope.go:117] "RemoveContainer" containerID="a395b1a61b525a8a4f55cd47f4902b555c46e2ccf2f89743ecce6dffc1f69e10"
Jul 16 21:27:16.507362 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:16.507330    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12923c49-ca79-44b0-98f2-bd91015bff50" path="/var/lib/kubelet/pods/12923c49-ca79-44b0-98f2-bd91015bff50/volumes"
Jul 16 21:27:16.508053 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:16.508033    2530 scope.go:117] "RemoveContainer" containerID="ca28375d5e10903e86f3aa4367307cbc874b25aa7a2b6a598a4ada4d083bdd0f"
Jul 16 21:27:16.862208 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:16.862145    2530 generic.go:356] "Generic (PLEG): container finished" podID="08d648b0-964b-4052-a848-6c8c9040534b" containerID="b147c430e0b512a62029b6d456c4723f4d3b6e2242eb8808059aa529986c379c" exitCode=6
Jul 16 21:27:16.862309 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:16.862230    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj" event={"ID":"08d648b0-964b-4052-a848-6c8c9040534b","Type":"ContainerDied","Data":"b147c430e0b512a62029b6d456c4723f4d3b6e2242eb8808059aa529986c379c"}
Jul 16 21:27:16.862309 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:16.862295    2530 scope.go:117] "RemoveContainer" containerID="a395b1a61b525a8a4f55cd47f4902b555c46e2ccf2f89743ecce6dffc1f69e10"
Jul 16 21:27:16.862419 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:16.862405    2530 scope.go:117] "RemoveContainer" containerID="b147c430e0b512a62029b6d456c4723f4d3b6e2242eb8808059aa529986c379c"
Jul 16 21:27:16.862616 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:27:16.862585    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj_odh-ai-gateway-infra(08d648b0-964b-4052-a848-6c8c9040534b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj" podUID="08d648b0-964b-4052-a848-6c8c9040534b"
Jul 16 21:27:16.870746 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:16.870721    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm" event={"ID":"5f5aa09c-97f4-437b-93e2-9aa6155b441f","Type":"ContainerStarted","Data":"c1145757d3e947bd65a1b78e4ce3c5631a274f5ee3b8c351c7908df9383edccc"}
Jul 16 21:27:16.896126 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:16.896053    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm" podStartSLOduration=14.896041213 podStartE2EDuration="14.896041213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:27:16.895605743 +0000 UTC m=+3274.917193510" watchObservedRunningTime="2026-07-16 21:27:16.896041213 +0000 UTC m=+3274.917628978"
Jul 16 21:27:17.877865 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:17.877832    2530 scope.go:117] "RemoveContainer" containerID="b147c430e0b512a62029b6d456c4723f4d3b6e2242eb8808059aa529986c379c"
Jul 16 21:27:17.878307 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:27:17.878026    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj_odh-ai-gateway-infra(08d648b0-964b-4052-a848-6c8c9040534b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj" podUID="08d648b0-964b-4052-a848-6c8c9040534b"
Jul 16 21:27:17.879425 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:17.879401    2530 generic.go:356] "Generic (PLEG): container finished" podID="5f5aa09c-97f4-437b-93e2-9aa6155b441f" containerID="c1145757d3e947bd65a1b78e4ce3c5631a274f5ee3b8c351c7908df9383edccc" exitCode=6
Jul 16 21:27:17.879572 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:17.879475    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm" event={"ID":"5f5aa09c-97f4-437b-93e2-9aa6155b441f","Type":"ContainerDied","Data":"c1145757d3e947bd65a1b78e4ce3c5631a274f5ee3b8c351c7908df9383edccc"}
Jul 16 21:27:17.879572 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:17.879542    2530 scope.go:117] "RemoveContainer" containerID="ca28375d5e10903e86f3aa4367307cbc874b25aa7a2b6a598a4ada4d083bdd0f"
Jul 16 21:27:17.879754 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:17.879741    2530 scope.go:117] "RemoveContainer" containerID="c1145757d3e947bd65a1b78e4ce3c5631a274f5ee3b8c351c7908df9383edccc"
Jul 16 21:27:17.879925 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:27:17.879905    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm_odh-ai-gateway-infra(5f5aa09c-97f4-437b-93e2-9aa6155b441f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm" podUID="5f5aa09c-97f4-437b-93e2-9aa6155b441f"
Jul 16 21:27:17.904379 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:17.904348    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm"]
Jul 16 21:27:18.126169 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.126129    2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs" podUID="43866392-5c48-42e3-bec1-55ee518ff72b" containerName="main" containerID="cri-o://d30b6cc9b5003fdd34980e6b0b068bea098529068da3e8c18f123e7403d670dc" gracePeriod=45
Jul 16 21:27:18.385906 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.385844    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:27:18.549802 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.549771    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43866392-5c48-42e3-bec1-55ee518ff72b-tls-certs\") pod \"43866392-5c48-42e3-bec1-55ee518ff72b\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") "
Jul 16 21:27:18.549951 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.549867    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-home\") pod \"43866392-5c48-42e3-bec1-55ee518ff72b\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") "
Jul 16 21:27:18.549951 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.549887    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-model-cache\") pod \"43866392-5c48-42e3-bec1-55ee518ff72b\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") "
Jul 16 21:27:18.549951 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.549904    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-dshm\") pod \"43866392-5c48-42e3-bec1-55ee518ff72b\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") "
Jul 16 21:27:18.549951 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.549931    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dmvr\" (UniqueName: \"kubernetes.io/projected/43866392-5c48-42e3-bec1-55ee518ff72b-kube-api-access-4dmvr\") pod \"43866392-5c48-42e3-bec1-55ee518ff72b\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") "
Jul 16 21:27:18.549951 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.549952    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-tmp-dir\") pod \"43866392-5c48-42e3-bec1-55ee518ff72b\" (UID: \"43866392-5c48-42e3-bec1-55ee518ff72b\") "
Jul 16 21:27:18.550244 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.550183    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-model-cache" (OuterVolumeSpecName: "model-cache") pod "43866392-5c48-42e3-bec1-55ee518ff72b" (UID: "43866392-5c48-42e3-bec1-55ee518ff72b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:27:18.550244 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.550188    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-home" (OuterVolumeSpecName: "home") pod "43866392-5c48-42e3-bec1-55ee518ff72b" (UID: "43866392-5c48-42e3-bec1-55ee518ff72b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:27:18.550367 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.550341    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43866392-5c48-42e3-bec1-55ee518ff72b" (UID: "43866392-5c48-42e3-bec1-55ee518ff72b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:27:18.552196 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.552169    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43866392-5c48-42e3-bec1-55ee518ff72b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43866392-5c48-42e3-bec1-55ee518ff72b" (UID: "43866392-5c48-42e3-bec1-55ee518ff72b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:27:18.552196 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.552185    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43866392-5c48-42e3-bec1-55ee518ff72b-kube-api-access-4dmvr" (OuterVolumeSpecName: "kube-api-access-4dmvr") pod "43866392-5c48-42e3-bec1-55ee518ff72b" (UID: "43866392-5c48-42e3-bec1-55ee518ff72b"). InnerVolumeSpecName "kube-api-access-4dmvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:27:18.552380 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.552204    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-dshm" (OuterVolumeSpecName: "dshm") pod "43866392-5c48-42e3-bec1-55ee518ff72b" (UID: "43866392-5c48-42e3-bec1-55ee518ff72b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:27:18.651226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.651152    2530 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-home\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:27:18.651226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.651175    2530 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-model-cache\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:27:18.651226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.651185    2530 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-dshm\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:27:18.651226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.651193    2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dmvr\" (UniqueName: \"kubernetes.io/projected/43866392-5c48-42e3-bec1-55ee518ff72b-kube-api-access-4dmvr\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:27:18.651226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.651201    2530 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43866392-5c48-42e3-bec1-55ee518ff72b-tmp-dir\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:27:18.651226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.651210    2530 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43866392-5c48-42e3-bec1-55ee518ff72b-tls-certs\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:27:18.884709 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.884667    2530 generic.go:356] "Generic (PLEG): container finished" podID="43866392-5c48-42e3-bec1-55ee518ff72b" containerID="d30b6cc9b5003fdd34980e6b0b068bea098529068da3e8c18f123e7403d670dc" exitCode=0
Jul 16 21:27:18.884709 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.884709    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"
Jul 16 21:27:18.885232 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.884756    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs" event={"ID":"43866392-5c48-42e3-bec1-55ee518ff72b","Type":"ContainerDied","Data":"d30b6cc9b5003fdd34980e6b0b068bea098529068da3e8c18f123e7403d670dc"}
Jul 16 21:27:18.885232 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.884796    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs" event={"ID":"43866392-5c48-42e3-bec1-55ee518ff72b","Type":"ContainerDied","Data":"9e8a86dc8a50d4af7181b3d1fb75383f4b0d538f4c0b1ddbdad138b1bc12eafd"}
Jul 16 21:27:18.885232 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.884815    2530 scope.go:117] "RemoveContainer" containerID="d30b6cc9b5003fdd34980e6b0b068bea098529068da3e8c18f123e7403d670dc"
Jul 16 21:27:18.894972 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.894947    2530 scope.go:117] "RemoveContainer" containerID="d30b6cc9b5003fdd34980e6b0b068bea098529068da3e8c18f123e7403d670dc"
Jul 16 21:27:18.895234 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:27:18.895213    2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30b6cc9b5003fdd34980e6b0b068bea098529068da3e8c18f123e7403d670dc\": container with ID starting with d30b6cc9b5003fdd34980e6b0b068bea098529068da3e8c18f123e7403d670dc not found: ID does not exist" containerID="d30b6cc9b5003fdd34980e6b0b068bea098529068da3e8c18f123e7403d670dc"
Jul 16 21:27:18.895351 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.895241    2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30b6cc9b5003fdd34980e6b0b068bea098529068da3e8c18f123e7403d670dc"} err="failed to get container status \"d30b6cc9b5003fdd34980e6b0b068bea098529068da3e8c18f123e7403d670dc\": rpc error: code = NotFound desc = could not find container \"d30b6cc9b5003fdd34980e6b0b068bea098529068da3e8c18f123e7403d670dc\": container with ID starting with d30b6cc9b5003fdd34980e6b0b068bea098529068da3e8c18f123e7403d670dc not found: ID does not exist"
Jul 16 21:27:18.914842 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.914777    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"]
Jul 16 21:27:18.919457 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:18.919428    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9331e3/sub-test-model-9331e3-kserve-84c4d9948c-5flfs"]
Jul 16 21:27:19.008261 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:19.008235    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm"
Jul 16 21:27:19.154660 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:19.154629    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5aa09c-97f4-437b-93e2-9aa6155b441f-maas-api-service-ca\") pod \"5f5aa09c-97f4-437b-93e2-9aa6155b441f\" (UID: \"5f5aa09c-97f4-437b-93e2-9aa6155b441f\") "
Jul 16 21:27:19.154961 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:19.154939    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5aa09c-97f4-437b-93e2-9aa6155b441f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5f5aa09c-97f4-437b-93e2-9aa6155b441f" (UID: "5f5aa09c-97f4-437b-93e2-9aa6155b441f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:27:19.256118 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:19.256076    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f5aa09c-97f4-437b-93e2-9aa6155b441f-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:27:19.892881 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:19.892845    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm" event={"ID":"5f5aa09c-97f4-437b-93e2-9aa6155b441f","Type":"ContainerDied","Data":"49fde930c75550b3c693be79e26e9089a3118e6aec21096bf39e253f43fc6916"}
Jul 16 21:27:19.893335 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:19.892910    2530 scope.go:117] "RemoveContainer" containerID="c1145757d3e947bd65a1b78e4ce3c5631a274f5ee3b8c351c7908df9383edccc"
Jul 16 21:27:19.893335 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:19.892854    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm"
Jul 16 21:27:19.916336 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:19.916307    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm"]
Jul 16 21:27:19.920332 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:19.920310    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-w5sfm"]
Jul 16 21:27:20.507114 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:20.507077    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43866392-5c48-42e3-bec1-55ee518ff72b" path="/var/lib/kubelet/pods/43866392-5c48-42e3-bec1-55ee518ff72b/volumes"
Jul 16 21:27:20.507420 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:20.507407    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f5aa09c-97f4-437b-93e2-9aa6155b441f" path="/var/lib/kubelet/pods/5f5aa09c-97f4-437b-93e2-9aa6155b441f/volumes"
Jul 16 21:27:29.502079 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.502049    2530 scope.go:117] "RemoveContainer" containerID="b147c430e0b512a62029b6d456c4723f4d3b6e2242eb8808059aa529986c379c"
Jul 16 21:27:29.805939 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.805911    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g"]
Jul 16 21:27:29.806344 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.806325    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f5aa09c-97f4-437b-93e2-9aa6155b441f" containerName="revoke-keys"
Jul 16 21:27:29.806426 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.806347    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f5aa09c-97f4-437b-93e2-9aa6155b441f" containerName="revoke-keys"
Jul 16 21:27:29.806426 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.806370    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f5aa09c-97f4-437b-93e2-9aa6155b441f" containerName="revoke-keys"
Jul 16 21:27:29.806426 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.806376    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f5aa09c-97f4-437b-93e2-9aa6155b441f" containerName="revoke-keys"
Jul 16 21:27:29.806426 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.806386    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f5aa09c-97f4-437b-93e2-9aa6155b441f" containerName="revoke-keys"
Jul 16 21:27:29.806426 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.806391    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f5aa09c-97f4-437b-93e2-9aa6155b441f" containerName="revoke-keys"
Jul 16 21:27:29.806426 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.806402    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43866392-5c48-42e3-bec1-55ee518ff72b" containerName="main"
Jul 16 21:27:29.806426 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.806407    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="43866392-5c48-42e3-bec1-55ee518ff72b" containerName="main"
Jul 16 21:27:29.806648 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.806465    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="43866392-5c48-42e3-bec1-55ee518ff72b" containerName="main"
Jul 16 21:27:29.806648 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.806475    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f5aa09c-97f4-437b-93e2-9aa6155b441f" containerName="revoke-keys"
Jul 16 21:27:29.806648 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.806482    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f5aa09c-97f4-437b-93e2-9aa6155b441f" containerName="revoke-keys"
Jul 16 21:27:29.809386 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.809369    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g"
Jul 16 21:27:29.820586 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.820532    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g"]
Jul 16 21:27:29.846107 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.846079    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6351cf03-acb7-4eaf-a079-4eecd6fc1c54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g\" (UID: \"6351cf03-acb7-4eaf-a079-4eecd6fc1c54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g"
Jul 16 21:27:29.946785 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.946756    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6351cf03-acb7-4eaf-a079-4eecd6fc1c54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g\" (UID: \"6351cf03-acb7-4eaf-a079-4eecd6fc1c54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g"
Jul 16 21:27:29.947439 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.947416    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6351cf03-acb7-4eaf-a079-4eecd6fc1c54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g\" (UID: \"6351cf03-acb7-4eaf-a079-4eecd6fc1c54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g"
Jul 16 21:27:29.951955 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.951931    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj" event={"ID":"08d648b0-964b-4052-a848-6c8c9040534b","Type":"ContainerStarted","Data":"031447f485ff63c7c517bdb0ae6bd97db6ed000f7caebda08c4d9d4c0963aa40"}
Jul 16 21:27:29.968899 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:29.968857    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj" podStartSLOduration=15.968840726 podStartE2EDuration="15.968840726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:27:29.968533529 +0000 UTC m=+3287.990121297" watchObservedRunningTime="2026-07-16 21:27:29.968840726 +0000 UTC m=+3287.990428494"
Jul 16 21:27:30.119911 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:30.119835    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g"
Jul 16 21:27:30.458027 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:30.457997    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g"]
Jul 16 21:27:30.459815 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:27:30.459787    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6351cf03_acb7_4eaf_a079_4eecd6fc1c54.slice/crio-e089aad3adf78b2393a9e8bb34d7774d00925410b4f46b43257974252d4c24e7 WatchSource:0}: Error finding container e089aad3adf78b2393a9e8bb34d7774d00925410b4f46b43257974252d4c24e7: Status 404 returned error can't find the container with id e089aad3adf78b2393a9e8bb34d7774d00925410b4f46b43257974252d4c24e7
Jul 16 21:27:30.958322 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:30.958286    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g" event={"ID":"6351cf03-acb7-4eaf-a079-4eecd6fc1c54","Type":"ContainerStarted","Data":"1d32dee730128b8f9850c21c1c6645efe7932d724d8bdfff35671ae5cf21d760"}
Jul 16 21:27:30.958744 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:30.958337    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g" event={"ID":"6351cf03-acb7-4eaf-a079-4eecd6fc1c54","Type":"ContainerStarted","Data":"e089aad3adf78b2393a9e8bb34d7774d00925410b4f46b43257974252d4c24e7"}
Jul 16 21:27:30.960092 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:30.960067    2530 generic.go:356] "Generic (PLEG): container finished" podID="08d648b0-964b-4052-a848-6c8c9040534b" containerID="031447f485ff63c7c517bdb0ae6bd97db6ed000f7caebda08c4d9d4c0963aa40" exitCode=6
Jul 16 21:27:30.960219 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:30.960141    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj" event={"ID":"08d648b0-964b-4052-a848-6c8c9040534b","Type":"ContainerDied","Data":"031447f485ff63c7c517bdb0ae6bd97db6ed000f7caebda08c4d9d4c0963aa40"}
Jul 16 21:27:30.960219 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:30.960193    2530 scope.go:117] "RemoveContainer" containerID="b147c430e0b512a62029b6d456c4723f4d3b6e2242eb8808059aa529986c379c"
Jul 16 21:27:30.960416 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:30.960399    2530 scope.go:117] "RemoveContainer" containerID="031447f485ff63c7c517bdb0ae6bd97db6ed000f7caebda08c4d9d4c0963aa40"
Jul 16 21:27:30.960581 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:27:30.960563    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj_odh-ai-gateway-infra(08d648b0-964b-4052-a848-6c8c9040534b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj" podUID="08d648b0-964b-4052-a848-6c8c9040534b"
Jul 16 21:27:30.976457 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:30.976423    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g" podStartSLOduration=1.97641029 podStartE2EDuration="1.97641029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:27:30.974706608 +0000 UTC m=+3288.996294374" watchObservedRunningTime="2026-07-16 21:27:30.97641029 +0000 UTC m=+3288.997998055"
Jul 16 21:27:30.981365 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:30.981345    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj"]
Jul 16 21:27:31.968516 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:31.968480    2530 generic.go:356] "Generic (PLEG): container finished" podID="6351cf03-acb7-4eaf-a079-4eecd6fc1c54" containerID="1d32dee730128b8f9850c21c1c6645efe7932d724d8bdfff35671ae5cf21d760" exitCode=6
Jul 16 21:27:31.968972 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:31.968558    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g" event={"ID":"6351cf03-acb7-4eaf-a079-4eecd6fc1c54","Type":"ContainerDied","Data":"1d32dee730128b8f9850c21c1c6645efe7932d724d8bdfff35671ae5cf21d760"}
Jul 16 21:27:31.968972 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:31.968710    2530 scope.go:117] "RemoveContainer" containerID="1d32dee730128b8f9850c21c1c6645efe7932d724d8bdfff35671ae5cf21d760"
Jul 16 21:27:32.105849 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:32.105827    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj"
Jul 16 21:27:32.161175 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:32.161142    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d648b0-964b-4052-a848-6c8c9040534b-maas-api-service-ca\") pod \"08d648b0-964b-4052-a848-6c8c9040534b\" (UID: \"08d648b0-964b-4052-a848-6c8c9040534b\") "
Jul 16 21:27:32.161482 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:32.161461    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d648b0-964b-4052-a848-6c8c9040534b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08d648b0-964b-4052-a848-6c8c9040534b" (UID: "08d648b0-964b-4052-a848-6c8c9040534b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:27:32.262444 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:32.262416    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d648b0-964b-4052-a848-6c8c9040534b-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:27:32.976540 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:32.976503    2530 generic.go:356] "Generic (PLEG): container finished" podID="6351cf03-acb7-4eaf-a079-4eecd6fc1c54" containerID="1bb4c1116814c0ac38481f158e500310cea0dbd0bf23a2739c889fe1bcbde063" exitCode=6
Jul 16 21:27:32.976911 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:32.976588    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g" event={"ID":"6351cf03-acb7-4eaf-a079-4eecd6fc1c54","Type":"ContainerDied","Data":"1bb4c1116814c0ac38481f158e500310cea0dbd0bf23a2739c889fe1bcbde063"}
Jul 16 21:27:32.976911 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:32.976699    2530 scope.go:117] "RemoveContainer" containerID="1d32dee730128b8f9850c21c1c6645efe7932d724d8bdfff35671ae5cf21d760"
Jul 16 21:27:32.976911 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:32.976748    2530 scope.go:117] "RemoveContainer" containerID="1bb4c1116814c0ac38481f158e500310cea0dbd0bf23a2739c889fe1bcbde063"
Jul 16 21:27:32.977080 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:27:32.976923    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g_odh-ai-gateway-infra(6351cf03-acb7-4eaf-a079-4eecd6fc1c54)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g" podUID="6351cf03-acb7-4eaf-a079-4eecd6fc1c54"
Jul 16 21:27:32.977976 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:32.977945    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj"
Jul 16 21:27:32.977976 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:32.977955    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj" event={"ID":"08d648b0-964b-4052-a848-6c8c9040534b","Type":"ContainerDied","Data":"d05f7ecba5cbcb24a996ebb1ae925c6bf9b8630ce26da62c6ee2f8aff8895ef2"}
Jul 16 21:27:32.987358 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:32.987342    2530 scope.go:117] "RemoveContainer" containerID="031447f485ff63c7c517bdb0ae6bd97db6ed000f7caebda08c4d9d4c0963aa40"
Jul 16 21:27:33.007007 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:33.006987    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj"]
Jul 16 21:27:33.011090 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:33.011070    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-9frmj"]
Jul 16 21:27:33.984670 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:33.984648    2530 scope.go:117] "RemoveContainer" containerID="1bb4c1116814c0ac38481f158e500310cea0dbd0bf23a2739c889fe1bcbde063"
Jul 16 21:27:33.985207 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:27:33.984810    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g_odh-ai-gateway-infra(6351cf03-acb7-4eaf-a079-4eecd6fc1c54)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g" podUID="6351cf03-acb7-4eaf-a079-4eecd6fc1c54"
Jul 16 21:27:34.507379 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:34.507348    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08d648b0-964b-4052-a848-6c8c9040534b" path="/var/lib/kubelet/pods/08d648b0-964b-4052-a848-6c8c9040534b/volumes"
Jul 16 21:27:43.473429 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:43.473395    2530 scope.go:117] "RemoveContainer" containerID="cd917b1ea45ff20aa2ba874e4eec9163c6427c621e7b9f78b74a7c63b7a20aad"
Jul 16 21:27:49.501998 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:49.501966    2530 scope.go:117] "RemoveContainer" containerID="1bb4c1116814c0ac38481f158e500310cea0dbd0bf23a2739c889fe1bcbde063"
Jul 16 21:27:50.062076 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:50.062046    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g" event={"ID":"6351cf03-acb7-4eaf-a079-4eecd6fc1c54","Type":"ContainerStarted","Data":"bc4fb012fc48a6bb8be70319a2201970b21a0b2b01303223c83e88dccf887a0c"}
Jul 16 21:27:51.068114 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:51.068075    2530 generic.go:356] "Generic (PLEG): container finished" podID="6351cf03-acb7-4eaf-a079-4eecd6fc1c54" containerID="bc4fb012fc48a6bb8be70319a2201970b21a0b2b01303223c83e88dccf887a0c" exitCode=6
Jul 16 21:27:51.068522 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:51.068150    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g" event={"ID":"6351cf03-acb7-4eaf-a079-4eecd6fc1c54","Type":"ContainerDied","Data":"bc4fb012fc48a6bb8be70319a2201970b21a0b2b01303223c83e88dccf887a0c"}
Jul 16 21:27:51.068522 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:51.068210    2530 scope.go:117] "RemoveContainer" containerID="1bb4c1116814c0ac38481f158e500310cea0dbd0bf23a2739c889fe1bcbde063"
Jul 16 21:27:51.068522 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:51.068497    2530 scope.go:117] "RemoveContainer" containerID="bc4fb012fc48a6bb8be70319a2201970b21a0b2b01303223c83e88dccf887a0c"
Jul 16 21:27:51.068705 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:27:51.068686    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g_odh-ai-gateway-infra(6351cf03-acb7-4eaf-a079-4eecd6fc1c54)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g" podUID="6351cf03-acb7-4eaf-a079-4eecd6fc1c54"
Jul 16 21:27:51.094403 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:51.094383    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g"]
Jul 16 21:27:52.207792 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.207768    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g"
Jul 16 21:27:52.320527 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.320502    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6351cf03-acb7-4eaf-a079-4eecd6fc1c54-maas-api-service-ca\") pod \"6351cf03-acb7-4eaf-a079-4eecd6fc1c54\" (UID: \"6351cf03-acb7-4eaf-a079-4eecd6fc1c54\") "
Jul 16 21:27:52.320791 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.320768    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6351cf03-acb7-4eaf-a079-4eecd6fc1c54-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6351cf03-acb7-4eaf-a079-4eecd6fc1c54" (UID: "6351cf03-acb7-4eaf-a079-4eecd6fc1c54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:27:52.422030 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.421981    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6351cf03-acb7-4eaf-a079-4eecd6fc1c54-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:27:52.427575 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.427552    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf"]
Jul 16 21:27:52.427915 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.427899    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d648b0-964b-4052-a848-6c8c9040534b" containerName="revoke-keys"
Jul 16 21:27:52.427983 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.427918    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d648b0-964b-4052-a848-6c8c9040534b" containerName="revoke-keys"
Jul 16 21:27:52.427983 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.427937    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d648b0-964b-4052-a848-6c8c9040534b" containerName="revoke-keys"
Jul 16 21:27:52.427983 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.427945    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d648b0-964b-4052-a848-6c8c9040534b" containerName="revoke-keys"
Jul 16 21:27:52.427983 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.427958    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6351cf03-acb7-4eaf-a079-4eecd6fc1c54" containerName="revoke-keys"
Jul 16 21:27:52.427983 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.427966    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="6351cf03-acb7-4eaf-a079-4eecd6fc1c54" containerName="revoke-keys"
Jul 16 21:27:52.428257 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.427983    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6351cf03-acb7-4eaf-a079-4eecd6fc1c54" containerName="revoke-keys"
Jul 16 21:27:52.428257 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.427992    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="6351cf03-acb7-4eaf-a079-4eecd6fc1c54" containerName="revoke-keys"
Jul 16 21:27:52.428257 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.428004    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6351cf03-acb7-4eaf-a079-4eecd6fc1c54" containerName="revoke-keys"
Jul 16 21:27:52.428257 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.428012    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="6351cf03-acb7-4eaf-a079-4eecd6fc1c54" containerName="revoke-keys"
Jul 16 21:27:52.428257 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.428114    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="08d648b0-964b-4052-a848-6c8c9040534b" containerName="revoke-keys"
Jul 16 21:27:52.428257 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.428127    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="08d648b0-964b-4052-a848-6c8c9040534b" containerName="revoke-keys"
Jul 16 21:27:52.428257 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.428142    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="6351cf03-acb7-4eaf-a079-4eecd6fc1c54" containerName="revoke-keys"
Jul 16 21:27:52.428257 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.428150    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="6351cf03-acb7-4eaf-a079-4eecd6fc1c54" containerName="revoke-keys"
Jul 16 21:27:52.428257 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.428159    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="6351cf03-acb7-4eaf-a079-4eecd6fc1c54" containerName="revoke-keys"
Jul 16 21:27:52.428257 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.428168    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f5aa09c-97f4-437b-93e2-9aa6155b441f" containerName="revoke-keys"
Jul 16 21:27:52.431241 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.431223    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf"
Jul 16 21:27:52.438158 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.438134    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf"]
Jul 16 21:27:52.522728 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.522701    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbfc5ee-c06b-4f97-bff3-4daccad99781-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-flgrf\" (UID: \"7fbfc5ee-c06b-4f97-bff3-4daccad99781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf"
Jul 16 21:27:52.623434 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.623409    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbfc5ee-c06b-4f97-bff3-4daccad99781-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-flgrf\" (UID: \"7fbfc5ee-c06b-4f97-bff3-4daccad99781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf"
Jul 16 21:27:52.624013 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.623995    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbfc5ee-c06b-4f97-bff3-4daccad99781-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-flgrf\" (UID: \"7fbfc5ee-c06b-4f97-bff3-4daccad99781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf"
Jul 16 21:27:52.741980 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.741953    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf"
Jul 16 21:27:52.865660 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.865633    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf"]
Jul 16 21:27:52.867195 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:27:52.867161    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbfc5ee_c06b_4f97_bff3_4daccad99781.slice/crio-9ba5a0ccedd16ae5513c8c215aae361bc5f2ccee1582983cb16e3f5b640b2b3b WatchSource:0}: Error finding container 9ba5a0ccedd16ae5513c8c215aae361bc5f2ccee1582983cb16e3f5b640b2b3b: Status 404 returned error can't find the container with id 9ba5a0ccedd16ae5513c8c215aae361bc5f2ccee1582983cb16e3f5b640b2b3b
Jul 16 21:27:52.922986 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.922961    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk"]
Jul 16 21:27:52.923296 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.923284    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d648b0-964b-4052-a848-6c8c9040534b" containerName="revoke-keys"
Jul 16 21:27:52.923344 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.923298    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d648b0-964b-4052-a848-6c8c9040534b" containerName="revoke-keys"
Jul 16 21:27:52.923380 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.923359    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="08d648b0-964b-4052-a848-6c8c9040534b" containerName="revoke-keys"
Jul 16 21:27:52.925927 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.925912    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk"
Jul 16 21:27:52.934134 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:52.934032    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk"]
Jul 16 21:27:53.025599 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:53.025551    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01335f11-a228-4015-b31f-cf670fa1acd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk\" (UID: \"01335f11-a228-4015-b31f-cf670fa1acd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk"
Jul 16 21:27:53.085548 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:53.085531    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g"
Jul 16 21:27:53.085676 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:53.085579    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g" event={"ID":"6351cf03-acb7-4eaf-a079-4eecd6fc1c54","Type":"ContainerDied","Data":"e089aad3adf78b2393a9e8bb34d7774d00925410b4f46b43257974252d4c24e7"}
Jul 16 21:27:53.085676 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:53.085639    2530 scope.go:117] "RemoveContainer" containerID="bc4fb012fc48a6bb8be70319a2201970b21a0b2b01303223c83e88dccf887a0c"
Jul 16 21:27:53.087074 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:53.087042    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf" event={"ID":"7fbfc5ee-c06b-4f97-bff3-4daccad99781","Type":"ContainerStarted","Data":"33b509fb77f3f7a83f47a3817b3df0a2f41dedd5d213effe49bfbc3dec6cc66f"}
Jul 16 21:27:53.087172 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:53.087088    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf" event={"ID":"7fbfc5ee-c06b-4f97-bff3-4daccad99781","Type":"ContainerStarted","Data":"9ba5a0ccedd16ae5513c8c215aae361bc5f2ccee1582983cb16e3f5b640b2b3b"}
Jul 16 21:27:53.111581 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:53.111542    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf" podStartSLOduration=1.111531378 podStartE2EDuration="1.111531378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:27:53.109597256 +0000 UTC m=+3311.131185021" watchObservedRunningTime="2026-07-16 21:27:53.111531378 +0000 UTC m=+3311.133119142"
Jul 16 21:27:53.126214 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:53.126187    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01335f11-a228-4015-b31f-cf670fa1acd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk\" (UID: \"01335f11-a228-4015-b31f-cf670fa1acd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk"
Jul 16 21:27:53.126675 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:53.126658    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01335f11-a228-4015-b31f-cf670fa1acd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk\" (UID: \"01335f11-a228-4015-b31f-cf670fa1acd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk"
Jul 16 21:27:53.126806 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:53.126789    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g"]
Jul 16 21:27:53.130919 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:53.130898    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-r8f6g"]
Jul 16 21:27:53.236211 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:53.236193    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk"
Jul 16 21:27:53.365928 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:53.365902    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk"]
Jul 16 21:27:53.370345 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:27:53.370317    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01335f11_a228_4015_b31f_cf670fa1acd6.slice/crio-a580ac95d7d6b6c4fdee26f252afc7b70256140473b9fa52a61f108f3a817856 WatchSource:0}: Error finding container a580ac95d7d6b6c4fdee26f252afc7b70256140473b9fa52a61f108f3a817856: Status 404 returned error can't find the container with id a580ac95d7d6b6c4fdee26f252afc7b70256140473b9fa52a61f108f3a817856
Jul 16 21:27:54.093324 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:54.093294    2530 generic.go:356] "Generic (PLEG): container finished" podID="01335f11-a228-4015-b31f-cf670fa1acd6" containerID="c2af60592d0c5dfaf07e2f977550dcb7557c7303fbb929f3c6690671e4b7e7fb" exitCode=6
Jul 16 21:27:54.093461 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:54.093383    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk" event={"ID":"01335f11-a228-4015-b31f-cf670fa1acd6","Type":"ContainerDied","Data":"c2af60592d0c5dfaf07e2f977550dcb7557c7303fbb929f3c6690671e4b7e7fb"}
Jul 16 21:27:54.093461 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:54.093450    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk" event={"ID":"01335f11-a228-4015-b31f-cf670fa1acd6","Type":"ContainerStarted","Data":"a580ac95d7d6b6c4fdee26f252afc7b70256140473b9fa52a61f108f3a817856"}
Jul 16 21:27:54.093583 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:54.093530    2530 scope.go:117] "RemoveContainer" containerID="c2af60592d0c5dfaf07e2f977550dcb7557c7303fbb929f3c6690671e4b7e7fb"
Jul 16 21:27:54.095006 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:54.094985    2530 generic.go:356] "Generic (PLEG): container finished" podID="7fbfc5ee-c06b-4f97-bff3-4daccad99781" containerID="33b509fb77f3f7a83f47a3817b3df0a2f41dedd5d213effe49bfbc3dec6cc66f" exitCode=6
Jul 16 21:27:54.095136 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:54.095016    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf" event={"ID":"7fbfc5ee-c06b-4f97-bff3-4daccad99781","Type":"ContainerDied","Data":"33b509fb77f3f7a83f47a3817b3df0a2f41dedd5d213effe49bfbc3dec6cc66f"}
Jul 16 21:27:54.095245 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:54.095231    2530 scope.go:117] "RemoveContainer" containerID="33b509fb77f3f7a83f47a3817b3df0a2f41dedd5d213effe49bfbc3dec6cc66f"
Jul 16 21:27:54.506946 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:54.506919    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6351cf03-acb7-4eaf-a079-4eecd6fc1c54" path="/var/lib/kubelet/pods/6351cf03-acb7-4eaf-a079-4eecd6fc1c54/volumes"
Jul 16 21:27:55.101451 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:55.101382    2530 generic.go:356] "Generic (PLEG): container finished" podID="7fbfc5ee-c06b-4f97-bff3-4daccad99781" containerID="f5cce8b1e06f53d1144b57ca1c1d12bee076adf2a411e1321f6f96ffa4370792" exitCode=6
Jul 16 21:27:55.101594 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:55.101467    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf" event={"ID":"7fbfc5ee-c06b-4f97-bff3-4daccad99781","Type":"ContainerDied","Data":"f5cce8b1e06f53d1144b57ca1c1d12bee076adf2a411e1321f6f96ffa4370792"}
Jul 16 21:27:55.101594 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:55.101521    2530 scope.go:117] "RemoveContainer" containerID="33b509fb77f3f7a83f47a3817b3df0a2f41dedd5d213effe49bfbc3dec6cc66f"
Jul 16 21:27:55.101709 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:55.101681    2530 scope.go:117] "RemoveContainer" containerID="f5cce8b1e06f53d1144b57ca1c1d12bee076adf2a411e1321f6f96ffa4370792"
Jul 16 21:27:55.101879 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:27:55.101853    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-flgrf_odh-ai-gateway-infra(7fbfc5ee-c06b-4f97-bff3-4daccad99781)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf" podUID="7fbfc5ee-c06b-4f97-bff3-4daccad99781"
Jul 16 21:27:55.103512 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:55.103484    2530 generic.go:356] "Generic (PLEG): container finished" podID="01335f11-a228-4015-b31f-cf670fa1acd6" containerID="6ad2f79987420e214433b08808a16a65a212fd90a17aa6f84d30849b405c0284" exitCode=6
Jul 16 21:27:55.103620 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:55.103561    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk" event={"ID":"01335f11-a228-4015-b31f-cf670fa1acd6","Type":"ContainerDied","Data":"6ad2f79987420e214433b08808a16a65a212fd90a17aa6f84d30849b405c0284"}
Jul 16 21:27:55.103686 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:55.103674    2530 scope.go:117] "RemoveContainer" containerID="6ad2f79987420e214433b08808a16a65a212fd90a17aa6f84d30849b405c0284"
Jul 16 21:27:55.103841 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:27:55.103826    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk_odh-ai-gateway-infra(01335f11-a228-4015-b31f-cf670fa1acd6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk" podUID="01335f11-a228-4015-b31f-cf670fa1acd6"
Jul 16 21:27:55.112133 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:55.112115    2530 scope.go:117] "RemoveContainer" containerID="c2af60592d0c5dfaf07e2f977550dcb7557c7303fbb929f3c6690671e4b7e7fb"
Jul 16 21:27:56.110195 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:56.110155    2530 scope.go:117] "RemoveContainer" containerID="6ad2f79987420e214433b08808a16a65a212fd90a17aa6f84d30849b405c0284"
Jul 16 21:27:56.110669 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:27:56.110390    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk_odh-ai-gateway-infra(01335f11-a228-4015-b31f-cf670fa1acd6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk" podUID="01335f11-a228-4015-b31f-cf670fa1acd6"
Jul 16 21:27:56.111682 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:27:56.111667    2530 scope.go:117] "RemoveContainer" containerID="f5cce8b1e06f53d1144b57ca1c1d12bee076adf2a411e1321f6f96ffa4370792"
Jul 16 21:27:56.111814 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:27:56.111799    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-flgrf_odh-ai-gateway-infra(7fbfc5ee-c06b-4f97-bff3-4daccad99781)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf" podUID="7fbfc5ee-c06b-4f97-bff3-4daccad99781"
Jul 16 21:28:04.658489 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:04.658405    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x"]
Jul 16 21:28:04.663073 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:04.663052    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x"
Jul 16 21:28:04.668680 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:04.668635    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x"]
Jul 16 21:28:04.721627 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:04.721599    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0cc5659-1d93-4998-b541-4b3ecc74aee4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x\" (UID: \"a0cc5659-1d93-4998-b541-4b3ecc74aee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x"
Jul 16 21:28:04.822023 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:04.821998    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0cc5659-1d93-4998-b541-4b3ecc74aee4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x\" (UID: \"a0cc5659-1d93-4998-b541-4b3ecc74aee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x"
Jul 16 21:28:04.822578 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:04.822558    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0cc5659-1d93-4998-b541-4b3ecc74aee4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x\" (UID: \"a0cc5659-1d93-4998-b541-4b3ecc74aee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x"
Jul 16 21:28:04.974203 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:04.974161    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x"
Jul 16 21:28:05.102777 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:05.102506    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x"]
Jul 16 21:28:05.106683 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:28:05.106656    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0cc5659_1d93_4998_b541_4b3ecc74aee4.slice/crio-0f1d107c8dc7cc13b0bd88f46086d073152505ee247b0a670c6a1ec06687d253 WatchSource:0}: Error finding container 0f1d107c8dc7cc13b0bd88f46086d073152505ee247b0a670c6a1ec06687d253: Status 404 returned error can't find the container with id 0f1d107c8dc7cc13b0bd88f46086d073152505ee247b0a670c6a1ec06687d253
Jul 16 21:28:05.154077 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:05.154053    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x" event={"ID":"a0cc5659-1d93-4998-b541-4b3ecc74aee4","Type":"ContainerStarted","Data":"0f1d107c8dc7cc13b0bd88f46086d073152505ee247b0a670c6a1ec06687d253"}
Jul 16 21:28:06.160529 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:06.160439    2530 generic.go:356] "Generic (PLEG): container finished" podID="a0cc5659-1d93-4998-b541-4b3ecc74aee4" containerID="bb962aa3a48907924ca85247efba4937f956f8a782f00a3c20c842aa66d91d37" exitCode=6
Jul 16 21:28:06.160900 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:06.160533    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x" event={"ID":"a0cc5659-1d93-4998-b541-4b3ecc74aee4","Type":"ContainerDied","Data":"bb962aa3a48907924ca85247efba4937f956f8a782f00a3c20c842aa66d91d37"}
Jul 16 21:28:06.160900 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:06.160675    2530 scope.go:117] "RemoveContainer" containerID="bb962aa3a48907924ca85247efba4937f956f8a782f00a3c20c842aa66d91d37"
Jul 16 21:28:07.166910 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:07.166811    2530 generic.go:356] "Generic (PLEG): container finished" podID="a0cc5659-1d93-4998-b541-4b3ecc74aee4" containerID="5cc87896ee0cf2498475fccfd4e93a56bf3333a80bb4f8089639a5143f50e49b" exitCode=6
Jul 16 21:28:07.166910 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:07.166895    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x" event={"ID":"a0cc5659-1d93-4998-b541-4b3ecc74aee4","Type":"ContainerDied","Data":"5cc87896ee0cf2498475fccfd4e93a56bf3333a80bb4f8089639a5143f50e49b"}
Jul 16 21:28:07.167440 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:07.166951    2530 scope.go:117] "RemoveContainer" containerID="bb962aa3a48907924ca85247efba4937f956f8a782f00a3c20c842aa66d91d37"
Jul 16 21:28:07.167440 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:07.167164    2530 scope.go:117] "RemoveContainer" containerID="5cc87896ee0cf2498475fccfd4e93a56bf3333a80bb4f8089639a5143f50e49b"
Jul 16 21:28:07.167440 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:28:07.167390    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x_odh-ai-gateway-infra(a0cc5659-1d93-4998-b541-4b3ecc74aee4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x" podUID="a0cc5659-1d93-4998-b541-4b3ecc74aee4"
Jul 16 21:28:07.502183 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:07.502150    2530 scope.go:117] "RemoveContainer" containerID="6ad2f79987420e214433b08808a16a65a212fd90a17aa6f84d30849b405c0284"
Jul 16 21:28:08.172832 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:08.172811    2530 scope.go:117] "RemoveContainer" containerID="5cc87896ee0cf2498475fccfd4e93a56bf3333a80bb4f8089639a5143f50e49b"
Jul 16 21:28:08.173218 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:28:08.173036    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x_odh-ai-gateway-infra(a0cc5659-1d93-4998-b541-4b3ecc74aee4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x" podUID="a0cc5659-1d93-4998-b541-4b3ecc74aee4"
Jul 16 21:28:08.174454 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:08.174431    2530 generic.go:356] "Generic (PLEG): container finished" podID="01335f11-a228-4015-b31f-cf670fa1acd6" containerID="d06a50c656ba331bcd51fa11dbd49bbb4dd6335bc66fe00ae8e7cba4eeaea4ac" exitCode=6
Jul 16 21:28:08.174569 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:08.174460    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk" event={"ID":"01335f11-a228-4015-b31f-cf670fa1acd6","Type":"ContainerDied","Data":"d06a50c656ba331bcd51fa11dbd49bbb4dd6335bc66fe00ae8e7cba4eeaea4ac"}
Jul 16 21:28:08.174569 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:08.174502    2530 scope.go:117] "RemoveContainer" containerID="6ad2f79987420e214433b08808a16a65a212fd90a17aa6f84d30849b405c0284"
Jul 16 21:28:08.174716 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:08.174704    2530 scope.go:117] "RemoveContainer" containerID="d06a50c656ba331bcd51fa11dbd49bbb4dd6335bc66fe00ae8e7cba4eeaea4ac"
Jul 16 21:28:08.174881 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:28:08.174865    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk_odh-ai-gateway-infra(01335f11-a228-4015-b31f-cf670fa1acd6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk" podUID="01335f11-a228-4015-b31f-cf670fa1acd6"
Jul 16 21:28:08.502327 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:08.502296    2530 scope.go:117] "RemoveContainer" containerID="f5cce8b1e06f53d1144b57ca1c1d12bee076adf2a411e1321f6f96ffa4370792"
Jul 16 21:28:09.180647 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:09.180617    2530 generic.go:356] "Generic (PLEG): container finished" podID="7fbfc5ee-c06b-4f97-bff3-4daccad99781" containerID="2a72511f8588092af269ea1ecb57aa7bb9b23078d13c136c2541d5def0ebeb96" exitCode=6
Jul 16 21:28:09.181080 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:09.180691    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf" event={"ID":"7fbfc5ee-c06b-4f97-bff3-4daccad99781","Type":"ContainerDied","Data":"2a72511f8588092af269ea1ecb57aa7bb9b23078d13c136c2541d5def0ebeb96"}
Jul 16 21:28:09.181080 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:09.180763    2530 scope.go:117] "RemoveContainer" containerID="f5cce8b1e06f53d1144b57ca1c1d12bee076adf2a411e1321f6f96ffa4370792"
Jul 16 21:28:09.181232 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:09.181137    2530 scope.go:117] "RemoveContainer" containerID="2a72511f8588092af269ea1ecb57aa7bb9b23078d13c136c2541d5def0ebeb96"
Jul 16 21:28:09.181400 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:28:09.181374    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-flgrf_odh-ai-gateway-infra(7fbfc5ee-c06b-4f97-bff3-4daccad99781)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf" podUID="7fbfc5ee-c06b-4f97-bff3-4daccad99781"
Jul 16 21:28:09.216798 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:09.216771    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk"]
Jul 16 21:28:09.341480 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:09.341459    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk"
Jul 16 21:28:09.462653 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:09.462625    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01335f11-a228-4015-b31f-cf670fa1acd6-maas-api-service-ca\") pod \"01335f11-a228-4015-b31f-cf670fa1acd6\" (UID: \"01335f11-a228-4015-b31f-cf670fa1acd6\") "
Jul 16 21:28:09.463004 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:09.462980    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01335f11-a228-4015-b31f-cf670fa1acd6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "01335f11-a228-4015-b31f-cf670fa1acd6" (UID: "01335f11-a228-4015-b31f-cf670fa1acd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:28:09.563559 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:09.563526    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01335f11-a228-4015-b31f-cf670fa1acd6-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:28:10.188195 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:10.188158    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk"
Jul 16 21:28:10.188664 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:10.188167    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk" event={"ID":"01335f11-a228-4015-b31f-cf670fa1acd6","Type":"ContainerDied","Data":"a580ac95d7d6b6c4fdee26f252afc7b70256140473b9fa52a61f108f3a817856"}
Jul 16 21:28:10.188664 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:10.188313    2530 scope.go:117] "RemoveContainer" containerID="d06a50c656ba331bcd51fa11dbd49bbb4dd6335bc66fe00ae8e7cba4eeaea4ac"
Jul 16 21:28:10.212863 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:10.212833    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf"]
Jul 16 21:28:10.218165 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:10.218143    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk"]
Jul 16 21:28:10.223511 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:10.223492    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-q5cdk"]
Jul 16 21:28:10.344718 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:10.344693    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf"
Jul 16 21:28:10.471440 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:10.471408    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbfc5ee-c06b-4f97-bff3-4daccad99781-maas-api-service-ca\") pod \"7fbfc5ee-c06b-4f97-bff3-4daccad99781\" (UID: \"7fbfc5ee-c06b-4f97-bff3-4daccad99781\") "
Jul 16 21:28:10.471765 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:10.471741    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbfc5ee-c06b-4f97-bff3-4daccad99781-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7fbfc5ee-c06b-4f97-bff3-4daccad99781" (UID: "7fbfc5ee-c06b-4f97-bff3-4daccad99781"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:28:10.507900 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:10.507869    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01335f11-a228-4015-b31f-cf670fa1acd6" path="/var/lib/kubelet/pods/01335f11-a228-4015-b31f-cf670fa1acd6/volumes"
Jul 16 21:28:10.571931 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:10.571904    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fbfc5ee-c06b-4f97-bff3-4daccad99781-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:28:11.196854 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:11.196828    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf"
Jul 16 21:28:11.196854 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:11.196818    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf" event={"ID":"7fbfc5ee-c06b-4f97-bff3-4daccad99781","Type":"ContainerDied","Data":"9ba5a0ccedd16ae5513c8c215aae361bc5f2ccee1582983cb16e3f5b640b2b3b"}
Jul 16 21:28:11.197373 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:11.196894    2530 scope.go:117] "RemoveContainer" containerID="2a72511f8588092af269ea1ecb57aa7bb9b23078d13c136c2541d5def0ebeb96"
Jul 16 21:28:11.212298 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:11.212273    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf"]
Jul 16 21:28:11.216493 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:11.216472    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-flgrf"]
Jul 16 21:28:12.508295 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:12.508263    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fbfc5ee-c06b-4f97-bff3-4daccad99781" path="/var/lib/kubelet/pods/7fbfc5ee-c06b-4f97-bff3-4daccad99781/volumes"
Jul 16 21:28:22.504705 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:22.504673    2530 scope.go:117] "RemoveContainer" containerID="5cc87896ee0cf2498475fccfd4e93a56bf3333a80bb4f8089639a5143f50e49b"
Jul 16 21:28:23.253852 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:23.253820    2530 generic.go:356] "Generic (PLEG): container finished" podID="a0cc5659-1d93-4998-b541-4b3ecc74aee4" containerID="959f43a0a9218d90293048834cc86b0297b8be1cfd6fbe8eabd28661536ff701" exitCode=6
Jul 16 21:28:23.253985 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:23.253895    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x" event={"ID":"a0cc5659-1d93-4998-b541-4b3ecc74aee4","Type":"ContainerDied","Data":"959f43a0a9218d90293048834cc86b0297b8be1cfd6fbe8eabd28661536ff701"}
Jul 16 21:28:23.253985 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:23.253955    2530 scope.go:117] "RemoveContainer" containerID="5cc87896ee0cf2498475fccfd4e93a56bf3333a80bb4f8089639a5143f50e49b"
Jul 16 21:28:23.254300 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:23.254272    2530 scope.go:117] "RemoveContainer" containerID="959f43a0a9218d90293048834cc86b0297b8be1cfd6fbe8eabd28661536ff701"
Jul 16 21:28:23.254514 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:28:23.254497    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x_odh-ai-gateway-infra(a0cc5659-1d93-4998-b541-4b3ecc74aee4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x" podUID="a0cc5659-1d93-4998-b541-4b3ecc74aee4"
Jul 16 21:28:24.285178 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:24.285140    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x"]
Jul 16 21:28:24.411400 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:24.411380    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x"
Jul 16 21:28:24.589892 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:24.589824    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0cc5659-1d93-4998-b541-4b3ecc74aee4-maas-api-service-ca\") pod \"a0cc5659-1d93-4998-b541-4b3ecc74aee4\" (UID: \"a0cc5659-1d93-4998-b541-4b3ecc74aee4\") "
Jul 16 21:28:24.590148 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:24.590128    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0cc5659-1d93-4998-b541-4b3ecc74aee4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0cc5659-1d93-4998-b541-4b3ecc74aee4" (UID: "a0cc5659-1d93-4998-b541-4b3ecc74aee4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:28:24.690309 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:24.690277    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0cc5659-1d93-4998-b541-4b3ecc74aee4-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:28:25.264972 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:25.264940    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x"
Jul 16 21:28:25.265137 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:25.264982    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x" event={"ID":"a0cc5659-1d93-4998-b541-4b3ecc74aee4","Type":"ContainerDied","Data":"0f1d107c8dc7cc13b0bd88f46086d073152505ee247b0a670c6a1ec06687d253"}
Jul 16 21:28:25.265137 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:25.265045    2530 scope.go:117] "RemoveContainer" containerID="959f43a0a9218d90293048834cc86b0297b8be1cfd6fbe8eabd28661536ff701"
Jul 16 21:28:25.286293 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:25.286273    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x"]
Jul 16 21:28:25.289578 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:25.289555    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-mtk2x"]
Jul 16 21:28:26.507313 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:26.507285    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0cc5659-1d93-4998-b541-4b3ecc74aee4" path="/var/lib/kubelet/pods/a0cc5659-1d93-4998-b541-4b3ecc74aee4/volumes"
Jul 16 21:28:29.843724 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.843693    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw"]
Jul 16 21:28:29.844085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.843989    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0cc5659-1d93-4998-b541-4b3ecc74aee4" containerName="revoke-keys"
Jul 16 21:28:29.844085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.843999    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0cc5659-1d93-4998-b541-4b3ecc74aee4" containerName="revoke-keys"
Jul 16 21:28:29.844085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844007    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbfc5ee-c06b-4f97-bff3-4daccad99781" containerName="revoke-keys"
Jul 16 21:28:29.844085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844012    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbfc5ee-c06b-4f97-bff3-4daccad99781" containerName="revoke-keys"
Jul 16 21:28:29.844085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844019    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01335f11-a228-4015-b31f-cf670fa1acd6" containerName="revoke-keys"
Jul 16 21:28:29.844085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844024    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="01335f11-a228-4015-b31f-cf670fa1acd6" containerName="revoke-keys"
Jul 16 21:28:29.844085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844034    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0cc5659-1d93-4998-b541-4b3ecc74aee4" containerName="revoke-keys"
Jul 16 21:28:29.844085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844039    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0cc5659-1d93-4998-b541-4b3ecc74aee4" containerName="revoke-keys"
Jul 16 21:28:29.844085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844044    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01335f11-a228-4015-b31f-cf670fa1acd6" containerName="revoke-keys"
Jul 16 21:28:29.844085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844048    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="01335f11-a228-4015-b31f-cf670fa1acd6" containerName="revoke-keys"
Jul 16 21:28:29.844085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844054    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01335f11-a228-4015-b31f-cf670fa1acd6" containerName="revoke-keys"
Jul 16 21:28:29.844085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844059    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="01335f11-a228-4015-b31f-cf670fa1acd6" containerName="revoke-keys"
Jul 16 21:28:29.844085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844066    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0cc5659-1d93-4998-b541-4b3ecc74aee4" containerName="revoke-keys"
Jul 16 21:28:29.844085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844070    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0cc5659-1d93-4998-b541-4b3ecc74aee4" containerName="revoke-keys"
Jul 16 21:28:29.844085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844080    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbfc5ee-c06b-4f97-bff3-4daccad99781" containerName="revoke-keys"
Jul 16 21:28:29.844085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844085    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbfc5ee-c06b-4f97-bff3-4daccad99781" containerName="revoke-keys"
Jul 16 21:28:29.844613 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844164    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="01335f11-a228-4015-b31f-cf670fa1acd6" containerName="revoke-keys"
Jul 16 21:28:29.844613 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844172    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0cc5659-1d93-4998-b541-4b3ecc74aee4" containerName="revoke-keys"
Jul 16 21:28:29.844613 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844177    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fbfc5ee-c06b-4f97-bff3-4daccad99781" containerName="revoke-keys"
Jul 16 21:28:29.844613 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844187    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fbfc5ee-c06b-4f97-bff3-4daccad99781" containerName="revoke-keys"
Jul 16 21:28:29.844613 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844192    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fbfc5ee-c06b-4f97-bff3-4daccad99781" containerName="revoke-keys"
Jul 16 21:28:29.844613 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844198    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0cc5659-1d93-4998-b541-4b3ecc74aee4" containerName="revoke-keys"
Jul 16 21:28:29.844613 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.844202    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="01335f11-a228-4015-b31f-cf670fa1acd6" containerName="revoke-keys"
Jul 16 21:28:29.848238 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.848223    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw"
Jul 16 21:28:29.850687 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.850663    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:28:29.854362 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:29.854340    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw"]
Jul 16 21:28:30.028516 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:30.028491    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77084780-b7fa-4a1e-b874-cb6cc5c72cc7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw\" (UID: \"77084780-b7fa-4a1e-b874-cb6cc5c72cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw"
Jul 16 21:28:30.129052 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:30.128989    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77084780-b7fa-4a1e-b874-cb6cc5c72cc7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw\" (UID: \"77084780-b7fa-4a1e-b874-cb6cc5c72cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw"
Jul 16 21:28:30.129537 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:30.129518    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77084780-b7fa-4a1e-b874-cb6cc5c72cc7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw\" (UID: \"77084780-b7fa-4a1e-b874-cb6cc5c72cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw"
Jul 16 21:28:30.159109 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:30.159070    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw"
Jul 16 21:28:30.287381 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:30.287360    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw"]
Jul 16 21:28:30.289633 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:28:30.289609    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77084780_b7fa_4a1e_b874_cb6cc5c72cc7.slice/crio-f3d85236fdf35a1b4ece067b49675ee63095a57eb77f34b3f934caeadfbe643c WatchSource:0}: Error finding container f3d85236fdf35a1b4ece067b49675ee63095a57eb77f34b3f934caeadfbe643c: Status 404 returned error can't find the container with id f3d85236fdf35a1b4ece067b49675ee63095a57eb77f34b3f934caeadfbe643c
Jul 16 21:28:31.294312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:31.294284    2530 generic.go:356] "Generic (PLEG): container finished" podID="77084780-b7fa-4a1e-b874-cb6cc5c72cc7" containerID="e4fcb2d53b0e0d5873ff20d786433dade9e09a2249a24b5915b9060a455a23c9" exitCode=6
Jul 16 21:28:31.294665 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:31.294330    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw" event={"ID":"77084780-b7fa-4a1e-b874-cb6cc5c72cc7","Type":"ContainerDied","Data":"e4fcb2d53b0e0d5873ff20d786433dade9e09a2249a24b5915b9060a455a23c9"}
Jul 16 21:28:31.294665 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:31.294365    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw" event={"ID":"77084780-b7fa-4a1e-b874-cb6cc5c72cc7","Type":"ContainerStarted","Data":"f3d85236fdf35a1b4ece067b49675ee63095a57eb77f34b3f934caeadfbe643c"}
Jul 16 21:28:31.294665 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:31.294501    2530 scope.go:117] "RemoveContainer" containerID="e4fcb2d53b0e0d5873ff20d786433dade9e09a2249a24b5915b9060a455a23c9"
Jul 16 21:28:32.301185 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:32.301150    2530 generic.go:356] "Generic (PLEG): container finished" podID="77084780-b7fa-4a1e-b874-cb6cc5c72cc7" containerID="7df8041a827d5c4074c6cf0be749e98f3c08b8100cd1e527762a3ee4a185cd1f" exitCode=6
Jul 16 21:28:32.301614 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:32.301193    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw" event={"ID":"77084780-b7fa-4a1e-b874-cb6cc5c72cc7","Type":"ContainerDied","Data":"7df8041a827d5c4074c6cf0be749e98f3c08b8100cd1e527762a3ee4a185cd1f"}
Jul 16 21:28:32.301614 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:32.301250    2530 scope.go:117] "RemoveContainer" containerID="e4fcb2d53b0e0d5873ff20d786433dade9e09a2249a24b5915b9060a455a23c9"
Jul 16 21:28:32.301614 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:32.301392    2530 scope.go:117] "RemoveContainer" containerID="7df8041a827d5c4074c6cf0be749e98f3c08b8100cd1e527762a3ee4a185cd1f"
Jul 16 21:28:32.301736 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:28:32.301612    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw_odh-ai-gateway-infra(77084780-b7fa-4a1e-b874-cb6cc5c72cc7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw" podUID="77084780-b7fa-4a1e-b874-cb6cc5c72cc7"
Jul 16 21:28:33.307424 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:33.307402    2530 scope.go:117] "RemoveContainer" containerID="7df8041a827d5c4074c6cf0be749e98f3c08b8100cd1e527762a3ee4a185cd1f"
Jul 16 21:28:33.307806 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:28:33.307574    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw_odh-ai-gateway-infra(77084780-b7fa-4a1e-b874-cb6cc5c72cc7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw" podUID="77084780-b7fa-4a1e-b874-cb6cc5c72cc7"
Jul 16 21:28:42.472030 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:42.471995    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk"]
Jul 16 21:28:42.472496 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:42.472341    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbfc5ee-c06b-4f97-bff3-4daccad99781" containerName="revoke-keys"
Jul 16 21:28:42.472496 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:42.472353    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbfc5ee-c06b-4f97-bff3-4daccad99781" containerName="revoke-keys"
Jul 16 21:28:42.472496 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:42.472427    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0cc5659-1d93-4998-b541-4b3ecc74aee4" containerName="revoke-keys"
Jul 16 21:28:42.472496 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:42.472434    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="01335f11-a228-4015-b31f-cf670fa1acd6" containerName="revoke-keys"
Jul 16 21:28:42.476688 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:42.476668    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk"
Jul 16 21:28:42.482986 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:42.482962    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk"]
Jul 16 21:28:42.524744 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:42.524711    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3b723e-50c1-449f-a0ed-a9ced903d369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-gclrk\" (UID: \"5f3b723e-50c1-449f-a0ed-a9ced903d369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk"
Jul 16 21:28:42.625663 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:42.625628    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3b723e-50c1-449f-a0ed-a9ced903d369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-gclrk\" (UID: \"5f3b723e-50c1-449f-a0ed-a9ced903d369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk"
Jul 16 21:28:42.626246 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:42.626228    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3b723e-50c1-449f-a0ed-a9ced903d369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-gclrk\" (UID: \"5f3b723e-50c1-449f-a0ed-a9ced903d369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk"
Jul 16 21:28:42.788524 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:42.788460    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk"
Jul 16 21:28:42.931013 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:42.930990    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk"]
Jul 16 21:28:42.933313 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:28:42.933280    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3b723e_50c1_449f_a0ed_a9ced903d369.slice/crio-cb4a880d2018c5ae29f952e74cb4a048136ec4a54d6b2b433d4c4b3522f8b0b4 WatchSource:0}: Error finding container cb4a880d2018c5ae29f952e74cb4a048136ec4a54d6b2b433d4c4b3522f8b0b4: Status 404 returned error can't find the container with id cb4a880d2018c5ae29f952e74cb4a048136ec4a54d6b2b433d4c4b3522f8b0b4
Jul 16 21:28:42.961783 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:42.961759    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt"]
Jul 16 21:28:42.966249 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:42.966232    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt"
Jul 16 21:28:42.972164 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:42.972088    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt"]
Jul 16 21:28:43.029424 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:43.029393    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt\" (UID: \"2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt"
Jul 16 21:28:43.130251 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:43.130187    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt\" (UID: \"2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt"
Jul 16 21:28:43.130788 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:43.130772    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt\" (UID: \"2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt"
Jul 16 21:28:43.276988 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:43.276968    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt"
Jul 16 21:28:43.353787 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:43.353759    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk" event={"ID":"5f3b723e-50c1-449f-a0ed-a9ced903d369","Type":"ContainerStarted","Data":"626a0cd174432e336b08e09ca82e8836ac4b907fc49245d4a5c5657a28e1ee95"}
Jul 16 21:28:43.353952 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:43.353813    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk" event={"ID":"5f3b723e-50c1-449f-a0ed-a9ced903d369","Type":"ContainerStarted","Data":"cb4a880d2018c5ae29f952e74cb4a048136ec4a54d6b2b433d4c4b3522f8b0b4"}
Jul 16 21:28:43.370677 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:43.370635    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk" podStartSLOduration=1.370618481 podStartE2EDuration="1.370618481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:28:43.368936071 +0000 UTC m=+3361.390523839" watchObservedRunningTime="2026-07-16 21:28:43.370618481 +0000 UTC m=+3361.392206294"
Jul 16 21:28:43.405379 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:43.405306    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt"]
Jul 16 21:28:43.409112 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:28:43.409077    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cfbd56e_7270_4c33_9ac0_9b45cb8dfb30.slice/crio-45d329c5dff077664fbc5b97e7b57c5abec4761af569c466d522936b37440970 WatchSource:0}: Error finding container 45d329c5dff077664fbc5b97e7b57c5abec4761af569c466d522936b37440970: Status 404 returned error can't find the container with id 45d329c5dff077664fbc5b97e7b57c5abec4761af569c466d522936b37440970
Jul 16 21:28:44.365024 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:44.364927    2530 generic.go:356] "Generic (PLEG): container finished" podID="2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30" containerID="4b1749704b01d1850f07e977624da2ac03c0c8d73ad583f72034df96ccd7ebdc" exitCode=6
Jul 16 21:28:44.365024 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:44.365005    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt" event={"ID":"2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30","Type":"ContainerDied","Data":"4b1749704b01d1850f07e977624da2ac03c0c8d73ad583f72034df96ccd7ebdc"}
Jul 16 21:28:44.365524 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:44.365089    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt" event={"ID":"2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30","Type":"ContainerStarted","Data":"45d329c5dff077664fbc5b97e7b57c5abec4761af569c466d522936b37440970"}
Jul 16 21:28:44.365524 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:44.365205    2530 scope.go:117] "RemoveContainer" containerID="4b1749704b01d1850f07e977624da2ac03c0c8d73ad583f72034df96ccd7ebdc"
Jul 16 21:28:44.366562 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:44.366537    2530 generic.go:356] "Generic (PLEG): container finished" podID="5f3b723e-50c1-449f-a0ed-a9ced903d369" containerID="626a0cd174432e336b08e09ca82e8836ac4b907fc49245d4a5c5657a28e1ee95" exitCode=6
Jul 16 21:28:44.366652 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:44.366603    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk" event={"ID":"5f3b723e-50c1-449f-a0ed-a9ced903d369","Type":"ContainerDied","Data":"626a0cd174432e336b08e09ca82e8836ac4b907fc49245d4a5c5657a28e1ee95"}
Jul 16 21:28:44.366704 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:44.366697    2530 scope.go:117] "RemoveContainer" containerID="626a0cd174432e336b08e09ca82e8836ac4b907fc49245d4a5c5657a28e1ee95"
Jul 16 21:28:45.372886 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:45.372853    2530 generic.go:356] "Generic (PLEG): container finished" podID="5f3b723e-50c1-449f-a0ed-a9ced903d369" containerID="252cbb198d379a2b72862ebbe5bc40a8ecb5de2b6c57aef8477c2dd3d8b2e4c1" exitCode=6
Jul 16 21:28:45.373333 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:45.372922    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk" event={"ID":"5f3b723e-50c1-449f-a0ed-a9ced903d369","Type":"ContainerDied","Data":"252cbb198d379a2b72862ebbe5bc40a8ecb5de2b6c57aef8477c2dd3d8b2e4c1"}
Jul 16 21:28:45.373333 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:45.372975    2530 scope.go:117] "RemoveContainer" containerID="626a0cd174432e336b08e09ca82e8836ac4b907fc49245d4a5c5657a28e1ee95"
Jul 16 21:28:45.373333 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:45.373150    2530 scope.go:117] "RemoveContainer" containerID="252cbb198d379a2b72862ebbe5bc40a8ecb5de2b6c57aef8477c2dd3d8b2e4c1"
Jul 16 21:28:45.373333 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:28:45.373330    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-gclrk_odh-ai-gateway-infra(5f3b723e-50c1-449f-a0ed-a9ced903d369)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk" podUID="5f3b723e-50c1-449f-a0ed-a9ced903d369"
Jul 16 21:28:45.374985 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:45.374966    2530 generic.go:356] "Generic (PLEG): container finished" podID="2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30" containerID="77fdfdd3013122aa97445aae2a1c0da557a67b5de2fdf12f47aaacc6aa7e56d7" exitCode=6
Jul 16 21:28:45.375085 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:45.375021    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt" event={"ID":"2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30","Type":"ContainerDied","Data":"77fdfdd3013122aa97445aae2a1c0da557a67b5de2fdf12f47aaacc6aa7e56d7"}
Jul 16 21:28:45.375252 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:45.375237    2530 scope.go:117] "RemoveContainer" containerID="77fdfdd3013122aa97445aae2a1c0da557a67b5de2fdf12f47aaacc6aa7e56d7"
Jul 16 21:28:45.375421 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:28:45.375403    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt_odh-ai-gateway-infra(2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt" podUID="2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30"
Jul 16 21:28:45.385993 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:45.385957    2530 scope.go:117] "RemoveContainer" containerID="4b1749704b01d1850f07e977624da2ac03c0c8d73ad583f72034df96ccd7ebdc"
Jul 16 21:28:46.381279 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:46.381251    2530 scope.go:117] "RemoveContainer" containerID="252cbb198d379a2b72862ebbe5bc40a8ecb5de2b6c57aef8477c2dd3d8b2e4c1"
Jul 16 21:28:46.381715 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:28:46.381458    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-gclrk_odh-ai-gateway-infra(5f3b723e-50c1-449f-a0ed-a9ced903d369)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk" podUID="5f3b723e-50c1-449f-a0ed-a9ced903d369"
Jul 16 21:28:46.382741 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:46.382727    2530 scope.go:117] "RemoveContainer" containerID="77fdfdd3013122aa97445aae2a1c0da557a67b5de2fdf12f47aaacc6aa7e56d7"
Jul 16 21:28:46.382864 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:28:46.382848    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt_odh-ai-gateway-infra(2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt" podUID="2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30"
Jul 16 21:28:47.501839 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:47.501814    2530 scope.go:117] "RemoveContainer" containerID="7df8041a827d5c4074c6cf0be749e98f3c08b8100cd1e527762a3ee4a185cd1f"
Jul 16 21:28:48.393911 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:48.393844    2530 generic.go:356] "Generic (PLEG): container finished" podID="77084780-b7fa-4a1e-b874-cb6cc5c72cc7" containerID="557b1fc99dc2b2ac10c75af676747d28f6780dec4e9b829587a1c91e79a3e84d" exitCode=6
Jul 16 21:28:48.394071 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:48.393925    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw" event={"ID":"77084780-b7fa-4a1e-b874-cb6cc5c72cc7","Type":"ContainerDied","Data":"557b1fc99dc2b2ac10c75af676747d28f6780dec4e9b829587a1c91e79a3e84d"}
Jul 16 21:28:48.394071 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:48.393994    2530 scope.go:117] "RemoveContainer" containerID="7df8041a827d5c4074c6cf0be749e98f3c08b8100cd1e527762a3ee4a185cd1f"
Jul 16 21:28:48.394316 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:48.394302    2530 scope.go:117] "RemoveContainer" containerID="557b1fc99dc2b2ac10c75af676747d28f6780dec4e9b829587a1c91e79a3e84d"
Jul 16 21:28:48.394508 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:28:48.394491    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw_odh-ai-gateway-infra(77084780-b7fa-4a1e-b874-cb6cc5c72cc7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw" podUID="77084780-b7fa-4a1e-b874-cb6cc5c72cc7"
Jul 16 21:28:49.421384 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:49.421354    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw"]
Jul 16 21:28:49.559073 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:49.559051    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw"
Jul 16 21:28:49.686199 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:49.686132    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77084780-b7fa-4a1e-b874-cb6cc5c72cc7-maas-api-service-ca\") pod \"77084780-b7fa-4a1e-b874-cb6cc5c72cc7\" (UID: \"77084780-b7fa-4a1e-b874-cb6cc5c72cc7\") "
Jul 16 21:28:49.686480 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:49.686461    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77084780-b7fa-4a1e-b874-cb6cc5c72cc7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "77084780-b7fa-4a1e-b874-cb6cc5c72cc7" (UID: "77084780-b7fa-4a1e-b874-cb6cc5c72cc7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:28:49.787439 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:49.787418    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77084780-b7fa-4a1e-b874-cb6cc5c72cc7-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:28:50.406072 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:50.406036    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw" event={"ID":"77084780-b7fa-4a1e-b874-cb6cc5c72cc7","Type":"ContainerDied","Data":"f3d85236fdf35a1b4ece067b49675ee63095a57eb77f34b3f934caeadfbe643c"}
Jul 16 21:28:50.406072 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:50.406069    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw"
Jul 16 21:28:50.406329 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:50.406139    2530 scope.go:117] "RemoveContainer" containerID="557b1fc99dc2b2ac10c75af676747d28f6780dec4e9b829587a1c91e79a3e84d"
Jul 16 21:28:50.427471 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:50.427444    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw"]
Jul 16 21:28:50.433188 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:50.433170    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-k5kpw"]
Jul 16 21:28:50.507425 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:50.507395    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77084780-b7fa-4a1e-b874-cb6cc5c72cc7" path="/var/lib/kubelet/pods/77084780-b7fa-4a1e-b874-cb6cc5c72cc7/volumes"
Jul 16 21:28:58.502900 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:58.502872    2530 scope.go:117] "RemoveContainer" containerID="77fdfdd3013122aa97445aae2a1c0da557a67b5de2fdf12f47aaacc6aa7e56d7"
Jul 16 21:28:59.450009 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:59.449922    2530 generic.go:356] "Generic (PLEG): container finished" podID="2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30" containerID="b18fcb74b8f95f04accb2df966a6bd6e25e4767dd20e4157751b72c4c48af1b8" exitCode=6
Jul 16 21:28:59.450009 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:59.449996    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt" event={"ID":"2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30","Type":"ContainerDied","Data":"b18fcb74b8f95f04accb2df966a6bd6e25e4767dd20e4157751b72c4c48af1b8"}
Jul 16 21:28:59.450258 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:59.450053    2530 scope.go:117] "RemoveContainer" containerID="77fdfdd3013122aa97445aae2a1c0da557a67b5de2fdf12f47aaacc6aa7e56d7"
Jul 16 21:28:59.450411 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:28:59.450393    2530 scope.go:117] "RemoveContainer" containerID="b18fcb74b8f95f04accb2df966a6bd6e25e4767dd20e4157751b72c4c48af1b8"
Jul 16 21:28:59.450626 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:28:59.450606    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt_odh-ai-gateway-infra(2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt" podUID="2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30"
Jul 16 21:29:00.477814 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:00.477778    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt"]
Jul 16 21:29:00.609607 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:00.609587    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt"
Jul 16 21:29:00.675528 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:00.675495    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30-maas-api-service-ca\") pod \"2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30\" (UID: \"2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30\") "
Jul 16 21:29:00.675850 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:00.675828    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30" (UID: "2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:29:00.776939 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:00.776860    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:29:01.461891 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:01.461852    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt" event={"ID":"2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30","Type":"ContainerDied","Data":"45d329c5dff077664fbc5b97e7b57c5abec4761af569c466d522936b37440970"}
Jul 16 21:29:01.461891 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:01.461890    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt"
Jul 16 21:29:01.462132 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:01.461927    2530 scope.go:117] "RemoveContainer" containerID="b18fcb74b8f95f04accb2df966a6bd6e25e4767dd20e4157751b72c4c48af1b8"
Jul 16 21:29:01.483920 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:01.483891    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt"]
Jul 16 21:29:01.490089 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:01.490070    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-drsvt"]
Jul 16 21:29:01.502721 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:01.502700    2530 scope.go:117] "RemoveContainer" containerID="252cbb198d379a2b72862ebbe5bc40a8ecb5de2b6c57aef8477c2dd3d8b2e4c1"
Jul 16 21:29:02.469342 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:02.469257    2530 generic.go:356] "Generic (PLEG): container finished" podID="5f3b723e-50c1-449f-a0ed-a9ced903d369" containerID="f878c252d679c17a9ad603e9a619394499af9e19d5609684cbce825e8e7cb596" exitCode=6
Jul 16 21:29:02.469342 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:02.469321    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk" event={"ID":"5f3b723e-50c1-449f-a0ed-a9ced903d369","Type":"ContainerDied","Data":"f878c252d679c17a9ad603e9a619394499af9e19d5609684cbce825e8e7cb596"}
Jul 16 21:29:02.469523 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:02.469373    2530 scope.go:117] "RemoveContainer" containerID="252cbb198d379a2b72862ebbe5bc40a8ecb5de2b6c57aef8477c2dd3d8b2e4c1"
Jul 16 21:29:02.469698 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:02.469680    2530 scope.go:117] "RemoveContainer" containerID="f878c252d679c17a9ad603e9a619394499af9e19d5609684cbce825e8e7cb596"
Jul 16 21:29:02.469888 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:29:02.469870    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-gclrk_odh-ai-gateway-infra(5f3b723e-50c1-449f-a0ed-a9ced903d369)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk" podUID="5f3b723e-50c1-449f-a0ed-a9ced903d369"
Jul 16 21:29:02.512234 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:02.512203    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30" path="/var/lib/kubelet/pods/2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30/volumes"
Jul 16 21:29:03.038450 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.038416    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9331e3-2cfjp"]
Jul 16 21:29:03.038760 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.038748    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77084780-b7fa-4a1e-b874-cb6cc5c72cc7" containerName="revoke-keys"
Jul 16 21:29:03.038804 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.038762    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="77084780-b7fa-4a1e-b874-cb6cc5c72cc7" containerName="revoke-keys"
Jul 16 21:29:03.038804 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.038774    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77084780-b7fa-4a1e-b874-cb6cc5c72cc7" containerName="revoke-keys"
Jul 16 21:29:03.038804 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.038779    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="77084780-b7fa-4a1e-b874-cb6cc5c72cc7" containerName="revoke-keys"
Jul 16 21:29:03.038804 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.038786    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30" containerName="revoke-keys"
Jul 16 21:29:03.038804 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.038791    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30" containerName="revoke-keys"
Jul 16 21:29:03.038804 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.038804    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30" containerName="revoke-keys"
Jul 16 21:29:03.038986 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.038809    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30" containerName="revoke-keys"
Jul 16 21:29:03.038986 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.038860    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="77084780-b7fa-4a1e-b874-cb6cc5c72cc7" containerName="revoke-keys"
Jul 16 21:29:03.038986 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.038867    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30" containerName="revoke-keys"
Jul 16 21:29:03.038986 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.038872    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30" containerName="revoke-keys"
Jul 16 21:29:03.038986 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.038879    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30" containerName="revoke-keys"
Jul 16 21:29:03.038986 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.038886    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="77084780-b7fa-4a1e-b874-cb6cc5c72cc7" containerName="revoke-keys"
Jul 16 21:29:03.043283 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.043267    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9331e3-2cfjp"
Jul 16 21:29:03.049202 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.049177    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9331e3-2cfjp"]
Jul 16 21:29:03.198140 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.198084    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f019e659-03f4-4eea-946a-cc25a4c2e035-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-9331e3-2cfjp\" (UID: \"f019e659-03f4-4eea-946a-cc25a4c2e035\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9331e3-2cfjp"
Jul 16 21:29:03.298651 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.298575    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f019e659-03f4-4eea-946a-cc25a4c2e035-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-9331e3-2cfjp\" (UID: \"f019e659-03f4-4eea-946a-cc25a4c2e035\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9331e3-2cfjp"
Jul 16 21:29:03.299179 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.299159    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f019e659-03f4-4eea-946a-cc25a4c2e035-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-9331e3-2cfjp\" (UID: \"f019e659-03f4-4eea-946a-cc25a4c2e035\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9331e3-2cfjp"
Jul 16 21:29:03.355515 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.355489    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9331e3-2cfjp"
Jul 16 21:29:03.495013 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.494991    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9331e3-2cfjp"]
Jul 16 21:29:03.496988 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:29:03.496956    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf019e659_03f4_4eea_946a_cc25a4c2e035.slice/crio-2fd9c3bcd62730a76a008aa24e62ccbff7c921d244139eac1bf1460cd82ab731 WatchSource:0}: Error finding container 2fd9c3bcd62730a76a008aa24e62ccbff7c921d244139eac1bf1460cd82ab731: Status 404 returned error can't find the container with id 2fd9c3bcd62730a76a008aa24e62ccbff7c921d244139eac1bf1460cd82ab731
Jul 16 21:29:03.500340 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.500305    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk"]
Jul 16 21:29:03.626266 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.626243    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk"
Jul 16 21:29:03.702028 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.701991    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3b723e-50c1-449f-a0ed-a9ced903d369-maas-api-service-ca\") pod \"5f3b723e-50c1-449f-a0ed-a9ced903d369\" (UID: \"5f3b723e-50c1-449f-a0ed-a9ced903d369\") "
Jul 16 21:29:03.702446 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.702425    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3b723e-50c1-449f-a0ed-a9ced903d369-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5f3b723e-50c1-449f-a0ed-a9ced903d369" (UID: "5f3b723e-50c1-449f-a0ed-a9ced903d369"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:29:03.803083 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:03.803045    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f3b723e-50c1-449f-a0ed-a9ced903d369-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:29:04.482552 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.482516    2530 generic.go:356] "Generic (PLEG): container finished" podID="f019e659-03f4-4eea-946a-cc25a4c2e035" containerID="717e5a58e2a67de99d7211ac8ee2940733eea34b4fead3380c6de1af314eefee" exitCode=0
Jul 16 21:29:04.482710 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.482607    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9331e3-2cfjp" event={"ID":"f019e659-03f4-4eea-946a-cc25a4c2e035","Type":"ContainerDied","Data":"717e5a58e2a67de99d7211ac8ee2940733eea34b4fead3380c6de1af314eefee"}
Jul 16 21:29:04.482710 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.482676    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9331e3-2cfjp" event={"ID":"f019e659-03f4-4eea-946a-cc25a4c2e035","Type":"ContainerStarted","Data":"2fd9c3bcd62730a76a008aa24e62ccbff7c921d244139eac1bf1460cd82ab731"}
Jul 16 21:29:04.483837 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.483783    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk"
Jul 16 21:29:04.483837 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.483791    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk" event={"ID":"5f3b723e-50c1-449f-a0ed-a9ced903d369","Type":"ContainerDied","Data":"cb4a880d2018c5ae29f952e74cb4a048136ec4a54d6b2b433d4c4b3522f8b0b4"}
Jul 16 21:29:04.483837 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.483835    2530 scope.go:117] "RemoveContainer" containerID="f878c252d679c17a9ad603e9a619394499af9e19d5609684cbce825e8e7cb596"
Jul 16 21:29:04.513620 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.513591    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk"]
Jul 16 21:29:04.519236 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.519215    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-gclrk"]
Jul 16 21:29:04.714989 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.714955    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf"]
Jul 16 21:29:04.715355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.715298    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77084780-b7fa-4a1e-b874-cb6cc5c72cc7" containerName="revoke-keys"
Jul 16 21:29:04.715355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.715310    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="77084780-b7fa-4a1e-b874-cb6cc5c72cc7" containerName="revoke-keys"
Jul 16 21:29:04.715355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.715319    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30" containerName="revoke-keys"
Jul 16 21:29:04.715355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.715324    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cfbd56e-7270-4c33-9ac0-9b45cb8dfb30" containerName="revoke-keys"
Jul 16 21:29:04.715355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.715333    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3b723e-50c1-449f-a0ed-a9ced903d369" containerName="revoke-keys"
Jul 16 21:29:04.715355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.715339    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3b723e-50c1-449f-a0ed-a9ced903d369" containerName="revoke-keys"
Jul 16 21:29:04.715355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.715347    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3b723e-50c1-449f-a0ed-a9ced903d369" containerName="revoke-keys"
Jul 16 21:29:04.715355 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.715351    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3b723e-50c1-449f-a0ed-a9ced903d369" containerName="revoke-keys"
Jul 16 21:29:04.715613 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.715407    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="77084780-b7fa-4a1e-b874-cb6cc5c72cc7" containerName="revoke-keys"
Jul 16 21:29:04.715613 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.715432    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3b723e-50c1-449f-a0ed-a9ced903d369" containerName="revoke-keys"
Jul 16 21:29:04.715613 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.715438    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3b723e-50c1-449f-a0ed-a9ced903d369" containerName="revoke-keys"
Jul 16 21:29:04.715613 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.715444    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3b723e-50c1-449f-a0ed-a9ced903d369" containerName="revoke-keys"
Jul 16 21:29:04.719644 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.719627    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf"
Jul 16 21:29:04.725743 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.725669    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf"]
Jul 16 21:29:04.812830 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.812748    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46812e02-0214-4495-9ea9-497ead646901-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf\" (UID: \"46812e02-0214-4495-9ea9-497ead646901\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf"
Jul 16 21:29:04.913462 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.913428    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46812e02-0214-4495-9ea9-497ead646901-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf\" (UID: \"46812e02-0214-4495-9ea9-497ead646901\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf"
Jul 16 21:29:04.914022 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:04.914004    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46812e02-0214-4495-9ea9-497ead646901-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf\" (UID: \"46812e02-0214-4495-9ea9-497ead646901\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf"
Jul 16 21:29:05.030791 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:05.030758    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf"
Jul 16 21:29:05.163812 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:05.163745    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf"]
Jul 16 21:29:05.166173 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:29:05.166148    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46812e02_0214_4495_9ea9_497ead646901.slice/crio-eff90060eba1f74fb7c393f42e7c8a6543fe2319abd5f3bd1ccf53663a72e59a WatchSource:0}: Error finding container eff90060eba1f74fb7c393f42e7c8a6543fe2319abd5f3bd1ccf53663a72e59a: Status 404 returned error can't find the container with id eff90060eba1f74fb7c393f42e7c8a6543fe2319abd5f3bd1ccf53663a72e59a
Jul 16 21:29:05.490332 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:05.490291    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf" event={"ID":"46812e02-0214-4495-9ea9-497ead646901","Type":"ContainerStarted","Data":"cb2b600b63294dbda0fe55ccae4ee1f43fd7aa1d2a29d217f2724a87486fcc1d"}
Jul 16 21:29:05.490513 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:05.490358    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf" event={"ID":"46812e02-0214-4495-9ea9-497ead646901","Type":"ContainerStarted","Data":"eff90060eba1f74fb7c393f42e7c8a6543fe2319abd5f3bd1ccf53663a72e59a"}
Jul 16 21:29:05.505692 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:05.505645    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf" podStartSLOduration=1.5056245480000001 podStartE2EDuration="1.505624548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:29:05.504398172 +0000 UTC m=+3383.525985938" watchObservedRunningTime="2026-07-16 21:29:05.505624548 +0000 UTC m=+3383.527212314"
Jul 16 21:29:05.609532 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:05.609510    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9331e3-2cfjp"
Jul 16 21:29:05.723259 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:05.723221    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f019e659-03f4-4eea-946a-cc25a4c2e035-maas-api-service-ca\") pod \"f019e659-03f4-4eea-946a-cc25a4c2e035\" (UID: \"f019e659-03f4-4eea-946a-cc25a4c2e035\") "
Jul 16 21:29:05.723957 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:05.723609    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f019e659-03f4-4eea-946a-cc25a4c2e035-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f019e659-03f4-4eea-946a-cc25a4c2e035" (UID: "f019e659-03f4-4eea-946a-cc25a4c2e035"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:29:05.824685 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:05.824620    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f019e659-03f4-4eea-946a-cc25a4c2e035-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:29:06.498312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:06.498280    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9331e3-2cfjp"
Jul 16 21:29:06.498312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:06.498290    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9331e3-2cfjp" event={"ID":"f019e659-03f4-4eea-946a-cc25a4c2e035","Type":"ContainerDied","Data":"2fd9c3bcd62730a76a008aa24e62ccbff7c921d244139eac1bf1460cd82ab731"}
Jul 16 21:29:06.498522 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:06.498368    2530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fd9c3bcd62730a76a008aa24e62ccbff7c921d244139eac1bf1460cd82ab731"
Jul 16 21:29:06.499895 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:06.499866    2530 generic.go:356] "Generic (PLEG): container finished" podID="46812e02-0214-4495-9ea9-497ead646901" containerID="cb2b600b63294dbda0fe55ccae4ee1f43fd7aa1d2a29d217f2724a87486fcc1d" exitCode=6
Jul 16 21:29:06.500035 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:06.499956    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf" event={"ID":"46812e02-0214-4495-9ea9-497ead646901","Type":"ContainerDied","Data":"cb2b600b63294dbda0fe55ccae4ee1f43fd7aa1d2a29d217f2724a87486fcc1d"}
Jul 16 21:29:06.500090 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:06.500078    2530 scope.go:117] "RemoveContainer" containerID="cb2b600b63294dbda0fe55ccae4ee1f43fd7aa1d2a29d217f2724a87486fcc1d"
Jul 16 21:29:06.508683 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:06.508663    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f3b723e-50c1-449f-a0ed-a9ced903d369" path="/var/lib/kubelet/pods/5f3b723e-50c1-449f-a0ed-a9ced903d369/volumes"
Jul 16 21:29:07.506382 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:07.506343    2530 generic.go:356] "Generic (PLEG): container finished" podID="46812e02-0214-4495-9ea9-497ead646901" containerID="dd430faa91b9895cd35ea3dc549cc5c3f7eb22f0be20a1e7ede5bcb35236af2e" exitCode=6
Jul 16 21:29:07.506775 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:07.506417    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf" event={"ID":"46812e02-0214-4495-9ea9-497ead646901","Type":"ContainerDied","Data":"dd430faa91b9895cd35ea3dc549cc5c3f7eb22f0be20a1e7ede5bcb35236af2e"}
Jul 16 21:29:07.506775 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:07.506468    2530 scope.go:117] "RemoveContainer" containerID="cb2b600b63294dbda0fe55ccae4ee1f43fd7aa1d2a29d217f2724a87486fcc1d"
Jul 16 21:29:07.506775 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:07.506665    2530 scope.go:117] "RemoveContainer" containerID="dd430faa91b9895cd35ea3dc549cc5c3f7eb22f0be20a1e7ede5bcb35236af2e"
Jul 16 21:29:07.506913 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:29:07.506884    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf_odh-ai-gateway-infra(46812e02-0214-4495-9ea9-497ead646901)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf" podUID="46812e02-0214-4495-9ea9-497ead646901"
Jul 16 21:29:08.512897 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:08.512869    2530 scope.go:117] "RemoveContainer" containerID="dd430faa91b9895cd35ea3dc549cc5c3f7eb22f0be20a1e7ede5bcb35236af2e"
Jul 16 21:29:08.513311 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:29:08.513032    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf_odh-ai-gateway-infra(46812e02-0214-4495-9ea9-497ead646901)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf" podUID="46812e02-0214-4495-9ea9-497ead646901"
Jul 16 21:29:13.632429 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:13.632398    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"]
Jul 16 21:29:13.632950 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:13.632594    2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5" podUID="4eee1489-6c54-4c80-957f-3de730a58320" containerName="maas-api" containerID="cri-o://bc3c34a7156cc889a503abaf715d1a4f64848ec36520a6cf305bc6e3e842810b" gracePeriod=30
Jul 16 21:29:13.879996 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:13.879970    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"
Jul 16 21:29:13.997477 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:13.997450    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eee1489-6c54-4c80-957f-3de730a58320-openshift-service-ca\") pod \"4eee1489-6c54-4c80-957f-3de730a58320\" (UID: \"4eee1489-6c54-4c80-957f-3de730a58320\") "
Jul 16 21:29:13.997651 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:13.997498    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6wdl\" (UniqueName: \"kubernetes.io/projected/4eee1489-6c54-4c80-957f-3de730a58320-kube-api-access-l6wdl\") pod \"4eee1489-6c54-4c80-957f-3de730a58320\" (UID: \"4eee1489-6c54-4c80-957f-3de730a58320\") "
Jul 16 21:29:13.997651 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:13.997582    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eee1489-6c54-4c80-957f-3de730a58320-maas-api-tls\") pod \"4eee1489-6c54-4c80-957f-3de730a58320\" (UID: \"4eee1489-6c54-4c80-957f-3de730a58320\") "
Jul 16 21:29:13.997797 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:13.997774    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eee1489-6c54-4c80-957f-3de730a58320-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4eee1489-6c54-4c80-957f-3de730a58320" (UID: "4eee1489-6c54-4c80-957f-3de730a58320"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:29:13.999658 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:13.999633    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eee1489-6c54-4c80-957f-3de730a58320-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4eee1489-6c54-4c80-957f-3de730a58320" (UID: "4eee1489-6c54-4c80-957f-3de730a58320"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:29:13.999762 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:13.999745    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eee1489-6c54-4c80-957f-3de730a58320-kube-api-access-l6wdl" (OuterVolumeSpecName: "kube-api-access-l6wdl") pod "4eee1489-6c54-4c80-957f-3de730a58320" (UID: "4eee1489-6c54-4c80-957f-3de730a58320"). InnerVolumeSpecName "kube-api-access-l6wdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:29:14.098205 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:14.098178    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eee1489-6c54-4c80-957f-3de730a58320-maas-api-tls\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:29:14.098205 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:14.098201    2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eee1489-6c54-4c80-957f-3de730a58320-openshift-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:29:14.098348 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:14.098212    2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6wdl\" (UniqueName: \"kubernetes.io/projected/4eee1489-6c54-4c80-957f-3de730a58320-kube-api-access-l6wdl\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:29:14.541073 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:14.541039    2530 generic.go:356] "Generic (PLEG): container finished" podID="4eee1489-6c54-4c80-957f-3de730a58320" containerID="bc3c34a7156cc889a503abaf715d1a4f64848ec36520a6cf305bc6e3e842810b" exitCode=0
Jul 16 21:29:14.541253 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:14.541092    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5" event={"ID":"4eee1489-6c54-4c80-957f-3de730a58320","Type":"ContainerDied","Data":"bc3c34a7156cc889a503abaf715d1a4f64848ec36520a6cf305bc6e3e842810b"}
Jul 16 21:29:14.541253 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:14.541091    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"
Jul 16 21:29:14.541253 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:14.541168    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5" event={"ID":"4eee1489-6c54-4c80-957f-3de730a58320","Type":"ContainerDied","Data":"505d5c5ad0acd81c499d171cec37afcd6b0107984552d0acbac4afe9e69ba58e"}
Jul 16 21:29:14.541253 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:14.541191    2530 scope.go:117] "RemoveContainer" containerID="bc3c34a7156cc889a503abaf715d1a4f64848ec36520a6cf305bc6e3e842810b"
Jul 16 21:29:14.549730 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:14.549715    2530 scope.go:117] "RemoveContainer" containerID="bc3c34a7156cc889a503abaf715d1a4f64848ec36520a6cf305bc6e3e842810b"
Jul 16 21:29:14.549976 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:29:14.549957    2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3c34a7156cc889a503abaf715d1a4f64848ec36520a6cf305bc6e3e842810b\": container with ID starting with bc3c34a7156cc889a503abaf715d1a4f64848ec36520a6cf305bc6e3e842810b not found: ID does not exist" containerID="bc3c34a7156cc889a503abaf715d1a4f64848ec36520a6cf305bc6e3e842810b"
Jul 16 21:29:14.550033 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:14.549984    2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3c34a7156cc889a503abaf715d1a4f64848ec36520a6cf305bc6e3e842810b"} err="failed to get container status \"bc3c34a7156cc889a503abaf715d1a4f64848ec36520a6cf305bc6e3e842810b\": rpc error: code = NotFound desc = could not find container \"bc3c34a7156cc889a503abaf715d1a4f64848ec36520a6cf305bc6e3e842810b\": container with ID starting with bc3c34a7156cc889a503abaf715d1a4f64848ec36520a6cf305bc6e3e842810b not found: ID does not exist"
Jul 16 21:29:14.558374 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:14.558345    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"]
Jul 16 21:29:14.562672 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:14.562654    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9331e3-7bcfddc55d-qknx5"]
Jul 16 21:29:16.508262 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:16.508231    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eee1489-6c54-4c80-957f-3de730a58320" path="/var/lib/kubelet/pods/4eee1489-6c54-4c80-957f-3de730a58320/volumes"
Jul 16 21:29:21.501820 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:21.501792    2530 scope.go:117] "RemoveContainer" containerID="dd430faa91b9895cd35ea3dc549cc5c3f7eb22f0be20a1e7ede5bcb35236af2e"
Jul 16 21:29:22.582038 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:22.581954    2530 generic.go:356] "Generic (PLEG): container finished" podID="46812e02-0214-4495-9ea9-497ead646901" containerID="af91a1db7ad6b789dc3db0534ea7d1b7770e6da6b6aee3b5df827b4b43797045" exitCode=6
Jul 16 21:29:22.582453 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:22.582031    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf" event={"ID":"46812e02-0214-4495-9ea9-497ead646901","Type":"ContainerDied","Data":"af91a1db7ad6b789dc3db0534ea7d1b7770e6da6b6aee3b5df827b4b43797045"}
Jul 16 21:29:22.582453 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:22.582085    2530 scope.go:117] "RemoveContainer" containerID="dd430faa91b9895cd35ea3dc549cc5c3f7eb22f0be20a1e7ede5bcb35236af2e"
Jul 16 21:29:22.582453 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:22.582382    2530 scope.go:117] "RemoveContainer" containerID="af91a1db7ad6b789dc3db0534ea7d1b7770e6da6b6aee3b5df827b4b43797045"
Jul 16 21:29:22.582584 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:29:22.582572    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf_odh-ai-gateway-infra(46812e02-0214-4495-9ea9-497ead646901)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf" podUID="46812e02-0214-4495-9ea9-497ead646901"
Jul 16 21:29:23.607215 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:23.607185    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf"]
Jul 16 21:29:23.738075 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:23.738051    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf"
Jul 16 21:29:23.769605 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:23.769577    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46812e02-0214-4495-9ea9-497ead646901-maas-api-service-ca\") pod \"46812e02-0214-4495-9ea9-497ead646901\" (UID: \"46812e02-0214-4495-9ea9-497ead646901\") "
Jul 16 21:29:23.769908 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:23.769890    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46812e02-0214-4495-9ea9-497ead646901-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "46812e02-0214-4495-9ea9-497ead646901" (UID: "46812e02-0214-4495-9ea9-497ead646901"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:29:23.870904 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:23.870821    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46812e02-0214-4495-9ea9-497ead646901-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:29:24.593752 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:24.593727    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf"
Jul 16 21:29:24.593903 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:24.593750    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf" event={"ID":"46812e02-0214-4495-9ea9-497ead646901","Type":"ContainerDied","Data":"eff90060eba1f74fb7c393f42e7c8a6543fe2319abd5f3bd1ccf53663a72e59a"}
Jul 16 21:29:24.593903 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:24.593802    2530 scope.go:117] "RemoveContainer" containerID="af91a1db7ad6b789dc3db0534ea7d1b7770e6da6b6aee3b5df827b4b43797045"
Jul 16 21:29:24.610727 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:24.610705    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf"]
Jul 16 21:29:24.616799 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:24.616774    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23a5a19c-sksrf"]
Jul 16 21:29:26.508038 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:26.508007    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46812e02-0214-4495-9ea9-497ead646901" path="/var/lib/kubelet/pods/46812e02-0214-4495-9ea9-497ead646901/volumes"
Jul 16 21:29:29.232032 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.231996    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fb5e77-btstx"]
Jul 16 21:29:29.232440 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.232315    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46812e02-0214-4495-9ea9-497ead646901" containerName="revoke-keys"
Jul 16 21:29:29.232440 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.232326    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="46812e02-0214-4495-9ea9-497ead646901" containerName="revoke-keys"
Jul 16 21:29:29.232440 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.232339    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f019e659-03f4-4eea-946a-cc25a4c2e035" containerName="revoke-keys"
Jul 16 21:29:29.232440 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.232344    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="f019e659-03f4-4eea-946a-cc25a4c2e035" containerName="revoke-keys"
Jul 16 21:29:29.232440 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.232357    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3b723e-50c1-449f-a0ed-a9ced903d369" containerName="revoke-keys"
Jul 16 21:29:29.232440 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.232363    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3b723e-50c1-449f-a0ed-a9ced903d369" containerName="revoke-keys"
Jul 16 21:29:29.232440 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.232370    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46812e02-0214-4495-9ea9-497ead646901" containerName="revoke-keys"
Jul 16 21:29:29.232440 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.232375    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="46812e02-0214-4495-9ea9-497ead646901" containerName="revoke-keys"
Jul 16 21:29:29.232440 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.232385    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eee1489-6c54-4c80-957f-3de730a58320" containerName="maas-api"
Jul 16 21:29:29.232440 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.232390    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eee1489-6c54-4c80-957f-3de730a58320" containerName="maas-api"
Jul 16 21:29:29.232440 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.232441    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eee1489-6c54-4c80-957f-3de730a58320" containerName="maas-api"
Jul 16 21:29:29.232777 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.232448    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="46812e02-0214-4495-9ea9-497ead646901" containerName="revoke-keys"
Jul 16 21:29:29.232777 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.232454    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="46812e02-0214-4495-9ea9-497ead646901" containerName="revoke-keys"
Jul 16 21:29:29.232777 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.232462    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="f019e659-03f4-4eea-946a-cc25a4c2e035" containerName="revoke-keys"
Jul 16 21:29:29.236585 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.236558    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fb5e77-btstx"
Jul 16 21:29:29.238784 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.238765    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:29:29.243014 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.242990    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fb5e77-btstx"]
Jul 16 21:29:29.312363 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.312320    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e084b8e-a1b7-4907-bf86-7ff4696ac8e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-fb5e77-btstx\" (UID: \"7e084b8e-a1b7-4907-bf86-7ff4696ac8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fb5e77-btstx"
Jul 16 21:29:29.413213 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.413158    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e084b8e-a1b7-4907-bf86-7ff4696ac8e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-fb5e77-btstx\" (UID: \"7e084b8e-a1b7-4907-bf86-7ff4696ac8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fb5e77-btstx"
Jul 16 21:29:29.413810 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.413789    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e084b8e-a1b7-4907-bf86-7ff4696ac8e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-fb5e77-btstx\" (UID: \"7e084b8e-a1b7-4907-bf86-7ff4696ac8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fb5e77-btstx"
Jul 16 21:29:29.548197 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.548093    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fb5e77-btstx"
Jul 16 21:29:29.681906 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.681879    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fb5e77-btstx"]
Jul 16 21:29:29.683616 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:29:29.683584    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e084b8e_a1b7_4907_bf86_7ff4696ac8e7.slice/crio-49a2538a2e1d3f7abdd4ad707565aa3ead835299657ba96f28068dbc582353b3 WatchSource:0}: Error finding container 49a2538a2e1d3f7abdd4ad707565aa3ead835299657ba96f28068dbc582353b3: Status 404 returned error can't find the container with id 49a2538a2e1d3f7abdd4ad707565aa3ead835299657ba96f28068dbc582353b3
Jul 16 21:29:29.893066 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.892991    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j"]
Jul 16 21:29:29.893352 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.893338    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46812e02-0214-4495-9ea9-497ead646901" containerName="revoke-keys"
Jul 16 21:29:29.893352 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.893353    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="46812e02-0214-4495-9ea9-497ead646901" containerName="revoke-keys"
Jul 16 21:29:29.893447 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.893416    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="46812e02-0214-4495-9ea9-497ead646901" containerName="revoke-keys"
Jul 16 21:29:29.897405 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.897382    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j"
Jul 16 21:29:29.903176 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.903153    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j"]
Jul 16 21:29:29.918337 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:29.918307    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d1b2a2-0f59-4416-a252-98c84366c25f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j\" (UID: \"81d1b2a2-0f59-4416-a252-98c84366c25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j"
Jul 16 21:29:30.018970 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:30.018938    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d1b2a2-0f59-4416-a252-98c84366c25f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j\" (UID: \"81d1b2a2-0f59-4416-a252-98c84366c25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j"
Jul 16 21:29:30.019580 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:30.019557    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d1b2a2-0f59-4416-a252-98c84366c25f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j\" (UID: \"81d1b2a2-0f59-4416-a252-98c84366c25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j"
Jul 16 21:29:30.208468 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:30.208439    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j"
Jul 16 21:29:30.341200 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:30.341177    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j"]
Jul 16 21:29:30.342984 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:29:30.342957    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d1b2a2_0f59_4416_a252_98c84366c25f.slice/crio-55e2962378fa1ce5c60412e2e01dbef348cdd1503eac6e1ce7a49780a06d3cf8 WatchSource:0}: Error finding container 55e2962378fa1ce5c60412e2e01dbef348cdd1503eac6e1ce7a49780a06d3cf8: Status 404 returned error can't find the container with id 55e2962378fa1ce5c60412e2e01dbef348cdd1503eac6e1ce7a49780a06d3cf8
Jul 16 21:29:30.622537 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:30.622500    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j" event={"ID":"81d1b2a2-0f59-4416-a252-98c84366c25f","Type":"ContainerStarted","Data":"945ccb7e508dad92449b47a456585efa69854fb60dbf4dd7dc5e2045f711b99a"}
Jul 16 21:29:30.622705 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:30.622563    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j" event={"ID":"81d1b2a2-0f59-4416-a252-98c84366c25f","Type":"ContainerStarted","Data":"55e2962378fa1ce5c60412e2e01dbef348cdd1503eac6e1ce7a49780a06d3cf8"}
Jul 16 21:29:30.624029 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:30.624004    2530 generic.go:356] "Generic (PLEG): container finished" podID="7e084b8e-a1b7-4907-bf86-7ff4696ac8e7" containerID="8786e908d72f6bed95f11354a6047fa2c41bcc9fa19af6f01477751e98aa0e96" exitCode=0
Jul 16 21:29:30.624169 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:30.624059    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fb5e77-btstx" event={"ID":"7e084b8e-a1b7-4907-bf86-7ff4696ac8e7","Type":"ContainerDied","Data":"8786e908d72f6bed95f11354a6047fa2c41bcc9fa19af6f01477751e98aa0e96"}
Jul 16 21:29:30.624169 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:30.624116    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fb5e77-btstx" event={"ID":"7e084b8e-a1b7-4907-bf86-7ff4696ac8e7","Type":"ContainerStarted","Data":"49a2538a2e1d3f7abdd4ad707565aa3ead835299657ba96f28068dbc582353b3"}
Jul 16 21:29:30.639244 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:30.639191    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j" podStartSLOduration=1.6391772009999999 podStartE2EDuration="1.639177201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:29:30.637255824 +0000 UTC m=+3408.658843591" watchObservedRunningTime="2026-07-16 21:29:30.639177201 +0000 UTC m=+3408.660765006"
Jul 16 21:29:31.629980 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:31.629946    2530 generic.go:356] "Generic (PLEG): container finished" podID="81d1b2a2-0f59-4416-a252-98c84366c25f" containerID="945ccb7e508dad92449b47a456585efa69854fb60dbf4dd7dc5e2045f711b99a" exitCode=6
Jul 16 21:29:31.630439 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:31.630035    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j" event={"ID":"81d1b2a2-0f59-4416-a252-98c84366c25f","Type":"ContainerDied","Data":"945ccb7e508dad92449b47a456585efa69854fb60dbf4dd7dc5e2045f711b99a"}
Jul 16 21:29:31.630439 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:31.630260    2530 scope.go:117] "RemoveContainer" containerID="945ccb7e508dad92449b47a456585efa69854fb60dbf4dd7dc5e2045f711b99a"
Jul 16 21:29:31.762308 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:31.762283    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fb5e77-btstx"
Jul 16 21:29:31.833599 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:31.833576    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e084b8e-a1b7-4907-bf86-7ff4696ac8e7-maas-api-service-ca\") pod \"7e084b8e-a1b7-4907-bf86-7ff4696ac8e7\" (UID: \"7e084b8e-a1b7-4907-bf86-7ff4696ac8e7\") "
Jul 16 21:29:31.833970 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:31.833946    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e084b8e-a1b7-4907-bf86-7ff4696ac8e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e084b8e-a1b7-4907-bf86-7ff4696ac8e7" (UID: "7e084b8e-a1b7-4907-bf86-7ff4696ac8e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:29:31.935013 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:31.934984    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e084b8e-a1b7-4907-bf86-7ff4696ac8e7-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:29:32.636427 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:32.636329    2530 generic.go:356] "Generic (PLEG): container finished" podID="81d1b2a2-0f59-4416-a252-98c84366c25f" containerID="cb0f3037ce947e39a95247a1e834fa80792e6c2bb3ad3d09ec8df07b1f2651be" exitCode=6
Jul 16 21:29:32.636902 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:32.636418    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j" event={"ID":"81d1b2a2-0f59-4416-a252-98c84366c25f","Type":"ContainerDied","Data":"cb0f3037ce947e39a95247a1e834fa80792e6c2bb3ad3d09ec8df07b1f2651be"}
Jul 16 21:29:32.636902 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:32.636524    2530 scope.go:117] "RemoveContainer" containerID="945ccb7e508dad92449b47a456585efa69854fb60dbf4dd7dc5e2045f711b99a"
Jul 16 21:29:32.636902 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:32.636645    2530 scope.go:117] "RemoveContainer" containerID="cb0f3037ce947e39a95247a1e834fa80792e6c2bb3ad3d09ec8df07b1f2651be"
Jul 16 21:29:32.636902 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:29:32.636874    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j_odh-ai-gateway-infra(81d1b2a2-0f59-4416-a252-98c84366c25f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j" podUID="81d1b2a2-0f59-4416-a252-98c84366c25f"
Jul 16 21:29:32.637851 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:32.637825    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fb5e77-btstx"
Jul 16 21:29:32.637851 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:32.637827    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fb5e77-btstx" event={"ID":"7e084b8e-a1b7-4907-bf86-7ff4696ac8e7","Type":"ContainerDied","Data":"49a2538a2e1d3f7abdd4ad707565aa3ead835299657ba96f28068dbc582353b3"}
Jul 16 21:29:32.637957 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:32.637866    2530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49a2538a2e1d3f7abdd4ad707565aa3ead835299657ba96f28068dbc582353b3"
Jul 16 21:29:33.002367 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:33.002336    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8"]
Jul 16 21:29:33.002687 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:33.002675    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e084b8e-a1b7-4907-bf86-7ff4696ac8e7" containerName="revoke-keys"
Jul 16 21:29:33.002734 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:33.002689    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e084b8e-a1b7-4907-bf86-7ff4696ac8e7" containerName="revoke-keys"
Jul 16 21:29:33.002773 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:33.002742    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e084b8e-a1b7-4907-bf86-7ff4696ac8e7" containerName="revoke-keys"
Jul 16 21:29:33.007005 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:33.006984    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8"
Jul 16 21:29:33.015740 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:33.015666    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8"]
Jul 16 21:29:33.044806 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:33.044773    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7643778a-00eb-4f04-8193-4ab704b1f4c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8\" (UID: \"7643778a-00eb-4f04-8193-4ab704b1f4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8"
Jul 16 21:29:33.145739 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:33.145712    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7643778a-00eb-4f04-8193-4ab704b1f4c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8\" (UID: \"7643778a-00eb-4f04-8193-4ab704b1f4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8"
Jul 16 21:29:33.146330 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:33.146311    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7643778a-00eb-4f04-8193-4ab704b1f4c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8\" (UID: \"7643778a-00eb-4f04-8193-4ab704b1f4c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8"
Jul 16 21:29:33.318248 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:33.318143    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8"
Jul 16 21:29:33.644941 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:33.644877    2530 scope.go:117] "RemoveContainer" containerID="cb0f3037ce947e39a95247a1e834fa80792e6c2bb3ad3d09ec8df07b1f2651be"
Jul 16 21:29:33.645321 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:29:33.645068    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j_odh-ai-gateway-infra(81d1b2a2-0f59-4416-a252-98c84366c25f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j" podUID="81d1b2a2-0f59-4416-a252-98c84366c25f"
Jul 16 21:29:33.657979 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:33.657879    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8"]
Jul 16 21:29:33.660346 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:29:33.660316    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7643778a_00eb_4f04_8193_4ab704b1f4c3.slice/crio-53859c6a4a33baabd7c508c69b11d331d984057496b9109d18dc6d28cd8104c1 WatchSource:0}: Error finding container 53859c6a4a33baabd7c508c69b11d331d984057496b9109d18dc6d28cd8104c1: Status 404 returned error can't find the container with id 53859c6a4a33baabd7c508c69b11d331d984057496b9109d18dc6d28cd8104c1
Jul 16 21:29:34.650525 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:34.650426    2530 generic.go:356] "Generic (PLEG): container finished" podID="7643778a-00eb-4f04-8193-4ab704b1f4c3" containerID="2a2f086eab4b803ab4069e4504b715f7ba4503285668c9f196c99c4e777bbf4f" exitCode=6
Jul 16 21:29:34.650525 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:34.650511    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8" event={"ID":"7643778a-00eb-4f04-8193-4ab704b1f4c3","Type":"ContainerDied","Data":"2a2f086eab4b803ab4069e4504b715f7ba4503285668c9f196c99c4e777bbf4f"}
Jul 16 21:29:34.650883 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:34.650564    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8" event={"ID":"7643778a-00eb-4f04-8193-4ab704b1f4c3","Type":"ContainerStarted","Data":"53859c6a4a33baabd7c508c69b11d331d984057496b9109d18dc6d28cd8104c1"}
Jul 16 21:29:34.650883 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:34.650653    2530 scope.go:117] "RemoveContainer" containerID="2a2f086eab4b803ab4069e4504b715f7ba4503285668c9f196c99c4e777bbf4f"
Jul 16 21:29:35.657259 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:35.657173    2530 generic.go:356] "Generic (PLEG): container finished" podID="7643778a-00eb-4f04-8193-4ab704b1f4c3" containerID="cbe7c2a00cf0bc6aff5f53d9ac01b293d6ae362a224d143c2a85634f43e7b9a2" exitCode=6
Jul 16 21:29:35.657702 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:35.657259    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8" event={"ID":"7643778a-00eb-4f04-8193-4ab704b1f4c3","Type":"ContainerDied","Data":"cbe7c2a00cf0bc6aff5f53d9ac01b293d6ae362a224d143c2a85634f43e7b9a2"}
Jul 16 21:29:35.657702 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:35.657325    2530 scope.go:117] "RemoveContainer" containerID="2a2f086eab4b803ab4069e4504b715f7ba4503285668c9f196c99c4e777bbf4f"
Jul 16 21:29:35.657702 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:35.657472    2530 scope.go:117] "RemoveContainer" containerID="cbe7c2a00cf0bc6aff5f53d9ac01b293d6ae362a224d143c2a85634f43e7b9a2"
Jul 16 21:29:35.657702 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:29:35.657699    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8_odh-ai-gateway-infra(7643778a-00eb-4f04-8193-4ab704b1f4c3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8" podUID="7643778a-00eb-4f04-8193-4ab704b1f4c3"
Jul 16 21:29:36.665853 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:36.665825    2530 scope.go:117] "RemoveContainer" containerID="cbe7c2a00cf0bc6aff5f53d9ac01b293d6ae362a224d143c2a85634f43e7b9a2"
Jul 16 21:29:36.666244 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:29:36.666029    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8_odh-ai-gateway-infra(7643778a-00eb-4f04-8193-4ab704b1f4c3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8" podUID="7643778a-00eb-4f04-8193-4ab704b1f4c3"
Jul 16 21:29:39.785146 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:39.785092    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"]
Jul 16 21:29:39.785562 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:39.785377    2530 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg" podUID="7ab39e48-0068-48e4-8441-d5b4a000e9ff" containerName="maas-api" containerID="cri-o://cef5ab6df351fff02b22b198797be18c2cdd0d7d389f67faa3554ff94d4e226a" gracePeriod=30
Jul 16 21:29:40.033690 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.033663    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"
Jul 16 21:29:40.103939 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.103856    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab39e48-0068-48e4-8441-d5b4a000e9ff-maas-api-tls\") pod \"7ab39e48-0068-48e4-8441-d5b4a000e9ff\" (UID: \"7ab39e48-0068-48e4-8441-d5b4a000e9ff\") "
Jul 16 21:29:40.103939 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.103935    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ckn5\" (UniqueName: \"kubernetes.io/projected/7ab39e48-0068-48e4-8441-d5b4a000e9ff-kube-api-access-8ckn5\") pod \"7ab39e48-0068-48e4-8441-d5b4a000e9ff\" (UID: \"7ab39e48-0068-48e4-8441-d5b4a000e9ff\") "
Jul 16 21:29:40.104175 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.104040    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab39e48-0068-48e4-8441-d5b4a000e9ff-openshift-service-ca\") pod \"7ab39e48-0068-48e4-8441-d5b4a000e9ff\" (UID: \"7ab39e48-0068-48e4-8441-d5b4a000e9ff\") "
Jul 16 21:29:40.104424 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.104396    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab39e48-0068-48e4-8441-d5b4a000e9ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ab39e48-0068-48e4-8441-d5b4a000e9ff" (UID: "7ab39e48-0068-48e4-8441-d5b4a000e9ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:29:40.106207 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.106177    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab39e48-0068-48e4-8441-d5b4a000e9ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ab39e48-0068-48e4-8441-d5b4a000e9ff" (UID: "7ab39e48-0068-48e4-8441-d5b4a000e9ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:29:40.106295 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.106221    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab39e48-0068-48e4-8441-d5b4a000e9ff-kube-api-access-8ckn5" (OuterVolumeSpecName: "kube-api-access-8ckn5") pod "7ab39e48-0068-48e4-8441-d5b4a000e9ff" (UID: "7ab39e48-0068-48e4-8441-d5b4a000e9ff"). InnerVolumeSpecName "kube-api-access-8ckn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:29:40.204947 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.204909    2530 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ckn5\" (UniqueName: \"kubernetes.io/projected/7ab39e48-0068-48e4-8441-d5b4a000e9ff-kube-api-access-8ckn5\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:29:40.204947 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.204941    2530 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab39e48-0068-48e4-8441-d5b4a000e9ff-openshift-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:29:40.204947 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.204952    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab39e48-0068-48e4-8441-d5b4a000e9ff-maas-api-tls\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:29:40.686297 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.686258    2530 generic.go:356] "Generic (PLEG): container finished" podID="7ab39e48-0068-48e4-8441-d5b4a000e9ff" containerID="cef5ab6df351fff02b22b198797be18c2cdd0d7d389f67faa3554ff94d4e226a" exitCode=0
Jul 16 21:29:40.686482 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.686306    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"
Jul 16 21:29:40.686482 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.686346    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg" event={"ID":"7ab39e48-0068-48e4-8441-d5b4a000e9ff","Type":"ContainerDied","Data":"cef5ab6df351fff02b22b198797be18c2cdd0d7d389f67faa3554ff94d4e226a"}
Jul 16 21:29:40.686482 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.686400    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg" event={"ID":"7ab39e48-0068-48e4-8441-d5b4a000e9ff","Type":"ContainerDied","Data":"f787ecf4ad4d7ccd44cd17e5f42b0c45c8234ace4777b61851e4dd843807cf80"}
Jul 16 21:29:40.686482 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.686423    2530 scope.go:117] "RemoveContainer" containerID="cef5ab6df351fff02b22b198797be18c2cdd0d7d389f67faa3554ff94d4e226a"
Jul 16 21:29:40.695161 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.695141    2530 scope.go:117] "RemoveContainer" containerID="cef5ab6df351fff02b22b198797be18c2cdd0d7d389f67faa3554ff94d4e226a"
Jul 16 21:29:40.695443 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:29:40.695421    2530 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef5ab6df351fff02b22b198797be18c2cdd0d7d389f67faa3554ff94d4e226a\": container with ID starting with cef5ab6df351fff02b22b198797be18c2cdd0d7d389f67faa3554ff94d4e226a not found: ID does not exist" containerID="cef5ab6df351fff02b22b198797be18c2cdd0d7d389f67faa3554ff94d4e226a"
Jul 16 21:29:40.695493 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.695455    2530 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef5ab6df351fff02b22b198797be18c2cdd0d7d389f67faa3554ff94d4e226a"} err="failed to get container status \"cef5ab6df351fff02b22b198797be18c2cdd0d7d389f67faa3554ff94d4e226a\": rpc error: code = NotFound desc = could not find container \"cef5ab6df351fff02b22b198797be18c2cdd0d7d389f67faa3554ff94d4e226a\": container with ID starting with cef5ab6df351fff02b22b198797be18c2cdd0d7d389f67faa3554ff94d4e226a not found: ID does not exist"
Jul 16 21:29:40.704289 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.704265    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"]
Jul 16 21:29:40.707962 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:40.707936    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fb5e77-6bf94765c7-mlqxg"]
Jul 16 21:29:42.507758 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:42.507725    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ab39e48-0068-48e4-8441-d5b4a000e9ff" path="/var/lib/kubelet/pods/7ab39e48-0068-48e4-8441-d5b4a000e9ff/volumes"
Jul 16 21:29:42.513584 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:42.513560    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk"]
Jul 16 21:29:42.513860 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:42.513849    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ab39e48-0068-48e4-8441-d5b4a000e9ff" containerName="maas-api"
Jul 16 21:29:42.513907 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:42.513861    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ab39e48-0068-48e4-8441-d5b4a000e9ff" containerName="maas-api"
Jul 16 21:29:42.513943 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:42.513929    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ab39e48-0068-48e4-8441-d5b4a000e9ff" containerName="maas-api"
Jul 16 21:29:42.518203 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:42.518187    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk"
Jul 16 21:29:42.522843 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:42.522818    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk"]
Jul 16 21:29:42.624770 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:42.624737    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82851b9e-2b24-4095-ba87-cffae2319b62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk\" (UID: \"82851b9e-2b24-4095-ba87-cffae2319b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk"
Jul 16 21:29:42.725411 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:42.725381    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82851b9e-2b24-4095-ba87-cffae2319b62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk\" (UID: \"82851b9e-2b24-4095-ba87-cffae2319b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk"
Jul 16 21:29:42.726014 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:42.725995    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82851b9e-2b24-4095-ba87-cffae2319b62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk\" (UID: \"82851b9e-2b24-4095-ba87-cffae2319b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk"
Jul 16 21:29:42.829420 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:42.829359    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk"
Jul 16 21:29:42.954479 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:42.954445    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk"]
Jul 16 21:29:42.957526 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:29:42.957503    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82851b9e_2b24_4095_ba87_cffae2319b62.slice/crio-f73f2c9b55bf282b017bd13a7b37edabc2ea9e222cf382c6050afdad42761696 WatchSource:0}: Error finding container f73f2c9b55bf282b017bd13a7b37edabc2ea9e222cf382c6050afdad42761696: Status 404 returned error can't find the container with id f73f2c9b55bf282b017bd13a7b37edabc2ea9e222cf382c6050afdad42761696
Jul 16 21:29:43.704080 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:43.704045    2530 generic.go:356] "Generic (PLEG): container finished" podID="82851b9e-2b24-4095-ba87-cffae2319b62" containerID="783d5ed19b30b5a8579285e19ab03bc15083a0120f82c04d11ebf702a9fdc2d1" exitCode=6
Jul 16 21:29:43.704495 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:43.704136    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk" event={"ID":"82851b9e-2b24-4095-ba87-cffae2319b62","Type":"ContainerDied","Data":"783d5ed19b30b5a8579285e19ab03bc15083a0120f82c04d11ebf702a9fdc2d1"}
Jul 16 21:29:43.704495 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:43.704248    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk" event={"ID":"82851b9e-2b24-4095-ba87-cffae2319b62","Type":"ContainerStarted","Data":"f73f2c9b55bf282b017bd13a7b37edabc2ea9e222cf382c6050afdad42761696"}
Jul 16 21:29:43.704495 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:43.704326    2530 scope.go:117] "RemoveContainer" containerID="783d5ed19b30b5a8579285e19ab03bc15083a0120f82c04d11ebf702a9fdc2d1"
Jul 16 21:29:44.502033 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:44.502007    2530 scope.go:117] "RemoveContainer" containerID="cb0f3037ce947e39a95247a1e834fa80792e6c2bb3ad3d09ec8df07b1f2651be"
Jul 16 21:29:44.709861 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:44.709835    2530 generic.go:356] "Generic (PLEG): container finished" podID="82851b9e-2b24-4095-ba87-cffae2319b62" containerID="a399972a067a779cddcfd199582a494d0fa2151190aa49c376e12d09e180bfdd" exitCode=6
Jul 16 21:29:44.710250 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:44.709926    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk" event={"ID":"82851b9e-2b24-4095-ba87-cffae2319b62","Type":"ContainerDied","Data":"a399972a067a779cddcfd199582a494d0fa2151190aa49c376e12d09e180bfdd"}
Jul 16 21:29:44.710250 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:44.709977    2530 scope.go:117] "RemoveContainer" containerID="783d5ed19b30b5a8579285e19ab03bc15083a0120f82c04d11ebf702a9fdc2d1"
Jul 16 21:29:44.710250 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:44.710066    2530 scope.go:117] "RemoveContainer" containerID="a399972a067a779cddcfd199582a494d0fa2151190aa49c376e12d09e180bfdd"
Jul 16 21:29:44.710370 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:29:44.710263    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk_odh-ai-gateway-infra(82851b9e-2b24-4095-ba87-cffae2319b62)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk" podUID="82851b9e-2b24-4095-ba87-cffae2319b62"
Jul 16 21:29:45.716091 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:45.716064    2530 scope.go:117] "RemoveContainer" containerID="a399972a067a779cddcfd199582a494d0fa2151190aa49c376e12d09e180bfdd"
Jul 16 21:29:45.716557 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:29:45.716257    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk_odh-ai-gateway-infra(82851b9e-2b24-4095-ba87-cffae2319b62)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk" podUID="82851b9e-2b24-4095-ba87-cffae2319b62"
Jul 16 21:29:45.717704 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:45.717681    2530 generic.go:356] "Generic (PLEG): container finished" podID="81d1b2a2-0f59-4416-a252-98c84366c25f" containerID="f6970bc5a7a6f9579732126ac070139695e5904613bfd82ab8d3b97a62877062" exitCode=6
Jul 16 21:29:45.717840 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:45.717757    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j" event={"ID":"81d1b2a2-0f59-4416-a252-98c84366c25f","Type":"ContainerDied","Data":"f6970bc5a7a6f9579732126ac070139695e5904613bfd82ab8d3b97a62877062"}
Jul 16 21:29:45.717840 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:45.717808    2530 scope.go:117] "RemoveContainer" containerID="cb0f3037ce947e39a95247a1e834fa80792e6c2bb3ad3d09ec8df07b1f2651be"
Jul 16 21:29:45.718041 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:45.718023    2530 scope.go:117] "RemoveContainer" containerID="f6970bc5a7a6f9579732126ac070139695e5904613bfd82ab8d3b97a62877062"
Jul 16 21:29:45.718244 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:29:45.718222    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j_odh-ai-gateway-infra(81d1b2a2-0f59-4416-a252-98c84366c25f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j" podUID="81d1b2a2-0f59-4416-a252-98c84366c25f"
Jul 16 21:29:46.760794 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:46.760755    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j"]
Jul 16 21:29:46.894836 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:46.894813    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j"
Jul 16 21:29:46.961297 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:46.961265    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d1b2a2-0f59-4416-a252-98c84366c25f-maas-api-service-ca\") pod \"81d1b2a2-0f59-4416-a252-98c84366c25f\" (UID: \"81d1b2a2-0f59-4416-a252-98c84366c25f\") "
Jul 16 21:29:46.961582 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:46.961561    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d1b2a2-0f59-4416-a252-98c84366c25f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "81d1b2a2-0f59-4416-a252-98c84366c25f" (UID: "81d1b2a2-0f59-4416-a252-98c84366c25f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:29:47.061735 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:47.061677    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d1b2a2-0f59-4416-a252-98c84366c25f-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:29:47.730094 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:47.730066    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j"
Jul 16 21:29:47.730289 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:47.730071    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j" event={"ID":"81d1b2a2-0f59-4416-a252-98c84366c25f","Type":"ContainerDied","Data":"55e2962378fa1ce5c60412e2e01dbef348cdd1503eac6e1ce7a49780a06d3cf8"}
Jul 16 21:29:47.730289 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:47.730206    2530 scope.go:117] "RemoveContainer" containerID="f6970bc5a7a6f9579732126ac070139695e5904613bfd82ab8d3b97a62877062"
Jul 16 21:29:47.753061 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:47.753040    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j"]
Jul 16 21:29:47.760254 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:47.760234    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-7pb7j"]
Jul 16 21:29:48.513212 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:48.508022    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81d1b2a2-0f59-4416-a252-98c84366c25f" path="/var/lib/kubelet/pods/81d1b2a2-0f59-4416-a252-98c84366c25f/volumes"
Jul 16 21:29:50.502429 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:50.502400    2530 scope.go:117] "RemoveContainer" containerID="cbe7c2a00cf0bc6aff5f53d9ac01b293d6ae362a224d143c2a85634f43e7b9a2"
Jul 16 21:29:50.746132 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:50.746094    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8" event={"ID":"7643778a-00eb-4f04-8193-4ab704b1f4c3","Type":"ContainerStarted","Data":"90c46722aee786eb9006bf38a0890a0ea2c8a01ec7c7f9241765b10b395c2073"}
Jul 16 21:29:50.763642 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:50.763560    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8" podStartSLOduration=18.763546198 podStartE2EDuration="18.763546198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:29:50.761215295 +0000 UTC m=+3428.782803062" watchObservedRunningTime="2026-07-16 21:29:50.763546198 +0000 UTC m=+3428.785133964"
Jul 16 21:29:51.752919 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:51.752887    2530 generic.go:356] "Generic (PLEG): container finished" podID="7643778a-00eb-4f04-8193-4ab704b1f4c3" containerID="90c46722aee786eb9006bf38a0890a0ea2c8a01ec7c7f9241765b10b395c2073" exitCode=6
Jul 16 21:29:51.753376 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:51.752957    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8" event={"ID":"7643778a-00eb-4f04-8193-4ab704b1f4c3","Type":"ContainerDied","Data":"90c46722aee786eb9006bf38a0890a0ea2c8a01ec7c7f9241765b10b395c2073"}
Jul 16 21:29:51.753376 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:51.753023    2530 scope.go:117] "RemoveContainer" containerID="cbe7c2a00cf0bc6aff5f53d9ac01b293d6ae362a224d143c2a85634f43e7b9a2"
Jul 16 21:29:51.753376 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:51.753356    2530 scope.go:117] "RemoveContainer" containerID="90c46722aee786eb9006bf38a0890a0ea2c8a01ec7c7f9241765b10b395c2073"
Jul 16 21:29:51.753597 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:29:51.753575    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8_odh-ai-gateway-infra(7643778a-00eb-4f04-8193-4ab704b1f4c3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8" podUID="7643778a-00eb-4f04-8193-4ab704b1f4c3"
Jul 16 21:29:51.773308 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:51.773272    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8"]
Jul 16 21:29:52.891621 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:52.891599    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8"
Jul 16 21:29:53.008767 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:53.008732    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7643778a-00eb-4f04-8193-4ab704b1f4c3-maas-api-service-ca\") pod \"7643778a-00eb-4f04-8193-4ab704b1f4c3\" (UID: \"7643778a-00eb-4f04-8193-4ab704b1f4c3\") "
Jul 16 21:29:53.009136 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:53.009085    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7643778a-00eb-4f04-8193-4ab704b1f4c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7643778a-00eb-4f04-8193-4ab704b1f4c3" (UID: "7643778a-00eb-4f04-8193-4ab704b1f4c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:29:53.109830 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:53.109756    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7643778a-00eb-4f04-8193-4ab704b1f4c3-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:29:53.765312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:53.765275    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8"
Jul 16 21:29:53.765312 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:53.765302    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8" event={"ID":"7643778a-00eb-4f04-8193-4ab704b1f4c3","Type":"ContainerDied","Data":"53859c6a4a33baabd7c508c69b11d331d984057496b9109d18dc6d28cd8104c1"}
Jul 16 21:29:53.765590 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:53.765361    2530 scope.go:117] "RemoveContainer" containerID="90c46722aee786eb9006bf38a0890a0ea2c8a01ec7c7f9241765b10b395c2073"
Jul 16 21:29:53.789009 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:53.788980    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8"]
Jul 16 21:29:53.795792 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:53.795773    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-k82t8"]
Jul 16 21:29:54.507515 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:29:54.507479    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7643778a-00eb-4f04-8193-4ab704b1f4c3" path="/var/lib/kubelet/pods/7643778a-00eb-4f04-8193-4ab704b1f4c3/volumes"
Jul 16 21:30:00.143805 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.143765    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737290-254sh"]
Jul 16 21:30:00.144217 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.144146    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7643778a-00eb-4f04-8193-4ab704b1f4c3" containerName="revoke-keys"
Jul 16 21:30:00.144217 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.144163    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="7643778a-00eb-4f04-8193-4ab704b1f4c3" containerName="revoke-keys"
Jul 16 21:30:00.144217 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.144172    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81d1b2a2-0f59-4416-a252-98c84366c25f" containerName="revoke-keys"
Jul 16 21:30:00.144217 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.144177    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="81d1b2a2-0f59-4416-a252-98c84366c25f" containerName="revoke-keys"
Jul 16 21:30:00.144217 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.144187    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7643778a-00eb-4f04-8193-4ab704b1f4c3" containerName="revoke-keys"
Jul 16 21:30:00.144217 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.144193    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="7643778a-00eb-4f04-8193-4ab704b1f4c3" containerName="revoke-keys"
Jul 16 21:30:00.144217 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.144200    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81d1b2a2-0f59-4416-a252-98c84366c25f" containerName="revoke-keys"
Jul 16 21:30:00.144217 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.144205    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="81d1b2a2-0f59-4416-a252-98c84366c25f" containerName="revoke-keys"
Jul 16 21:30:00.144217 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.144212    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81d1b2a2-0f59-4416-a252-98c84366c25f" containerName="revoke-keys"
Jul 16 21:30:00.144217 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.144217    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="81d1b2a2-0f59-4416-a252-98c84366c25f" containerName="revoke-keys"
Jul 16 21:30:00.144534 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.144281    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="7643778a-00eb-4f04-8193-4ab704b1f4c3" containerName="revoke-keys"
Jul 16 21:30:00.144534 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.144289    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="7643778a-00eb-4f04-8193-4ab704b1f4c3" containerName="revoke-keys"
Jul 16 21:30:00.144534 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.144296    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="7643778a-00eb-4f04-8193-4ab704b1f4c3" containerName="revoke-keys"
Jul 16 21:30:00.144534 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.144303    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="81d1b2a2-0f59-4416-a252-98c84366c25f" containerName="revoke-keys"
Jul 16 21:30:00.144534 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.144310    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="81d1b2a2-0f59-4416-a252-98c84366c25f" containerName="revoke-keys"
Jul 16 21:30:00.148454 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.148438    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737290-254sh"
Jul 16 21:30:00.155598 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.155573    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737290-254sh"]
Jul 16 21:30:00.160588 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.160567    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737290-254sh"
Jul 16 21:30:00.497040 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.497016    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737290-254sh"]
Jul 16 21:30:00.499201 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:30:00.499173    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd0b800_6773_4538_8a4a_17b0a86e1388.slice/crio-35cdc91d9f7775c746cf5753c02d5f06f81cdc46a00425ecc38127b9840a6567 WatchSource:0}: Error finding container 35cdc91d9f7775c746cf5753c02d5f06f81cdc46a00425ecc38127b9840a6567: Status 404 returned error can't find the container with id 35cdc91d9f7775c746cf5753c02d5f06f81cdc46a00425ecc38127b9840a6567
Jul 16 21:30:00.501900 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.501880    2530 scope.go:117] "RemoveContainer" containerID="a399972a067a779cddcfd199582a494d0fa2151190aa49c376e12d09e180bfdd"
Jul 16 21:30:00.808181 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:00.808084    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737290-254sh" event={"ID":"5cd0b800-6773-4538-8a4a-17b0a86e1388","Type":"ContainerStarted","Data":"35cdc91d9f7775c746cf5753c02d5f06f81cdc46a00425ecc38127b9840a6567"}
Jul 16 21:30:01.338017 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:01.337992    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-tkrwz_88af9955-7488-44ce-a8a4-2589b1b302ae/manager/0.log"
Jul 16 21:30:01.466397 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:01.466370    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-sdzsg_a8f11bc3-e9ed-4832-b76f-94e5e31522dd/manager/0.log"
Jul 16 21:30:01.814287 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:01.814252    2530 generic.go:356] "Generic (PLEG): container finished" podID="82851b9e-2b24-4095-ba87-cffae2319b62" containerID="e7fb443f2b1ce9715f168ba5f3a603c918815220c4ca3c5524f669ccaccfed28" exitCode=6
Jul 16 21:30:01.814443 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:01.814322    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk" event={"ID":"82851b9e-2b24-4095-ba87-cffae2319b62","Type":"ContainerDied","Data":"e7fb443f2b1ce9715f168ba5f3a603c918815220c4ca3c5524f669ccaccfed28"}
Jul 16 21:30:01.814443 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:01.814380    2530 scope.go:117] "RemoveContainer" containerID="a399972a067a779cddcfd199582a494d0fa2151190aa49c376e12d09e180bfdd"
Jul 16 21:30:01.814689 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:01.814674    2530 scope.go:117] "RemoveContainer" containerID="e7fb443f2b1ce9715f168ba5f3a603c918815220c4ca3c5524f669ccaccfed28"
Jul 16 21:30:01.814901 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:30:01.814869    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk_odh-ai-gateway-infra(82851b9e-2b24-4095-ba87-cffae2319b62)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk" podUID="82851b9e-2b24-4095-ba87-cffae2319b62"
Jul 16 21:30:01.815996 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:01.815978    2530 generic.go:356] "Generic (PLEG): container finished" podID="5cd0b800-6773-4538-8a4a-17b0a86e1388" containerID="3ce85a7795ee67eff74ed51a07dd7f857c04102a65c5e9851a22587932d1bbb7" exitCode=0
Jul 16 21:30:01.816127 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:01.816054    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737290-254sh" event={"ID":"5cd0b800-6773-4538-8a4a-17b0a86e1388","Type":"ContainerDied","Data":"3ce85a7795ee67eff74ed51a07dd7f857c04102a65c5e9851a22587932d1bbb7"}
Jul 16 21:30:02.225965 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:02.225934    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58fdbbf655-zgnxb_34fa3be8-1154-4bdd-bdd5-690f98005299/copy-manifests/0.log"
Jul 16 21:30:02.247843 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:02.247813    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58fdbbf655-zgnxb_34fa3be8-1154-4bdd-bdd5-690f98005299/manager/0.log"
Jul 16 21:30:02.619485 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:02.619407    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d7676bd74-hc9zt_a71d5c09-6f12-49df-8e5c-dfc6e75c9a85/maas-api/0.log"
Jul 16 21:30:02.839755 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:02.839722    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk"]
Jul 16 21:30:02.963219 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:02.963199    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737290-254sh"
Jul 16 21:30:02.994616 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:02.994594    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk"
Jul 16 21:30:03.084397 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:03.084365    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82851b9e-2b24-4095-ba87-cffae2319b62-maas-api-service-ca\") pod \"82851b9e-2b24-4095-ba87-cffae2319b62\" (UID: \"82851b9e-2b24-4095-ba87-cffae2319b62\") "
Jul 16 21:30:03.084712 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:03.084689    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82851b9e-2b24-4095-ba87-cffae2319b62-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "82851b9e-2b24-4095-ba87-cffae2319b62" (UID: "82851b9e-2b24-4095-ba87-cffae2319b62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:30:03.086293 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:03.086266    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737290-254sh_5cd0b800-6773-4538-8a4a-17b0a86e1388/cleanup/0.log"
Jul 16 21:30:03.186003 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:03.185940    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82851b9e-2b24-4095-ba87-cffae2319b62-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:30:03.325567 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:03.325538    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk_82851b9e-2b24-4095-ba87-cffae2319b62/revoke-keys/2.log"
Jul 16 21:30:03.439937 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:03.439873    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-ed076e-67mcb_39e0bbd2-48bd-415e-a200-16f70e4d8a17/revoke-keys/0.log"
Jul 16 21:30:03.559056 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:03.559032    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-df0daf-wdh9j_2916edeb-4ced-40b7-8a26-5b2541ce61ed/revoke-keys/0.log"
Jul 16 21:30:03.675262 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:03.675237    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-c69b60-hkxpd_09612e85-d4bc-465f-8518-5d421687491b/revoke-keys/0.log"
Jul 16 21:30:03.793496 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:03.793472    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-72cda6-2zsfg_3b5f5425-622b-41a3-aa8a-68abe4efe80e/revoke-keys/0.log"
Jul 16 21:30:03.828517 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:03.828492    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk"
Jul 16 21:30:03.828517 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:03.828510    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk" event={"ID":"82851b9e-2b24-4095-ba87-cffae2319b62","Type":"ContainerDied","Data":"f73f2c9b55bf282b017bd13a7b37edabc2ea9e222cf382c6050afdad42761696"}
Jul 16 21:30:03.828672 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:03.828565    2530 scope.go:117] "RemoveContainer" containerID="e7fb443f2b1ce9715f168ba5f3a603c918815220c4ca3c5524f669ccaccfed28"
Jul 16 21:30:03.829908 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:03.829888    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737290-254sh" event={"ID":"5cd0b800-6773-4538-8a4a-17b0a86e1388","Type":"ContainerDied","Data":"35cdc91d9f7775c746cf5753c02d5f06f81cdc46a00425ecc38127b9840a6567"}
Jul 16 21:30:03.829989 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:03.829911    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737290-254sh"
Jul 16 21:30:03.829989 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:03.829924    2530 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35cdc91d9f7775c746cf5753c02d5f06f81cdc46a00425ecc38127b9840a6567"
Jul 16 21:30:03.853662 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:03.853641    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk"]
Jul 16 21:30:03.861476 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:03.861455    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-fvxmk"]
Jul 16 21:30:03.914483 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:03.914466    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-2a7694fd-g7glb_210510e4-59c4-4e4e-a5a4-2168c6c2e9eb/revoke-keys/0.log"
Jul 16 21:30:04.024433 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:04.024410    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-e9d9b90e-mdhj9_ba61385c-c6f1-4a82-87d9-125ad45e918a/revoke-keys/0.log"
Jul 16 21:30:04.139420 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:04.139359    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-9331e3-2cfjp_f019e659-03f4-4eea-946a-cc25a4c2e035/revoke-keys/0.log"
Jul 16 21:30:04.262143 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:04.262120    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-fb5e77-btstx_7e084b8e-a1b7-4907-bf86-7ff4696ac8e7/revoke-keys/0.log"
Jul 16 21:30:04.376307 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:04.376284    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-hvclx_f2f1cf1d-ba83-4ede-8fac-f1734033dee6/postgres/0.log"
Jul 16 21:30:04.506986 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:04.506960    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82851b9e-2b24-4095-ba87-cffae2319b62" path="/var/lib/kubelet/pods/82851b9e-2b24-4095-ba87-cffae2319b62/volumes"
Jul 16 21:30:06.576039 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:06.576002    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-bxpjf_76444973-2719-4125-aa2d-663c2758b13e/manager/0.log"
Jul 16 21:30:07.067229 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:07.067197    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-czqvm_1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96/discovery/0.log"
Jul 16 21:30:07.177935 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:07.177903    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74779b8bd6-b48mq_487f8c35-e7db-47aa-aa92-b9187479bae0/kube-auth-proxy/0.log"
Jul 16 21:30:07.662298 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:07.662251    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tc2bv_e3b7ccc1-812d-45b7-a525-a6f017dd936c/payload-processing/0.log"
Jul 16 21:30:08.151703 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:08.151675    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-zh6x8_3fdd0719-44d3-454f-b45f-d6d3dd0726e9/main/0.log"
Jul 16 21:30:08.517581 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:08.517544    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbzdphz_756ba2bd-efa3-40a2-b89a-a29285e54a84/main/0.log"
Jul 16 21:30:17.469538 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:17.469505    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tjnr2_7f1aaf1f-7cc2-4e44-bb74-fac52ddbb13e/global-pull-secret-syncer/0.log"
Jul 16 21:30:17.603212 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:17.603186    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nf9hg_205ec3c3-fd73-4c63-9298-2fe9b3f8a625/konnectivity-agent/0.log"
Jul 16 21:30:17.660541 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:17.660499    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-249.ec2.internal_19742d85a0eb69c9b3078865a8f5b5d4/haproxy/0.log"
Jul 16 21:30:19.941395 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.941364    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg"]
Jul 16 21:30:19.941777 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.941678    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82851b9e-2b24-4095-ba87-cffae2319b62" containerName="revoke-keys"
Jul 16 21:30:19.941777 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.941690    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="82851b9e-2b24-4095-ba87-cffae2319b62" containerName="revoke-keys"
Jul 16 21:30:19.941777 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.941699    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82851b9e-2b24-4095-ba87-cffae2319b62" containerName="revoke-keys"
Jul 16 21:30:19.941777 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.941704    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="82851b9e-2b24-4095-ba87-cffae2319b62" containerName="revoke-keys"
Jul 16 21:30:19.941777 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.941712    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd0b800-6773-4538-8a4a-17b0a86e1388" containerName="cleanup"
Jul 16 21:30:19.941777 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.941717    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd0b800-6773-4538-8a4a-17b0a86e1388" containerName="cleanup"
Jul 16 21:30:19.941777 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.941725    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82851b9e-2b24-4095-ba87-cffae2319b62" containerName="revoke-keys"
Jul 16 21:30:19.941777 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.941731    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="82851b9e-2b24-4095-ba87-cffae2319b62" containerName="revoke-keys"
Jul 16 21:30:19.941777 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.941742    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7643778a-00eb-4f04-8193-4ab704b1f4c3" containerName="revoke-keys"
Jul 16 21:30:19.941777 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.941747    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="7643778a-00eb-4f04-8193-4ab704b1f4c3" containerName="revoke-keys"
Jul 16 21:30:19.942169 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.941795    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="82851b9e-2b24-4095-ba87-cffae2319b62" containerName="revoke-keys"
Jul 16 21:30:19.942169 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.941802    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cd0b800-6773-4538-8a4a-17b0a86e1388" containerName="cleanup"
Jul 16 21:30:19.942169 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.941807    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="82851b9e-2b24-4095-ba87-cffae2319b62" containerName="revoke-keys"
Jul 16 21:30:19.942169 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.941814    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="82851b9e-2b24-4095-ba87-cffae2319b62" containerName="revoke-keys"
Jul 16 21:30:19.942169 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.941821    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="81d1b2a2-0f59-4416-a252-98c84366c25f" containerName="revoke-keys"
Jul 16 21:30:19.946080 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.946063    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg"
Jul 16 21:30:19.948668 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.948640    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hb99\""
Jul 16 21:30:19.954398 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:19.954375    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg"]
Jul 16 21:30:20.114332 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:20.114302    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eda240b-46d7-4475-b3f8-67780ff29a27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg\" (UID: \"8eda240b-46d7-4475-b3f8-67780ff29a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg"
Jul 16 21:30:20.215714 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:20.215682    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eda240b-46d7-4475-b3f8-67780ff29a27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg\" (UID: \"8eda240b-46d7-4475-b3f8-67780ff29a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg"
Jul 16 21:30:20.216352 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:20.216332    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eda240b-46d7-4475-b3f8-67780ff29a27-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg\" (UID: \"8eda240b-46d7-4475-b3f8-67780ff29a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg"
Jul 16 21:30:20.256935 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:20.256914    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg"
Jul 16 21:30:20.384053 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:20.384031    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg"]
Jul 16 21:30:20.386762 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:30:20.386734    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eda240b_46d7_4475_b3f8_67780ff29a27.slice/crio-40e64c1364a2c35cf60171046aa95eb16da4d0a88a79c5fc23628273335f8b9d WatchSource:0}: Error finding container 40e64c1364a2c35cf60171046aa95eb16da4d0a88a79c5fc23628273335f8b9d: Status 404 returned error can't find the container with id 40e64c1364a2c35cf60171046aa95eb16da4d0a88a79c5fc23628273335f8b9d
Jul 16 21:30:20.910563 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:20.910480    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg" event={"ID":"8eda240b-46d7-4475-b3f8-67780ff29a27","Type":"ContainerStarted","Data":"85efa7cd905fdc4f15ebd8acdec47795ec8985452ce1fca8913433ec8ca7e9fb"}
Jul 16 21:30:20.910563 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:20.910528    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg" event={"ID":"8eda240b-46d7-4475-b3f8-67780ff29a27","Type":"ContainerStarted","Data":"40e64c1364a2c35cf60171046aa95eb16da4d0a88a79c5fc23628273335f8b9d"}
Jul 16 21:30:20.929226 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:20.929178    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg" podStartSLOduration=1.9291636479999998 podStartE2EDuration="1.929163648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:30:20.927714404 +0000 UTC m=+3458.949302167" watchObservedRunningTime="2026-07-16 21:30:20.929163648 +0000 UTC m=+3458.950751414"
Jul 16 21:30:21.922123 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:21.922071    2530 generic.go:356] "Generic (PLEG): container finished" podID="8eda240b-46d7-4475-b3f8-67780ff29a27" containerID="85efa7cd905fdc4f15ebd8acdec47795ec8985452ce1fca8913433ec8ca7e9fb" exitCode=6
Jul 16 21:30:21.922605 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:21.922143    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg" event={"ID":"8eda240b-46d7-4475-b3f8-67780ff29a27","Type":"ContainerDied","Data":"85efa7cd905fdc4f15ebd8acdec47795ec8985452ce1fca8913433ec8ca7e9fb"}
Jul 16 21:30:21.922605 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:21.922423    2530 scope.go:117] "RemoveContainer" containerID="85efa7cd905fdc4f15ebd8acdec47795ec8985452ce1fca8913433ec8ca7e9fb"
Jul 16 21:30:22.929553 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:22.929463    2530 generic.go:356] "Generic (PLEG): container finished" podID="8eda240b-46d7-4475-b3f8-67780ff29a27" containerID="0b7f99cb6ae49b06be2bee1f77697aa1a5850b0ee66fd878c65e8eac7edd6f35" exitCode=6
Jul 16 21:30:22.929968 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:22.929548    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg" event={"ID":"8eda240b-46d7-4475-b3f8-67780ff29a27","Type":"ContainerDied","Data":"0b7f99cb6ae49b06be2bee1f77697aa1a5850b0ee66fd878c65e8eac7edd6f35"}
Jul 16 21:30:22.929968 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:22.929610    2530 scope.go:117] "RemoveContainer" containerID="85efa7cd905fdc4f15ebd8acdec47795ec8985452ce1fca8913433ec8ca7e9fb"
Jul 16 21:30:22.929968 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:22.929807    2530 scope.go:117] "RemoveContainer" containerID="0b7f99cb6ae49b06be2bee1f77697aa1a5850b0ee66fd878c65e8eac7edd6f35"
Jul 16 21:30:22.930168 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:30:22.930004    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg_odh-ai-gateway-infra(8eda240b-46d7-4475-b3f8-67780ff29a27)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg" podUID="8eda240b-46d7-4475-b3f8-67780ff29a27"
Jul 16 21:30:23.390715 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:23.390686    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-bxpjf_76444973-2719-4125-aa2d-663c2758b13e/manager/0.log"
Jul 16 21:30:23.935855 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:23.935820    2530 scope.go:117] "RemoveContainer" containerID="0b7f99cb6ae49b06be2bee1f77697aa1a5850b0ee66fd878c65e8eac7edd6f35"
Jul 16 21:30:23.936301 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:30:23.935989    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg_odh-ai-gateway-infra(8eda240b-46d7-4475-b3f8-67780ff29a27)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg" podUID="8eda240b-46d7-4475-b3f8-67780ff29a27"
Jul 16 21:30:25.597185 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:25.597153    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mlgln_a9aedd3d-3901-4d0d-ac17-13f5475deee3/node-exporter/0.log"
Jul 16 21:30:25.619233 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:25.619212    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mlgln_a9aedd3d-3901-4d0d-ac17-13f5475deee3/kube-rbac-proxy/0.log"
Jul 16 21:30:25.650551 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:25.650535    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mlgln_a9aedd3d-3901-4d0d-ac17-13f5475deee3/init-textfile/0.log"
Jul 16 21:30:27.595269 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.595239    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"]
Jul 16 21:30:27.598659 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.598634    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:27.600918 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.600893    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7plfz\"/\"kube-root-ca.crt\""
Jul 16 21:30:27.601057 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.600895    2530 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7plfz\"/\"openshift-service-ca.crt\""
Jul 16 21:30:27.601057 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.600960    2530 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7plfz\"/\"default-dockercfg-pzbhc\""
Jul 16 21:30:27.608661 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.608641    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"]
Jul 16 21:30:27.672686 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.672664    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a-lib-modules\") pod \"perf-node-gather-daemonset-ckpxn\" (UID: \"1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a\") " pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:27.672800 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.672705    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a-sys\") pod \"perf-node-gather-daemonset-ckpxn\" (UID: \"1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a\") " pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:27.672800 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.672748    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcpnk\" (UniqueName: \"kubernetes.io/projected/1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a-kube-api-access-rcpnk\") pod \"perf-node-gather-daemonset-ckpxn\" (UID: \"1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a\") " pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:27.672889 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.672832    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a-podres\") pod \"perf-node-gather-daemonset-ckpxn\" (UID: \"1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a\") " pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:27.672948 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.672912    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a-proc\") pod \"perf-node-gather-daemonset-ckpxn\" (UID: \"1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a\") " pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:27.773359 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.773326    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a-lib-modules\") pod \"perf-node-gather-daemonset-ckpxn\" (UID: \"1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a\") " pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:27.773482 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.773365    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a-sys\") pod \"perf-node-gather-daemonset-ckpxn\" (UID: \"1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a\") " pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:27.773482 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.773382    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcpnk\" (UniqueName: \"kubernetes.io/projected/1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a-kube-api-access-rcpnk\") pod \"perf-node-gather-daemonset-ckpxn\" (UID: \"1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a\") " pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:27.773585 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.773506    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a-sys\") pod \"perf-node-gather-daemonset-ckpxn\" (UID: \"1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a\") " pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:27.773585 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.773518    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a-lib-modules\") pod \"perf-node-gather-daemonset-ckpxn\" (UID: \"1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a\") " pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:27.773585 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.773571    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a-podres\") pod \"perf-node-gather-daemonset-ckpxn\" (UID: \"1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a\") " pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:27.773702 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.773624    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a-proc\") pod \"perf-node-gather-daemonset-ckpxn\" (UID: \"1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a\") " pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:27.773702 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.773678    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a-proc\") pod \"perf-node-gather-daemonset-ckpxn\" (UID: \"1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a\") " pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:27.773702 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.773693    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a-podres\") pod \"perf-node-gather-daemonset-ckpxn\" (UID: \"1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a\") " pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:27.782147 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.782119    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcpnk\" (UniqueName: \"kubernetes.io/projected/1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a-kube-api-access-rcpnk\") pod \"perf-node-gather-daemonset-ckpxn\" (UID: \"1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a\") " pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:27.909796 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:27.909737    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:28.035718 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:28.035690    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"]
Jul 16 21:30:28.037435 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:30:28.037407    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1647c1c9_ee7f_4ab4_ac71_2b602ba70f0a.slice/crio-245886780e25173416157d48f73e4ec88cf48b6eb0051a742f31fdb61cb17dc1 WatchSource:0}: Error finding container 245886780e25173416157d48f73e4ec88cf48b6eb0051a742f31fdb61cb17dc1: Status 404 returned error can't find the container with id 245886780e25173416157d48f73e4ec88cf48b6eb0051a742f31fdb61cb17dc1
Jul 16 21:30:28.959960 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:28.959920    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn" event={"ID":"1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a","Type":"ContainerStarted","Data":"ea7c9e41e15156bb65b765c457e3a56501b7365a5f54400b4bb2fe1181be4ba6"}
Jul 16 21:30:28.960391 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:28.959981    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn" event={"ID":"1647c1c9-ee7f-4ab4-ac71-2b602ba70f0a","Type":"ContainerStarted","Data":"245886780e25173416157d48f73e4ec88cf48b6eb0051a742f31fdb61cb17dc1"}
Jul 16 21:30:28.960391 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:28.960046    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:28.976558 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:28.976510    2530 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn" podStartSLOduration=1.976497495 podStartE2EDuration="1.976497495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:30:28.975881906 +0000 UTC m=+3466.997469673" watchObservedRunningTime="2026-07-16 21:30:28.976497495 +0000 UTC m=+3466.998085262"
Jul 16 21:30:30.319375 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:30.319348    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x7pgr_3589675b-514d-46eb-b879-9c66f98576ae/dns/0.log"
Jul 16 21:30:30.342220 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:30.342196    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x7pgr_3589675b-514d-46eb-b879-9c66f98576ae/kube-rbac-proxy/0.log"
Jul 16 21:30:30.369571 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:30.369551    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-764h9_9388c864-76b1-415e-a3d8-6114d5968730/dns-node-resolver/0.log"
Jul 16 21:30:30.956421 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:30.956357    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-85d4cdcc5d-h29lp_fba88453-5e73-4e08-a483-0a764c1a6b6f/registry/0.log"
Jul 16 21:30:31.009233 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:31.009213    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cwld6_4288933c-4329-4853-8fb3-e3d46c34479f/node-ca/0.log"
Jul 16 21:30:32.210606 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:32.210567    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-czqvm_1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96/discovery/0.log"
Jul 16 21:30:32.234562 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:32.234511    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74779b8bd6-b48mq_487f8c35-e7db-47aa-aa92-b9187479bae0/kube-auth-proxy/0.log"
Jul 16 21:30:32.417238 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:32.417210    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tc2bv_e3b7ccc1-812d-45b7-a525-a6f017dd936c/payload-processing/0.log"
Jul 16 21:30:33.050969 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:33.050941    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n5mqv"]
Jul 16 21:30:33.054461 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:33.054446    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n5mqv"
Jul 16 21:30:33.066122 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:33.066083    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n5mqv"]
Jul 16 21:30:33.106023 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:33.105998    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/318e2346-028c-47bd-a5e7-5856f78a6f8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-n5mqv\" (UID: \"318e2346-028c-47bd-a5e7-5856f78a6f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n5mqv"
Jul 16 21:30:33.176160 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:33.176127    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hmjxf_a949c15b-79d3-42e2-ba6f-aaa345dd0bcf/serve-healthcheck-canary/0.log"
Jul 16 21:30:33.206429 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:33.206402    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/318e2346-028c-47bd-a5e7-5856f78a6f8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-n5mqv\" (UID: \"318e2346-028c-47bd-a5e7-5856f78a6f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n5mqv"
Jul 16 21:30:33.206954 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:33.206937    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/318e2346-028c-47bd-a5e7-5856f78a6f8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ad09ff70-n5mqv\" (UID: \"318e2346-028c-47bd-a5e7-5856f78a6f8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n5mqv"
Jul 16 21:30:33.364429 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:33.364365    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n5mqv"
Jul 16 21:30:33.705523 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:33.705497    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n5mqv"]
Jul 16 21:30:33.707553 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:30:33.707527    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318e2346_028c_47bd_a5e7_5856f78a6f8e.slice/crio-946f0fcdbcb1e955db2af5128f89d0fd0f9e842a4c7da363bee790d209b3b412 WatchSource:0}: Error finding container 946f0fcdbcb1e955db2af5128f89d0fd0f9e842a4c7da363bee790d209b3b412: Status 404 returned error can't find the container with id 946f0fcdbcb1e955db2af5128f89d0fd0f9e842a4c7da363bee790d209b3b412
Jul 16 21:30:33.718344 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:33.718315    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-55rbb_71aec882-bb2e-4daa-9975-3eb3b1879cfa/kube-rbac-proxy/0.log"
Jul 16 21:30:33.739674 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:33.739658    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-55rbb_71aec882-bb2e-4daa-9975-3eb3b1879cfa/exporter/0.log"
Jul 16 21:30:33.760033 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:33.760005    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-55rbb_71aec882-bb2e-4daa-9975-3eb3b1879cfa/extractor/0.log"
Jul 16 21:30:33.985396 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:33.985323    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n5mqv" event={"ID":"318e2346-028c-47bd-a5e7-5856f78a6f8e","Type":"ContainerStarted","Data":"946f0fcdbcb1e955db2af5128f89d0fd0f9e842a4c7da363bee790d209b3b412"}
Jul 16 21:30:34.502009 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:34.501984    2530 scope.go:117] "RemoveContainer" containerID="0b7f99cb6ae49b06be2bee1f77697aa1a5850b0ee66fd878c65e8eac7edd6f35"
Jul 16 21:30:34.974968 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:34.974942    2530 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7plfz/perf-node-gather-daemonset-ckpxn"
Jul 16 21:30:34.991408 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:34.991346    2530 generic.go:356] "Generic (PLEG): container finished" podID="318e2346-028c-47bd-a5e7-5856f78a6f8e" containerID="8ba87b711e70f021c04c46b06a75003a1a4280e7db39fd1689a43af5f27ee2d9" exitCode=6
Jul 16 21:30:34.991408 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:34.991376    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n5mqv" event={"ID":"318e2346-028c-47bd-a5e7-5856f78a6f8e","Type":"ContainerDied","Data":"8ba87b711e70f021c04c46b06a75003a1a4280e7db39fd1689a43af5f27ee2d9"}
Jul 16 21:30:34.991585 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:34.991552    2530 scope.go:117] "RemoveContainer" containerID="8ba87b711e70f021c04c46b06a75003a1a4280e7db39fd1689a43af5f27ee2d9"
Jul 16 21:30:34.993805 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:34.993790    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg" event={"ID":"8eda240b-46d7-4475-b3f8-67780ff29a27","Type":"ContainerStarted","Data":"3094956fb2066dc0a23a11e212a5368dcc497c82261a208ce122bcddce58aeb8"}
Jul 16 21:30:36.001261 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:36.001171    2530 generic.go:356] "Generic (PLEG): container finished" podID="8eda240b-46d7-4475-b3f8-67780ff29a27" containerID="3094956fb2066dc0a23a11e212a5368dcc497c82261a208ce122bcddce58aeb8" exitCode=6
Jul 16 21:30:36.001261 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:36.001226    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg" event={"ID":"8eda240b-46d7-4475-b3f8-67780ff29a27","Type":"ContainerDied","Data":"3094956fb2066dc0a23a11e212a5368dcc497c82261a208ce122bcddce58aeb8"}
Jul 16 21:30:36.001755 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:36.001297    2530 scope.go:117] "RemoveContainer" containerID="0b7f99cb6ae49b06be2bee1f77697aa1a5850b0ee66fd878c65e8eac7edd6f35"
Jul 16 21:30:36.001755 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:36.001605    2530 scope.go:117] "RemoveContainer" containerID="3094956fb2066dc0a23a11e212a5368dcc497c82261a208ce122bcddce58aeb8"
Jul 16 21:30:36.001865 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:30:36.001822    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg_odh-ai-gateway-infra(8eda240b-46d7-4475-b3f8-67780ff29a27)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg" podUID="8eda240b-46d7-4475-b3f8-67780ff29a27"
Jul 16 21:30:36.003418 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:36.003393    2530 generic.go:356] "Generic (PLEG): container finished" podID="318e2346-028c-47bd-a5e7-5856f78a6f8e" containerID="b0a580539c215e1972f1b07db087ec1f7380e97bed04247f8245effc71178b3b" exitCode=6
Jul 16 21:30:36.003538 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:36.003473    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n5mqv" event={"ID":"318e2346-028c-47bd-a5e7-5856f78a6f8e","Type":"ContainerDied","Data":"b0a580539c215e1972f1b07db087ec1f7380e97bed04247f8245effc71178b3b"}
Jul 16 21:30:36.003628 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:36.003616    2530 scope.go:117] "RemoveContainer" containerID="b0a580539c215e1972f1b07db087ec1f7380e97bed04247f8245effc71178b3b"
Jul 16 21:30:36.003803 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:30:36.003786    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-n5mqv_odh-ai-gateway-infra(318e2346-028c-47bd-a5e7-5856f78a6f8e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n5mqv" podUID="318e2346-028c-47bd-a5e7-5856f78a6f8e"
Jul 16 21:30:36.012540 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:36.012518    2530 scope.go:117] "RemoveContainer" containerID="8ba87b711e70f021c04c46b06a75003a1a4280e7db39fd1689a43af5f27ee2d9"
Jul 16 21:30:36.034591 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:36.034570    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg"]
Jul 16 21:30:36.346407 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:36.346327    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-tkrwz_88af9955-7488-44ce-a8a4-2589b1b302ae/manager/0.log"
Jul 16 21:30:36.599812 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:36.599732    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-sdzsg_a8f11bc3-e9ed-4832-b76f-94e5e31522dd/manager/0.log"
Jul 16 21:30:37.013008 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:37.012986    2530 scope.go:117] "RemoveContainer" containerID="b0a580539c215e1972f1b07db087ec1f7380e97bed04247f8245effc71178b3b"
Jul 16 21:30:37.013665 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:30:37.013639    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-ad09ff70-n5mqv_odh-ai-gateway-infra(318e2346-028c-47bd-a5e7-5856f78a6f8e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ad09ff70-n5mqv" podUID="318e2346-028c-47bd-a5e7-5856f78a6f8e"
Jul 16 21:30:37.080369 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:37.080350    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58fdbbf655-zgnxb_34fa3be8-1154-4bdd-bdd5-690f98005299/manager/0.log"
Jul 16 21:30:37.104014 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:37.103993    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58fdbbf655-zgnxb_34fa3be8-1154-4bdd-bdd5-690f98005299/copy-manifests/0.log"
Jul 16 21:30:37.173588 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:37.173567    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg"
Jul 16 21:30:37.245220 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:37.245192    2530 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eda240b-46d7-4475-b3f8-67780ff29a27-maas-api-service-ca\") pod \"8eda240b-46d7-4475-b3f8-67780ff29a27\" (UID: \"8eda240b-46d7-4475-b3f8-67780ff29a27\") "
Jul 16 21:30:37.245497 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:37.245476    2530 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eda240b-46d7-4475-b3f8-67780ff29a27-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8eda240b-46d7-4475-b3f8-67780ff29a27" (UID: "8eda240b-46d7-4475-b3f8-67780ff29a27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:30:37.346346 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:37.346323    2530 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eda240b-46d7-4475-b3f8-67780ff29a27-maas-api-service-ca\") on node \"ip-10-0-136-249.ec2.internal\" DevicePath \"\""
Jul 16 21:30:37.797846 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:37.797812    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-czqvm_1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96/discovery/0.log"
Jul 16 21:30:37.808673 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:37.808641    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74779b8bd6-b48mq_487f8c35-e7db-47aa-aa92-b9187479bae0/kube-auth-proxy/0.log"
Jul 16 21:30:37.934257 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:37.934181    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tc2bv_e3b7ccc1-812d-45b7-a525-a6f017dd936c/payload-processing/0.log"
Jul 16 21:30:38.018911 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:38.018886    2530 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg"
Jul 16 21:30:38.018911 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:38.018894    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg" event={"ID":"8eda240b-46d7-4475-b3f8-67780ff29a27","Type":"ContainerDied","Data":"40e64c1364a2c35cf60171046aa95eb16da4d0a88a79c5fc23628273335f8b9d"}
Jul 16 21:30:38.019380 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:38.018971    2530 scope.go:117] "RemoveContainer" containerID="3094956fb2066dc0a23a11e212a5368dcc497c82261a208ce122bcddce58aeb8"
Jul 16 21:30:38.044243 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:38.044213    2530 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg"]
Jul 16 21:30:38.048614 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:38.048589    2530 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-90b68ec7-tm2jg"]
Jul 16 21:30:38.507904 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:38.507869    2530 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eda240b-46d7-4475-b3f8-67780ff29a27" path="/var/lib/kubelet/pods/8eda240b-46d7-4475-b3f8-67780ff29a27/volumes"
Jul 16 21:30:39.375517 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:39.375443    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-czqvm_1bc66d7b-0db0-4b3a-88a1-b14e9f6eaf96/discovery/0.log"
Jul 16 21:30:39.391620 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:39.391596    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74779b8bd6-b48mq_487f8c35-e7db-47aa-aa92-b9187479bae0/kube-auth-proxy/0.log"
Jul 16 21:30:39.539029 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:39.538996    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tc2bv_e3b7ccc1-812d-45b7-a525-a6f017dd936c/payload-processing/0.log"
Jul 16 21:30:42.609721 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:42.609687    2530 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-d87p2"]
Jul 16 21:30:42.610087 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:42.610006    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eda240b-46d7-4475-b3f8-67780ff29a27" containerName="revoke-keys"
Jul 16 21:30:42.610087 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:42.610017    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eda240b-46d7-4475-b3f8-67780ff29a27" containerName="revoke-keys"
Jul 16 21:30:42.610087 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:42.610026    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eda240b-46d7-4475-b3f8-67780ff29a27" containerName="revoke-keys"
Jul 16 21:30:42.610087 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:42.610032    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eda240b-46d7-4475-b3f8-67780ff29a27" containerName="revoke-keys"
Jul 16 21:30:42.610087 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:42.610040    2530 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eda240b-46d7-4475-b3f8-67780ff29a27" containerName="revoke-keys"
Jul 16 21:30:42.610087 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:42.610046    2530 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eda240b-46d7-4475-b3f8-67780ff29a27" containerName="revoke-keys"
Jul 16 21:30:42.610298 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:42.610095    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eda240b-46d7-4475-b3f8-67780ff29a27" containerName="revoke-keys"
Jul 16 21:30:42.610298 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:42.610119    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eda240b-46d7-4475-b3f8-67780ff29a27" containerName="revoke-keys"
Jul 16 21:30:42.610298 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:42.610125    2530 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eda240b-46d7-4475-b3f8-67780ff29a27" containerName="revoke-keys"
Jul 16 21:30:42.618961 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:42.618937    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-d87p2"
Jul 16 21:30:42.619895 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:42.619870    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-d87p2"]
Jul 16 21:30:42.683595 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:42.683568    2530 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d087485a-c733-4961-9b35-469024212460-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-d87p2\" (UID: \"d087485a-c733-4961-9b35-469024212460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-d87p2"
Jul 16 21:30:42.784714 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:42.784679    2530 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d087485a-c733-4961-9b35-469024212460-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-d87p2\" (UID: \"d087485a-c733-4961-9b35-469024212460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-d87p2"
Jul 16 21:30:42.785279 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:42.785263    2530 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d087485a-c733-4961-9b35-469024212460-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e65739ab-d87p2\" (UID: \"d087485a-c733-4961-9b35-469024212460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-d87p2"
Jul 16 21:30:42.930441 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:42.930379    2530 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-d87p2"
Jul 16 21:30:43.062061 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:43.062028    2530 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-d87p2"]
Jul 16 21:30:43.063074 ip-10-0-136-249 kubenswrapper[2530]: W0716 21:30:43.063044    2530 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd087485a_c733_4961_9b35_469024212460.slice/crio-c93c106ea6a7a7249d50a241915c2e5ab5a2b13013012985e025e446d5a36dd3 WatchSource:0}: Error finding container c93c106ea6a7a7249d50a241915c2e5ab5a2b13013012985e025e446d5a36dd3: Status 404 returned error can't find the container with id c93c106ea6a7a7249d50a241915c2e5ab5a2b13013012985e025e446d5a36dd3
Jul 16 21:30:43.271786 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:43.271764    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-fbf6t_210aa416-26e0-48e0-97c2-a04d85c4be5b/migrator/0.log"
Jul 16 21:30:43.294608 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:43.294588    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-fbf6t_210aa416-26e0-48e0-97c2-a04d85c4be5b/graceful-termination/0.log"
Jul 16 21:30:44.052382 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:44.052350    2530 generic.go:356] "Generic (PLEG): container finished" podID="d087485a-c733-4961-9b35-469024212460" containerID="a2218f0538ec333a53b2244396a4b5cb9bae66d4b7e34629a7b6de48b036bb90" exitCode=6
Jul 16 21:30:44.052730 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:44.052447    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-d87p2" event={"ID":"d087485a-c733-4961-9b35-469024212460","Type":"ContainerDied","Data":"a2218f0538ec333a53b2244396a4b5cb9bae66d4b7e34629a7b6de48b036bb90"}
Jul 16 21:30:44.052730 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:44.052507    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-d87p2" event={"ID":"d087485a-c733-4961-9b35-469024212460","Type":"ContainerStarted","Data":"c93c106ea6a7a7249d50a241915c2e5ab5a2b13013012985e025e446d5a36dd3"}
Jul 16 21:30:44.052730 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:44.052724    2530 scope.go:117] "RemoveContainer" containerID="a2218f0538ec333a53b2244396a4b5cb9bae66d4b7e34629a7b6de48b036bb90"
Jul 16 21:30:45.058113 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:45.058067    2530 generic.go:356] "Generic (PLEG): container finished" podID="d087485a-c733-4961-9b35-469024212460" containerID="5e5919d3f3d6037849a5608dadfe27109ebc4f985193472407e991cbd544a93b" exitCode=6
Jul 16 21:30:45.058501 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:45.058141    2530 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-d87p2" event={"ID":"d087485a-c733-4961-9b35-469024212460","Type":"ContainerDied","Data":"5e5919d3f3d6037849a5608dadfe27109ebc4f985193472407e991cbd544a93b"}
Jul 16 21:30:45.058501 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:45.058194    2530 scope.go:117] "RemoveContainer" containerID="a2218f0538ec333a53b2244396a4b5cb9bae66d4b7e34629a7b6de48b036bb90"
Jul 16 21:30:45.058501 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:45.058319    2530 scope.go:117] "RemoveContainer" containerID="5e5919d3f3d6037849a5608dadfe27109ebc4f985193472407e991cbd544a93b"
Jul 16 21:30:45.058631 ip-10-0-136-249 kubenswrapper[2530]: E0716 21:30:45.058509    2530 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-e65739ab-d87p2_odh-ai-gateway-infra(d087485a-c733-4961-9b35-469024212460)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e65739ab-d87p2" podUID="d087485a-c733-4961-9b35-469024212460"
Jul 16 21:30:45.062997 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:45.062981    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xttrm_586e85c8-fa49-4b8a-bc17-1d57c31b1c79/kube-multus-additional-cni-plugins/0.log"
Jul 16 21:30:45.085994 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:45.085975    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xttrm_586e85c8-fa49-4b8a-bc17-1d57c31b1c79/egress-router-binary-copy/0.log"
Jul 16 21:30:45.107577 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:45.107557    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xttrm_586e85c8-fa49-4b8a-bc17-1d57c31b1c79/cni-plugins/0.log"
Jul 16 21:30:45.132714 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:45.132693    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xttrm_586e85c8-fa49-4b8a-bc17-1d57c31b1c79/bond-cni-plugin/0.log"
Jul 16 21:30:45.154990 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:45.154973    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xttrm_586e85c8-fa49-4b8a-bc17-1d57c31b1c79/routeoverride-cni/0.log"
Jul 16 21:30:45.175327 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:45.175303    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xttrm_586e85c8-fa49-4b8a-bc17-1d57c31b1c79/whereabouts-cni-bincopy/0.log"
Jul 16 21:30:45.196921 ip-10-0-136-249 kubenswrapper[2530]: I0716 21:30:45.196903    2530 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xttrm_586e85c8-fa49-4b8a-bc17-1d57c31b1c79/whereabouts-cni/0.log"