Jul 17 21:06:30.796855 ip-10-0-128-246 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 17 21:06:30.796868 ip-10-0-128-246 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 17 21:06:30.796878 ip-10-0-128-246 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 17 21:06:30.797353 ip-10-0-128-246 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 17 21:06:40.798408 ip-10-0-128-246 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 17 21:06:40.798426 ip-10-0-128-246 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 68926a528b1f473cbfceaa1e80759df8 --
Jul 17 21:08:19.936549 ip-10-0-128-246 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 21:08:20.310272 ip-10-0-128-246 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 21:08:20.310272 ip-10-0-128-246 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 21:08:20.310272 ip-10-0-128-246 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 21:08:20.310770 ip-10-0-128-246 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 21:08:20.310770 ip-10-0-128-246 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 21:08:20.311233 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.311126    2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 21:08:20.316026 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.315971    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:08:20.316026 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.315994    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:08:20.316026 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316002    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:08:20.316026 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316007    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316035    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316046    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316050    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316056    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316062    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316182    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316188    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316191    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316194    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316197    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316200    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316203    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316206    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316209    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316211    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316214    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316217    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316220    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:08:20.316252 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316222    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316225    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316227    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316230    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316233    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316236    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316238    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316241    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316244    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316247    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316250    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316253    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316255    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316258    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316261    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316264    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316266    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316270    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316272    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:08:20.316759 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316275    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316279    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316293    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316297    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316299    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316302    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316305    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316308    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316311    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316313    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316318    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316320    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316324    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316326    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316329    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316332    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316335    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316338    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316341    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316343    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:08:20.317234 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316346    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316349    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316351    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316354    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316356    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316359    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316362    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316365    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316368    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316371    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316373    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316376    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316378    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316381    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316384    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316386    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316391    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316394    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316396    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316399    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:08:20.317733 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316402    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316404    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316407    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316410    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316412    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316415    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316418    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316421    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316424    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316427    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316430    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316434    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316438    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316441    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316444    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316447    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316450    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.316452    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317158    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:08:20.318219 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317164    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317167    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317169    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317172    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317175    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317179    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317183    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317186    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317189    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317192    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317194    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317198    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317202    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317205    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317207    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317210    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317213    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317216    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317219    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:08:20.318694 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317221    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317224    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317227    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317230    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317232    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317235    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317238    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317240    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317243    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317246    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317248    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317251    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317254    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317257    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317259    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317261    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317264    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317266    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317269    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317271    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:08:20.319161 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317274    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317276    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317279    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317297    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317300    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317302    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317305    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317308    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317310    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317313    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317316    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317318    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317321    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317324    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317326    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317329    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317332    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317335    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317337    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317340    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:08:20.319648 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317343    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317345    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317348    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317350    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317353    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317356    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317358    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317361    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317363    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317366    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317368    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317372    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317374    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317376    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317379    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317382    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317384    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317386    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317389    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:08:20.320127 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317392    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317394    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317396    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317399    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317402    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317404    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317408    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317412    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317416    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317418    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317421    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317423    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317426    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317428    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317432    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317436    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317439    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317442    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317445    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:08:20.320604 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317447    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317850    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317856    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317859    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317862    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317865    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317867    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317871    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317875    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317879    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317883    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317886    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317889    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317892    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317895    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317897    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317900    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317903    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317906    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:08:20.321083 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317908    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317912    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317914    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317917    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317920    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317923    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317925    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317928    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317930    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317933    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317935    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317937    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317940    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317942    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317945    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317948    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317950    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317952    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317955    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317958    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:08:20.321554 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317960    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317962    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317965    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317967    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317970    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317972    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317974    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317977    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317979    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317982    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317984    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317988    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317991    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317994    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317996    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.317999    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318002    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318004    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318007    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318009    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:08:20.322055 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318012    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318014    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318017    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318019    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318022    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318024    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318028    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318030    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318033    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318035    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318038    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318040    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318043    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318045    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318049    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318052    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318055    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318058    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318060    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:08:20.322610 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318063    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318065    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318068    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318071    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318073    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318077    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318088    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318092    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318095    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318097    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318100    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318102    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318105    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318108    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318110    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318113    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318115    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318118    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318121    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318123    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:08:20.323070 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318126    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318128    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318204    2573 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318215    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318240    2573 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318245    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318249    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318252    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318257    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318261    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318265    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318269    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318272    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318275    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318278    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318294    2573 flags.go:64] FLAG: --cgroup-root=""
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318297    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318300    2573 flags.go:64] FLAG: --client-ca-file=""
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318304    2573 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318307    2573 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318312    2573 flags.go:64] FLAG: --cluster-domain=""
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318314    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318317    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318320    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 21:08:20.323596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318323    2573 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318327    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318330    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318333    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318336    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318339    2573 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318341    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318346    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318349    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318352    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318356    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318359    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318362    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318365    2573 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318368    2573 flags.go:64] FLAG: --enable-server="true"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318372    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318376    2573 flags.go:64] FLAG: --event-burst="100"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318379    2573 flags.go:64] FLAG: --event-qps="50"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318382    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318385    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318388    2573 flags.go:64] FLAG: --eviction-hard=""
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318391    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318395    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318398    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318401    2573 flags.go:64] FLAG: --eviction-soft=""
Jul 17 21:08:20.324166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318404    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318407    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318410    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318413    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318416    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318418    2573 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318422    2573 flags.go:64] FLAG: --feature-gates=""
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318425    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318428    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318431    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318434    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318437    2573 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318440    2573 flags.go:64] FLAG: --help="false"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318443    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-128-246.ec2.internal"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318448    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318450    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318453    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318457    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318460    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318463    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318466    2573 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318469    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318472    2573 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 21:08:20.324775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318475    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318478    2573 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318482    2573 flags.go:64] FLAG: --kube-reserved=""
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318485    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318487    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318491    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318493    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318497    2573 flags.go:64] FLAG: --lock-file=""
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318499    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318502    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318505    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318511    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318514    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318517    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318519    2573 flags.go:64] FLAG: --logging-format="text"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318522    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318525    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318528    2573 flags.go:64] FLAG: --manifest-url=""
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318531    2573 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318535    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318538    2573 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318542    2573 flags.go:64] FLAG: --max-pods="110"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318545    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318549    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318552    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 21:08:20.325355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318554    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318558    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318560    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318563    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318570    2573 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318573    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318576    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318579    2573 flags.go:64] FLAG: --pod-cidr=""
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318582    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318586    2573 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318589    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318592    2573 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318595    2573 flags.go:64] FLAG: --port="10250"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318598    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318601    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bd9b8ce4113ec89e"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318604    2573 flags.go:64] FLAG: --qos-reserved=""
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318607    2573 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318610    2573 flags.go:64] FLAG: --register-node="true"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318612    2573 flags.go:64] FLAG: --register-with-taints=""
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318616    2573 flags.go:64] FLAG: --registry-burst="10"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318619    2573 flags.go:64] FLAG: --registry-qps="5"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318622    2573 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318625    2573 flags.go:64] FLAG: --reserved-memory=""
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318628    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318632    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 21:08:20.325960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318635    2573 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318637    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318640    2573 flags.go:64] FLAG: --runonce="false"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318643    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318646    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318649    2573 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318653    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318656    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318659    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318662    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318665    2573 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318667    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318670    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318673    2573 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318678    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318681    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318684    2573 flags.go:64] FLAG: --system-cgroups=""
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318686    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318691    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318694    2573 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318697    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318702    2573 flags.go:64] FLAG: --tls-min-version=""
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318705    2573 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318708    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318710    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 21:08:20.326597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318713    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318716    2573 flags.go:64] FLAG: --v="2"
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318720    2573 flags.go:64] FLAG: --version="false"
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318724    2573 flags.go:64] FLAG: --vmodule=""
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318728    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.318731    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318840    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318844    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318847    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318850    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318853    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318856    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318859    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318863    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318867    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318870    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318873    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318876    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318879    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318882    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318885    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:08:20.327225 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318891    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318894    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318897    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318900    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318902    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318905    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318909    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318912    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318915    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318918    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318920    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318923    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318926    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318928    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318931    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318934    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318937    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318939    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318942    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318944    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:08:20.327752 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318947    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318949    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318952    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318954    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318959    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318961    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318964    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318966    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318969    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318972    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318974    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318977    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318981    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318984    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318987    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318989    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318992    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318994    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318997    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:08:20.328230 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.318999    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319002    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319005    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319007    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319010    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319012    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319015    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319017    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319021    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319025    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319028    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319031    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319034    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319037    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319039    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319042    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319044    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319048    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319051    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:08:20.328706 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319053    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319056    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319059    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319061    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319064    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319066    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319070    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319073    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319075    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319078    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319081    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319083    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319086    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319088    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319091    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319093    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319096    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319098    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319101    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319103    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:08:20.329177 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319106    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:08:20.329690 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319109    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:08:20.329690 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319111    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:08:20.329690 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319120    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:08:20.329690 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319123    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:08:20.329690 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.319126    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:08:20.329690 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.319134    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 21:08:20.329690 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.325429    2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 21:08:20.329690 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.325444    2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 21:08:20.329690 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.326206    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 21:08:20.329690 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.326215    2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 21:08:20.329690 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326267    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:08:20.329690 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326271    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:08:20.329690 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326274    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:08:20.329690 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326278    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:08:20.329690 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326296    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326299    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326302    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326305    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326308    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326310    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326314    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326318    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326321    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326324    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326327    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326330    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326333    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326335    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326338    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326341    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326344    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326346    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326349    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:08:20.330059 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326352    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326355    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326358    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326360    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326363    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326365    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326369    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326371    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326374    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326377    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326379    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326382    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326384    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326387    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326390    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326392    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326394    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326397    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326399    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326402    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:08:20.330657 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326405    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326407    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326410    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326412    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326415    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326417    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326420    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326423    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326427    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326430    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326433    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326435    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326438    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326440    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326443    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326446    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326448    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326451    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326454    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326457    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:08:20.331140 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326460    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326462    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326464    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326467    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326469    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326472    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326475    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326477    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326480    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326482    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326485    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326488    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326490    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326494    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326498    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326501    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326503    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326506    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326509    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:08:20.331704 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326512    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:08:20.332163 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326514    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:08:20.332163 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326517    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:08:20.332163 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326519    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:08:20.332163 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326522    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:08:20.332163 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326524    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:08:20.332163 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326527    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:08:20.332163 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326529    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:08:20.332163 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326532    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:08:20.332163 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326535    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:08:20.332163 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326537    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:08:20.332163 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326540    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:08:20.332163 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326543    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:08:20.332163 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326546    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:08:20.332163 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326548    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:08:20.332163 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326551    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:08:20.332163 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326553    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:08:20.332584 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.326558    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 21:08:20.332584 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326667    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:08:20.332584 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326671    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:08:20.332584 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326674    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:08:20.332584 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326677    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:08:20.332584 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326680    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:08:20.332584 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326683    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:08:20.332584 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326686    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:08:20.332584 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326688    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:08:20.332584 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326691    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:08:20.332584 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326694    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:08:20.332584 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326696    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:08:20.332584 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326699    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:08:20.332584 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326701    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:08:20.332584 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326704    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326706    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326709    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326711    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326714    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326716    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326719    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326721    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326724    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326726    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326729    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326732    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326737    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326740    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326743    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326746    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326749    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326752    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326755    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:08:20.332960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326758    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326761    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326763    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326766    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326768    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326771    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326773    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326776    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326778    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326781    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326783    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326786    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326788    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326791    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326793    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326796    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326798    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326801    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326803    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326806    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:08:20.333497 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326808    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326810    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326813    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326815    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326818    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326821    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326824    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326826    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326829    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326831    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326834    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326836    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326839    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326841    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326844    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326846    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326849    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326851    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326854    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:08:20.333989 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326856    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326860    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326863    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326867    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326870    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326872    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326876    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326879    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326881    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326884    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326886    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326889    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326891    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326893    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326896    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326898    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326901    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326903    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326908    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:08:20.334474 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326911    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:08:20.334924 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326913    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:08:20.334924 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326916    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:08:20.334924 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326918    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:08:20.334924 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326921    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:08:20.334924 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326923    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:08:20.334924 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326926    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:08:20.334924 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326928    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:08:20.334924 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:20.326931    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:08:20.334924 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.326935    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 21:08:20.334924 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.327027    2573 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 21:08:20.334924 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.328860    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 21:08:20.334924 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.330128    2573 server.go:1019] "Starting client certificate rotation"
Jul 17 21:08:20.334924 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.330228    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 21:08:20.334924 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.330260    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 21:08:20.335261 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.332695    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 21:08:20.335261 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.335253    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 21:08:20.344109 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.344090    2573 log.go:25] "Validated CRI v1 runtime API"
Jul 17 21:08:20.345880 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.345863    2573 log.go:25] "Validated CRI v1 image API"
Jul 17 21:08:20.348812 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.348795    2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 21:08:20.351332 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.351311    2573 fs.go:135] Filesystem UUIDs: map[315048a5-dfe3-449d-8d19-f82eca851aef:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8b595bc7-aa42-44f8-8d20-1f631e7d6fd7:/dev/nvme0n1p4]
Jul 17 21:08:20.351391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.351331    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 21:08:20.357152 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.357044    2573 manager.go:216] Machine: {Timestamp:2026-07-17 21:08:20.354984971 +0000 UTC m=+0.319105583 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092706 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f3b82abdb3ba1927bb4bfa3384bb6 SystemUUID:ec2f3b82-abdb-3ba1-927b-b4bfa3384bb6 BootID:68926a52-8b1f-473c-bfce-aa1e80759df8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ed:e3:33:8b:4d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ed:e3:33:8b:4d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:b1:14:2a:d1:ce Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 21:08:20.357152 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.357142    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 21:08:20.357263 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.357215    2573 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 21:08:20.358744 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.358725    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 21:08:20.359958 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.359924    2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 21:08:20.360113 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.359958    2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-246.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 21:08:20.360662 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.360649    2573 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 21:08:20.360728 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.360666    2573 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 21:08:20.360728 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.360676    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 21:08:20.360728 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.360694    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 21:08:20.361732 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.361719    2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 21:08:20.364605 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.364591    2573 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 21:08:20.364733 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.364722    2573 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 21:08:20.364788 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.364759    2573 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 21:08:20.364788 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.364772    2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 21:08:20.364879 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.364796    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 21:08:20.364879 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.364809    2573 kubelet.go:398] "Adding apiserver pod source"
Jul 17 21:08:20.364879 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.364848    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 21:08:20.365204 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.365189    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 21:08:20.365237 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.365212    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 21:08:20.367069 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.367055    2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 21:08:20.368213 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.368198    2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 21:08:20.368265 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.368230    2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 21:08:20.368781 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.368771    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 21:08:20.368814 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.368787    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 21:08:20.368814 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.368793    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 21:08:20.368814 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.368799    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 21:08:20.368814 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.368809    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 21:08:20.368921 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.368818    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 21:08:20.368953 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.368935    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 21:08:20.368953 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.368942    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 21:08:20.368953 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.368948    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 21:08:20.368953 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.368953    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 21:08:20.369055 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.368961    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 21:08:20.369055 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.368969    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 21:08:20.369579 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.369571    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 21:08:20.369611 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.369582    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 21:08:20.372618 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.372605    2573 server.go:1268] "Started kubelet"
Jul 17 21:08:20.372761 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.372690    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 21:08:20.372822 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.372797    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 21:08:20.372905 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.372717    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 21:08:20.373345 ip-10-0-128-246 systemd[1]: Started Kubernetes Kubelet.
Jul 17 21:08:20.380310 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.380270    2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 21:08:20.381787 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:20.381752    2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 21:08:20.382113 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.382094    2573 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 21:08:20.384787 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:20.384763    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 21:08:20.384937 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:20.384919    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-246.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 21:08:20.384991 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.384959    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-246.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 17 21:08:20.385272 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.385259    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7j9st"
Jul 17 21:08:20.385515 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:20.384725    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-246.ec2.internal.18c32fdc3f71c088  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-246.ec2.internal,UID:ip-10-0-128-246.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-246.ec2.internal,},FirstTimestamp:2026-07-17 21:08:20.37257844 +0000 UTC m=+0.336699041,LastTimestamp:2026-07-17 21:08:20.37257844 +0000 UTC m=+0.336699041,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-246.ec2.internal,}"
Jul 17 21:08:20.387301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.387187    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 21:08:20.387301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.387191    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 21:08:20.387301 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:20.387257    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found"
Jul 17 21:08:20.387716 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.387698    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 21:08:20.387716 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.387699    2573 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 21:08:20.387841 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.387727    2573 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 21:08:20.387889 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.387842    2573 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 21:08:20.387889 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.387852    2573 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 21:08:20.388201 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.388185    2573 factory.go:153] Registering CRI-O factory
Jul 17 21:08:20.388301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.388225    2573 factory.go:223] Registration of the crio container factory successfully
Jul 17 21:08:20.388301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.388272    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 21:08:20.388301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.388298    2573 factory.go:55] Registering systemd factory
Jul 17 21:08:20.388447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.388307    2573 factory.go:223] Registration of the systemd container factory successfully
Jul 17 21:08:20.388447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.388327    2573 factory.go:103] Registering Raw factory
Jul 17 21:08:20.388447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.388341    2573 manager.go:1196] Started watching for new ooms in manager
Jul 17 21:08:20.388839 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.388741    2573 manager.go:349] Starting recovery of all containers
Jul 17 21:08:20.392095 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.392073    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7j9st"
Jul 17 21:08:20.399796 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.399772    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:08:20.400444 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.400428    2573 manager.go:354] Recovery completed
Jul 17 21:08:20.401909 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:20.401892    2573 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-246.ec2.internal\" not found" node="ip-10-0-128-246.ec2.internal"
Jul 17 21:08:20.404554 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.404542    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:08:20.406229 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.406216    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:08:20.406273 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.406241    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:08:20.406273 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.406251    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:08:20.406763 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.406752    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 21:08:20.406810 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.406763    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 21:08:20.406810 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.406778    2573 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 21:08:20.406880 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.406813    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:08:20.407859 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.407848    2573 policy_none.go:49] "None policy: Start"
Jul 17 21:08:20.407897 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.407862    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 21:08:20.407897 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.407872    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 21:08:20.412436 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.412314    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:08:20.412503 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.412451    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:08:20.412503 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.412462    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:08:20.487584 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:20.487529    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found"
Jul 17 21:08:20.560252 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.522675    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 21:08:20.560252 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.523809    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 21:08:20.560252 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.523848    2573 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 21:08:20.560252 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.523871    2573 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 21:08:20.560252 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:20.523908    2573 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Jul 17 21:08:20.560252 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.525923    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:08:20.560252 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.559487    2573 policy_none.go:47] "Start"
Jul 17 21:08:20.588612 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:20.588597    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found"
Jul 17 21:08:20.594870 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.594853    2573 manager.go:337] "Starting Device Plugin manager"
Jul 17 21:08:20.594935 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:20.594881    2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 21:08:20.594935 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.594890    2573 server.go:87] "Starting device plugin registration server"
Jul 17 21:08:20.595152 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.595142    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 21:08:20.595223 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.595154    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 21:08:20.595315 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.595299    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 21:08:20.595390 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.595373    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 21:08:20.595390 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.595386    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 21:08:20.595965 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:20.595946    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 21:08:20.596049 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:20.596000    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-246.ec2.internal\" not found"
Jul 17 21:08:20.625003 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.624981    2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal"]
Jul 17 21:08:20.625066 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.625047    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:08:20.626062 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.626048    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:08:20.626153 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.626087    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:08:20.626153 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.626104    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:08:20.627368 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.627327    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:08:20.627452 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.627425    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal"
Jul 17 21:08:20.627452 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.627446    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:08:20.628034 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.628016    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:08:20.628034 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.628027    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:08:20.628164 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.628042    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:08:20.628164 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.628047    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:08:20.628164 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.628053    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:08:20.628164 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.628056    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:08:20.629345 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.629329    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal"
Jul 17 21:08:20.629388 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.629361    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:08:20.630048 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.630034    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:08:20.630121 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.630061    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:08:20.630121 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.630073    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:08:20.655880 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:20.655858    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-246.ec2.internal\" not found" node="ip-10-0-128-246.ec2.internal"
Jul 17 21:08:20.660035 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:20.660019    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-246.ec2.internal\" not found" node="ip-10-0-128-246.ec2.internal"
Jul 17 21:08:20.690571 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.690540    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27af577f6839740a0585a4e6b9dd13dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" (UID: \"27af577f6839740a0585a4e6b9dd13dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal"
Jul 17 21:08:20.690571 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.690577    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27af577f6839740a0585a4e6b9dd13dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" (UID: \"27af577f6839740a0585a4e6b9dd13dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal"
Jul 17 21:08:20.690699 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.690595    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a9ca977836d346c8c6c32a4fb71a6efd-config\") pod \"kube-apiserver-proxy-ip-10-0-128-246.ec2.internal\" (UID: \"a9ca977836d346c8c6c32a4fb71a6efd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal"
Jul 17 21:08:20.695644 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.695630    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:08:20.699121 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.699107    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:08:20.699176 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.699131    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:08:20.699176 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.699145    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:08:20.699176 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.699170    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-246.ec2.internal"
Jul 17 21:08:20.704842 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.704830    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-246.ec2.internal"
Jul 17 21:08:20.704882 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:20.704848    2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-246.ec2.internal\": node \"ip-10-0-128-246.ec2.internal\" not found"
Jul 17 21:08:20.719651 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:20.719633    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found"
Jul 17 21:08:20.791038 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.791018    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27af577f6839740a0585a4e6b9dd13dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" (UID: \"27af577f6839740a0585a4e6b9dd13dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal"
Jul 17 21:08:20.791122 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.791042    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27af577f6839740a0585a4e6b9dd13dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" (UID: \"27af577f6839740a0585a4e6b9dd13dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal"
Jul 17 21:08:20.791122 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.791060    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a9ca977836d346c8c6c32a4fb71a6efd-config\") pod \"kube-apiserver-proxy-ip-10-0-128-246.ec2.internal\" (UID: \"a9ca977836d346c8c6c32a4fb71a6efd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal"
Jul 17 21:08:20.791122 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.791112    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a9ca977836d346c8c6c32a4fb71a6efd-config\") pod \"kube-apiserver-proxy-ip-10-0-128-246.ec2.internal\" (UID: \"a9ca977836d346c8c6c32a4fb71a6efd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal"
Jul 17 21:08:20.791238 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.791113    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/27af577f6839740a0585a4e6b9dd13dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" (UID: \"27af577f6839740a0585a4e6b9dd13dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal"
Jul 17 21:08:20.791238 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.791149    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27af577f6839740a0585a4e6b9dd13dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal\" (UID: \"27af577f6839740a0585a4e6b9dd13dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal"
Jul 17 21:08:20.820154 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:20.820126    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found"
Jul 17 21:08:20.920926 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:20.920876    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found"
Jul 17 21:08:20.958047 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.958029    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal"
Jul 17 21:08:20.963845 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:20.963827    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal"
Jul 17 21:08:21.021439 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:21.021416    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found"
Jul 17 21:08:21.121937 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:21.121909    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found"
Jul 17 21:08:21.222372 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:21.222328    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found"
Jul 17 21:08:21.287040 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:21.287013    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:08:21.322397 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:21.322372    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found"
Jul 17 21:08:21.330117 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:21.330099    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 21:08:21.330222 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:21.330205    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 21:08:21.330267 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:21.330249    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 21:08:21.330316 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:21.330252    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 21:08:21.396244 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:21.396212    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 21:03:20 +0000 UTC" deadline="2028-01-19 21:51:20.084653782 +0000 UTC"
Jul 17 21:08:21.396244 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:21.396239    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13224h42m58.688417668s"
Jul 17 21:08:21.422887 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:21.422857    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found"
Jul 17 21:08:21.497715 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:21.497669    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ca977836d346c8c6c32a4fb71a6efd.slice/crio-404df2903d7d403d47625afdbcaf6304891427f60e062d04eb6d606477bd9edf WatchSource:0}: Error finding container 404df2903d7d403d47625afdbcaf6304891427f60e062d04eb6d606477bd9edf: Status 404 returned error can't find the container with id 404df2903d7d403d47625afdbcaf6304891427f60e062d04eb6d606477bd9edf
Jul 17 21:08:21.501264 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:21.501250    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 21:08:21.523494 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:21.523460    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found"
Jul 17 21:08:21.527631 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:21.527592    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" event={"ID":"a9ca977836d346c8c6c32a4fb71a6efd","Type":"ContainerStarted","Data":"404df2903d7d403d47625afdbcaf6304891427f60e062d04eb6d606477bd9edf"}
Jul 17 21:08:21.593077 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:21.593053    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27af577f6839740a0585a4e6b9dd13dc.slice/crio-32edcf473d5c06738f5da143ad9f3eff9883dc60f262997a2c6f94c322471829 WatchSource:0}: Error finding container 32edcf473d5c06738f5da143ad9f3eff9883dc60f262997a2c6f94c322471829: Status 404 returned error can't find the container with id 32edcf473d5c06738f5da143ad9f3eff9883dc60f262997a2c6f94c322471829
Jul 17 21:08:21.625168 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:21.625144    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found"
Jul 17 21:08:21.727249 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:21.727222    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-246.ec2.internal\" not found"
Jul 17 21:08:21.793651 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:21.793590    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:08:21.887541 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:21.887512    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal"
Jul 17 21:08:21.899681 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:21.899663    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 21:08:21.900600 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:21.900587    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal"
Jul 17 21:08:21.908623 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:21.908609    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 21:08:22.271034 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.270819    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:08:22.366014 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.365987    2573 apiserver.go:52] "Watching apiserver"
Jul 17 21:08:22.373976 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.373799    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 21:08:22.374209 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.374178    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-pd8zp","kube-system/global-pull-secret-syncer-6lpt4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq","openshift-cluster-node-tuning-operator/tuned-5tpb7","openshift-multus/network-metrics-daemon-87nv2","openshift-network-operator/iptables-alerter-cxpts","openshift-ovn-kubernetes/ovnkube-node-pkkqp","kube-system/konnectivity-agent-brktb","kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal","openshift-dns/node-resolver-xgwzb","openshift-image-registry/node-ca-6vr5m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal","openshift-multus/multus-additional-cni-plugins-99pvf","openshift-multus/multus-k9jkf"]
Jul 17 21:08:22.376380 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.376358    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.378466 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.378448    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.379087 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.379062    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 21:08:22.379395 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.379375    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 21:08:22.379481 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.379436    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 21:08:22.379726 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.379706    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.379799 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.379788    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 21:08:22.380128 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.380107    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-st79t\""
Jul 17 21:08:22.380250 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.380231    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 21:08:22.380393 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.380368    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 21:08:22.380989 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.380967    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dnvbk\""
Jul 17 21:08:22.381327 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.381309    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.381821 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.381801    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 21:08:22.381906 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.381885    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 21:08:22.381969 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.381905    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 21:08:22.382520 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.382068    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 21:08:22.382520 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.382182    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b86v4\""
Jul 17 21:08:22.382520 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.382273    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 21:08:22.382520 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.382304    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 21:08:22.382520 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.382380    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 21:08:22.382520 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.382399    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 21:08:22.383470 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.383186    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:22.383470 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:22.383270    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87nv2" podUID="4147724d-819d-4372-b2c8-c170b2d71411"
Jul 17 21:08:22.383656 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.383635    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mcghg\""
Jul 17 21:08:22.384082 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.384063    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 21:08:22.384429 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.384313    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:08:22.384727 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.384708    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cxpts"
Jul 17 21:08:22.387175 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.386779    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.388880 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.388839    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-przjx\""
Jul 17 21:08:22.389443 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.389415    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 21:08:22.389658 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.389634    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 21:08:22.389813 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.389793    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 21:08:22.390059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.390031    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:08:22.390176 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.390143    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:22.390274 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:22.390228    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd8zp" podUID="fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae"
Jul 17 21:08:22.392184 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.390096    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 21:08:22.392184 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.391607    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:22.392184 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:22.391711    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lpt4" podUID="ce0a4720-2597-4f74-8e8a-012be51716e8"
Jul 17 21:08:22.392184 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.391739    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qpx9f\""
Jul 17 21:08:22.393421 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.393404    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xgwzb"
Jul 17 21:08:22.394962 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.394942    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6vr5m"
Jul 17 21:08:22.395515 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.395493    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 21:08:22.395825 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.395807    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 21:08:22.395973 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.395888    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tppfj\""
Jul 17 21:08:22.397207 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.397186    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 21:08:22.397363 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.397345    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 21:08:22.397504 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.397481    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 21:08:22.397580 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.397565    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-brktb"
Jul 17 21:08:22.397634 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.397588    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7wgs9\""
Jul 17 21:08:22.400046 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.399999    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 21:08:22.400046 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.400031    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bnb28\""
Jul 17 21:08:22.400046 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.400004    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 21:08:22.401672 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401453    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-system-cni-dir\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.401672 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401485    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-host-run-netns\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.401672 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401509    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-env-overrides\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.401672 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401533    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2297d3f6-c109-4b3d-883c-1b0a411f9512-cnibin\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.401672 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401559    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a73e4522-94be-43de-87c3-e0db2654973f-registration-dir\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.401672 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401587    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce0a4720-2597-4f74-8e8a-012be51716e8-dbus\") pod \"global-pull-secret-syncer-6lpt4\" (UID: \"ce0a4720-2597-4f74-8e8a-012be51716e8\") " pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:22.401672 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401624    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvdrb\" (UniqueName: \"kubernetes.io/projected/4147724d-819d-4372-b2c8-c170b2d71411-kube-api-access-bvdrb\") pod \"network-metrics-daemon-87nv2\" (UID: \"4147724d-819d-4372-b2c8-c170b2d71411\") " pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:22.401672 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401659    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-systemd-units\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.402135 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401690    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxdbm\" (UniqueName: \"kubernetes.io/projected/cafe7b42-f3df-4284-8b7f-aa88520c1f28-kube-api-access-dxdbm\") pod \"iptables-alerter-cxpts\" (UID: \"cafe7b42-f3df-4284-8b7f-aa88520c1f28\") " pod="openshift-network-operator/iptables-alerter-cxpts"
Jul 17 21:08:22.402135 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401713    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-etc-systemd\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.402135 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401735    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2297d3f6-c109-4b3d-883c-1b0a411f9512-tuning-conf-dir\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.402135 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401768    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-run-netns\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.402135 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401790    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-log-socket\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.402135 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401816    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-ovn-node-metrics-cert\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.402135 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401840    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a73e4522-94be-43de-87c3-e0db2654973f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.402135 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401874    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-etc-kubernetes\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.402135 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401897    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-etc-sysctl-conf\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.402135 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401923    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.402135 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401947    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-sys\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.402135 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401972    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/782caf5b-542f-45e1-b011-263374137dcd-etc-tuned\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.402135 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.401996    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-cnibin\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.402135 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402022    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-host-run-k8s-cni-cncf-io\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.402135 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402045    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-host-var-lib-kubelet\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.402135 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402070    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a73e4522-94be-43de-87c3-e0db2654973f-etc-selinux\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.402932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402094    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a73e4522-94be-43de-87c3-e0db2654973f-sys-fs\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.402932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402121    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce0a4720-2597-4f74-8e8a-012be51716e8-kubelet-config\") pod \"global-pull-secret-syncer-6lpt4\" (UID: \"ce0a4720-2597-4f74-8e8a-012be51716e8\") " pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:22.402932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402144    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-etc-modprobe-d\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.402932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402169    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-etc-sysconfig\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.402932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402194    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-host\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.402932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402217    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-var-lib-openvswitch\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.402932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402243    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-run-ovn\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.402932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402276    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2297d3f6-c109-4b3d-883c-1b0a411f9512-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.402932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402302    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a73e4522-94be-43de-87c3-e0db2654973f-socket-dir\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.402932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402327    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f682d\" (UniqueName: \"kubernetes.io/projected/782caf5b-542f-45e1-b011-263374137dcd-kube-api-access-f682d\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.402932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402368    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72709796-e935-4cba-a007-cc17fb595e10-cni-binary-copy\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.402932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402394    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-node-log\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.402932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402419    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a73e4522-94be-43de-87c3-e0db2654973f-device-dir\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.402932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402450    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-run\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.402932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402474    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8zw6\" (UniqueName: \"kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6\") pod \"network-check-target-pd8zp\" (UID: \"fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae\") " pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:22.402932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402503    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cafe7b42-f3df-4284-8b7f-aa88520c1f28-iptables-alerter-script\") pod \"iptables-alerter-cxpts\" (UID: \"cafe7b42-f3df-4284-8b7f-aa88520c1f28\") " pod="openshift-network-operator/iptables-alerter-cxpts"
Jul 17 21:08:22.403736 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402532    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs\") pod \"network-metrics-daemon-87nv2\" (UID: \"4147724d-819d-4372-b2c8-c170b2d71411\") " pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:22.403736 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402561    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-etc-openvswitch\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.403736 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402587    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-run-ovn-kubernetes\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.403736 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402647    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2297d3f6-c109-4b3d-883c-1b0a411f9512-system-cni-dir\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.403736 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402676    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2297d3f6-c109-4b3d-883c-1b0a411f9512-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.403736 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402703    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfc9r\" (UniqueName: \"kubernetes.io/projected/2297d3f6-c109-4b3d-883c-1b0a411f9512-kube-api-access-vfc9r\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.403736 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402730    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-host-run-multus-certs\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.403736 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402755    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-cni-netd\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.403736 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402783    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-ovnkube-config\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.403736 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402940    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-ovnkube-script-lib\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.403736 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.402981    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2297d3f6-c109-4b3d-883c-1b0a411f9512-os-release\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.403736 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403016    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-multus-cni-dir\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.403736 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403044    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-os-release\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.403736 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403075    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-cni-bin\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.403736 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403115    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-host-var-lib-cni-bin\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.403736 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403147    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-hostroot\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.404568 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403173    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sjld\" (UniqueName: \"kubernetes.io/projected/72709796-e935-4cba-a007-cc17fb595e10-kube-api-access-2sjld\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.404568 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403207    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76rj4\" (UniqueName: \"kubernetes.io/projected/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-kube-api-access-76rj4\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.404568 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403232    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-lib-modules\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.404568 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403258    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/782caf5b-542f-45e1-b011-263374137dcd-tmp\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.404568 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403283    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-multus-socket-dir-parent\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.404568 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403309    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-multus-conf-dir\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.404568 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403336    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-run-systemd\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.404568 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403378    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/72709796-e935-4cba-a007-cc17fb595e10-multus-daemon-config\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.404568 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403420    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-etc-kubernetes\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.404568 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403446    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-run-openvswitch\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.404568 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403471    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cafe7b42-f3df-4284-8b7f-aa88520c1f28-host-slash\") pod \"iptables-alerter-cxpts\" (UID: \"cafe7b42-f3df-4284-8b7f-aa88520c1f28\") " pod="openshift-network-operator/iptables-alerter-cxpts"
Jul 17 21:08:22.404568 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403497    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-etc-sysctl-d\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.404568 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403522    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-var-lib-kubelet\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.404568 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403549    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-host-var-lib-cni-multus\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.404568 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403601    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2297d3f6-c109-4b3d-883c-1b0a411f9512-cni-binary-copy\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.404568 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403660    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-kubelet\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.404568 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403715    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-slash\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.405480 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403741    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptp28\" (UniqueName: \"kubernetes.io/projected/a73e4522-94be-43de-87c3-e0db2654973f-kube-api-access-ptp28\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.405480 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.403777    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret\") pod \"global-pull-secret-syncer-6lpt4\" (UID: \"ce0a4720-2597-4f74-8e8a-012be51716e8\") " pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:22.405480 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.404851    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 21:08:22.429628 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.429605    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n7gn7"
Jul 17 21:08:22.438457 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.438435    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n7gn7"
Jul 17 21:08:22.441050 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.441031    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:08:22.488842 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.488810    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 21:08:22.504339 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504311    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76rj4\" (UniqueName: \"kubernetes.io/projected/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-kube-api-access-76rj4\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.504590 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504353    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-lib-modules\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.504590 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504384    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/782caf5b-542f-45e1-b011-263374137dcd-tmp\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.504590 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504411    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-multus-socket-dir-parent\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.504590 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504436    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-multus-conf-dir\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.504590 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504465    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35528beb-a7c1-446b-bfae-6eb51d69a48c-tmp-dir\") pod \"node-resolver-xgwzb\" (UID: \"35528beb-a7c1-446b-bfae-6eb51d69a48c\") " pod="openshift-dns/node-resolver-xgwzb"
Jul 17 21:08:22.504590 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504509    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-run-systemd\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.504590 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504533    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/72709796-e935-4cba-a007-cc17fb595e10-multus-daemon-config\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.504590 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504558    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-etc-kubernetes\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.504999 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504593    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6205c591-20ac-4b53-aa08-656d2aecc409-konnectivity-ca\") pod \"konnectivity-agent-brktb\" (UID: \"6205c591-20ac-4b53-aa08-656d2aecc409\") " pod="kube-system/konnectivity-agent-brktb"
Jul 17 21:08:22.504999 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504626    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-run-openvswitch\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.504999 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504653    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cafe7b42-f3df-4284-8b7f-aa88520c1f28-host-slash\") pod \"iptables-alerter-cxpts\" (UID: \"cafe7b42-f3df-4284-8b7f-aa88520c1f28\") " pod="openshift-network-operator/iptables-alerter-cxpts"
Jul 17 21:08:22.504999 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504680    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-etc-sysctl-d\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.504999 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504734    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-run-systemd\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.504999 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504768    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-var-lib-kubelet\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.504999 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504786    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-run-openvswitch\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.504999 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504811    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-host-var-lib-cni-multus\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.504999 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504810    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 21:08:22.504999 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504842    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxjmv\" (UniqueName: \"kubernetes.io/projected/35528beb-a7c1-446b-bfae-6eb51d69a48c-kube-api-access-vxjmv\") pod \"node-resolver-xgwzb\" (UID: \"35528beb-a7c1-446b-bfae-6eb51d69a48c\") " pod="openshift-dns/node-resolver-xgwzb"
Jul 17 21:08:22.505465 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.504844    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-multus-socket-dir-parent\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.505714 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.505688    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-multus-conf-dir\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.505812 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.505799    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-lib-modules\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.505994 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.505968    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-etc-kubernetes\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.506052 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.505972    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2297d3f6-c109-4b3d-883c-1b0a411f9512-cni-binary-copy\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.506100 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506078    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/72709796-e935-4cba-a007-cc17fb595e10-multus-daemon-config\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.506100 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506085    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-kubelet\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.506200 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506124    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-slash\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.506200 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506177    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptp28\" (UniqueName: \"kubernetes.io/projected/a73e4522-94be-43de-87c3-e0db2654973f-kube-api-access-ptp28\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.506316 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506212    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-kubelet\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.506316 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506216    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret\") pod \"global-pull-secret-syncer-6lpt4\" (UID: \"ce0a4720-2597-4f74-8e8a-012be51716e8\") " pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:22.506316 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506286    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-system-cni-dir\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.506316 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506274    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-slash\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.506548 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506344    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-host-run-netns\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.506548 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506373    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-system-cni-dir\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.506548 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506385    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-env-overrides\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.506548 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506460    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-host-var-lib-cni-multus\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.506548 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506472    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-var-lib-kubelet\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.506548 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506498    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-etc-sysctl-d\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.506548 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506511    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2297d3f6-c109-4b3d-883c-1b0a411f9512-cnibin\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.506906 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:22.506558    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:22.506906 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506578    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cafe7b42-f3df-4284-8b7f-aa88520c1f28-host-slash\") pod \"iptables-alerter-cxpts\" (UID: \"cafe7b42-f3df-4284-8b7f-aa88520c1f28\") " pod="openshift-network-operator/iptables-alerter-cxpts"
Jul 17 21:08:22.506906 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506581    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-host-run-netns\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.506906 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506624    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2297d3f6-c109-4b3d-883c-1b0a411f9512-cni-binary-copy\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.506906 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506624    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a73e4522-94be-43de-87c3-e0db2654973f-registration-dir\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.506906 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:22.506696    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret podName:ce0a4720-2597-4f74-8e8a-012be51716e8 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:23.006640785 +0000 UTC m=+2.970761395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret") pod "global-pull-secret-syncer-6lpt4" (UID: "ce0a4720-2597-4f74-8e8a-012be51716e8") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:22.506906 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506739    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a73e4522-94be-43de-87c3-e0db2654973f-registration-dir\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.506906 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506744    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2297d3f6-c109-4b3d-883c-1b0a411f9512-cnibin\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.506906 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506793    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce0a4720-2597-4f74-8e8a-012be51716e8-dbus\") pod \"global-pull-secret-syncer-6lpt4\" (UID: \"ce0a4720-2597-4f74-8e8a-012be51716e8\") " pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:22.506906 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506845    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvdrb\" (UniqueName: \"kubernetes.io/projected/4147724d-819d-4372-b2c8-c170b2d71411-kube-api-access-bvdrb\") pod \"network-metrics-daemon-87nv2\" (UID: \"4147724d-819d-4372-b2c8-c170b2d71411\") " pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:22.506906 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506886    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-systemd-units\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.507469 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.506941    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxdbm\" (UniqueName: \"kubernetes.io/projected/cafe7b42-f3df-4284-8b7f-aa88520c1f28-kube-api-access-dxdbm\") pod \"iptables-alerter-cxpts\" (UID: \"cafe7b42-f3df-4284-8b7f-aa88520c1f28\") " pod="openshift-network-operator/iptables-alerter-cxpts"
Jul 17 21:08:22.507469 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507084    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ce0a4720-2597-4f74-8e8a-012be51716e8-dbus\") pod \"global-pull-secret-syncer-6lpt4\" (UID: \"ce0a4720-2597-4f74-8e8a-012be51716e8\") " pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:22.507469 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507119    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-systemd-units\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.507469 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507141    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-env-overrides\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.507469 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507153    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-etc-systemd\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.507469 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507201    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-etc-systemd\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.507469 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507193    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9d08404-9306-407f-b32a-a0e2d25473b1-host\") pod \"node-ca-6vr5m\" (UID: \"d9d08404-9306-407f-b32a-a0e2d25473b1\") " pod="openshift-image-registry/node-ca-6vr5m"
Jul 17 21:08:22.507469 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507249    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2297d3f6-c109-4b3d-883c-1b0a411f9512-tuning-conf-dir\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.507469 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507338    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-run-netns\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.507469 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507375    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-log-socket\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.507469 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507408    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-ovn-node-metrics-cert\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.507469 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507462    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a73e4522-94be-43de-87c3-e0db2654973f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.508049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507488    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-run-netns\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.508049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507515    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2297d3f6-c109-4b3d-883c-1b0a411f9512-tuning-conf-dir\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.508049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507599    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-etc-kubernetes\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.508049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507642    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a73e4522-94be-43de-87c3-e0db2654973f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.508049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507601    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-log-socket\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.508049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507669    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-etc-sysctl-conf\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.508049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507688    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-etc-kubernetes\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.508049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507733    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.508049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507766    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-sys\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.508049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507795    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/782caf5b-542f-45e1-b011-263374137dcd-etc-tuned\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.508049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507854    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-cnibin\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.508049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507879    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-host-run-k8s-cni-cncf-io\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.508049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507881    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-etc-sysctl-conf\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.508049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507909    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-host-var-lib-kubelet\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.508049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507955    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a73e4522-94be-43de-87c3-e0db2654973f-etc-selinux\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.508049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.507988    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.508049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508000    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a73e4522-94be-43de-87c3-e0db2654973f-sys-fs\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.508899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508030    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce0a4720-2597-4f74-8e8a-012be51716e8-kubelet-config\") pod \"global-pull-secret-syncer-6lpt4\" (UID: \"ce0a4720-2597-4f74-8e8a-012be51716e8\") " pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:22.508899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508066    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-sys\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.508899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508127    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-host-var-lib-kubelet\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.508899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508168    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-etc-modprobe-d\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.508899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508181    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ce0a4720-2597-4f74-8e8a-012be51716e8-kubelet-config\") pod \"global-pull-secret-syncer-6lpt4\" (UID: \"ce0a4720-2597-4f74-8e8a-012be51716e8\") " pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:22.508899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508209    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-etc-sysconfig\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.508899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508223    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-host-run-k8s-cni-cncf-io\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.508899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508242    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-host\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.508899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508276    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6205c591-20ac-4b53-aa08-656d2aecc409-agent-certs\") pod \"konnectivity-agent-brktb\" (UID: \"6205c591-20ac-4b53-aa08-656d2aecc409\") " pod="kube-system/konnectivity-agent-brktb"
Jul 17 21:08:22.508899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508304    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-var-lib-openvswitch\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.508899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508350    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a73e4522-94be-43de-87c3-e0db2654973f-sys-fs\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.508899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508356    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-run-ovn\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.508899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508369    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a73e4522-94be-43de-87c3-e0db2654973f-etc-selinux\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.508899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508171    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-cnibin\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.508899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508430    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-run-ovn\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.508899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508475    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2297d3f6-c109-4b3d-883c-1b0a411f9512-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.508899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508514    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a73e4522-94be-43de-87c3-e0db2654973f-socket-dir\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.509746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508519    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-etc-modprobe-d\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.509746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508523    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-etc-sysconfig\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.509746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508583    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f682d\" (UniqueName: \"kubernetes.io/projected/782caf5b-542f-45e1-b011-263374137dcd-kube-api-access-f682d\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.509746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508616    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72709796-e935-4cba-a007-cc17fb595e10-cni-binary-copy\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.509746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508587    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-host\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.509746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.508644    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a73e4522-94be-43de-87c3-e0db2654973f-socket-dir\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.509746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509041    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-var-lib-openvswitch\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.509746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509092    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82zqr\" (UniqueName: \"kubernetes.io/projected/d9d08404-9306-407f-b32a-a0e2d25473b1-kube-api-access-82zqr\") pod \"node-ca-6vr5m\" (UID: \"d9d08404-9306-407f-b32a-a0e2d25473b1\") " pod="openshift-image-registry/node-ca-6vr5m"
Jul 17 21:08:22.509746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509139    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-node-log\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.509746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509171    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a73e4522-94be-43de-87c3-e0db2654973f-device-dir\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.509746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509204    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-run\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.509746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509203    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72709796-e935-4cba-a007-cc17fb595e10-cni-binary-copy\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.509746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509213    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2297d3f6-c109-4b3d-883c-1b0a411f9512-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.509746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509236    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zw6\" (UniqueName: \"kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6\") pod \"network-check-target-pd8zp\" (UID: \"fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae\") " pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:22.509746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509267    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cafe7b42-f3df-4284-8b7f-aa88520c1f28-iptables-alerter-script\") pod \"iptables-alerter-cxpts\" (UID: \"cafe7b42-f3df-4284-8b7f-aa88520c1f28\") " pod="openshift-network-operator/iptables-alerter-cxpts"
Jul 17 21:08:22.509746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509283    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-node-log\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.509746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509301    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs\") pod \"network-metrics-daemon-87nv2\" (UID: \"4147724d-819d-4372-b2c8-c170b2d71411\") " pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:22.510598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509316    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/782caf5b-542f-45e1-b011-263374137dcd-run\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.510598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509409    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a73e4522-94be-43de-87c3-e0db2654973f-device-dir\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.510598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509464    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d9d08404-9306-407f-b32a-a0e2d25473b1-serviceca\") pod \"node-ca-6vr5m\" (UID: \"d9d08404-9306-407f-b32a-a0e2d25473b1\") " pod="openshift-image-registry/node-ca-6vr5m"
Jul 17 21:08:22.510598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509501    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-etc-openvswitch\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.510598 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:22.509468    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:22.510598 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:22.509775    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs podName:4147724d-819d-4372-b2c8-c170b2d71411 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:23.009734029 +0000 UTC m=+2.973854620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs") pod "network-metrics-daemon-87nv2" (UID: "4147724d-819d-4372-b2c8-c170b2d71411") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:22.510598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509849    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-etc-openvswitch\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.510598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509873    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cafe7b42-f3df-4284-8b7f-aa88520c1f28-iptables-alerter-script\") pod \"iptables-alerter-cxpts\" (UID: \"cafe7b42-f3df-4284-8b7f-aa88520c1f28\") " pod="openshift-network-operator/iptables-alerter-cxpts"
Jul 17 21:08:22.510598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509975    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-run-ovn-kubernetes\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.510598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.509536    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-run-ovn-kubernetes\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.510598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.510061    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2297d3f6-c109-4b3d-883c-1b0a411f9512-system-cni-dir\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.510598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.510096    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2297d3f6-c109-4b3d-883c-1b0a411f9512-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.510598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.510125    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfc9r\" (UniqueName: \"kubernetes.io/projected/2297d3f6-c109-4b3d-883c-1b0a411f9512-kube-api-access-vfc9r\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.510598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.510153    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-host-run-multus-certs\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.510598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.510180    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-cni-netd\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.510598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.510206    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-ovnkube-config\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.510598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.510234    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-ovnkube-script-lib\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.511447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.510265    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2297d3f6-c109-4b3d-883c-1b0a411f9512-os-release\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.511447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.510295    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-multus-cni-dir\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.511447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.510321    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-os-release\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.511447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.510347    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35528beb-a7c1-446b-bfae-6eb51d69a48c-hosts-file\") pod \"node-resolver-xgwzb\" (UID: \"35528beb-a7c1-446b-bfae-6eb51d69a48c\") " pod="openshift-dns/node-resolver-xgwzb"
Jul 17 21:08:22.511447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.510403    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-cni-bin\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.511447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.510438    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-host-var-lib-cni-bin\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.511447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.510470    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-hostroot\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.511447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.510497    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sjld\" (UniqueName: \"kubernetes.io/projected/72709796-e935-4cba-a007-cc17fb595e10-kube-api-access-2sjld\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.511447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.510825    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2297d3f6-c109-4b3d-883c-1b0a411f9512-system-cni-dir\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.511447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.511250    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2297d3f6-c109-4b3d-883c-1b0a411f9512-os-release\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.511447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.511323    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2297d3f6-c109-4b3d-883c-1b0a411f9512-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.511447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.511448    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-multus-cni-dir\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.512124 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.511515    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-os-release\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.512124 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.511577    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-host-run-multus-certs\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.512124 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.511610    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-cni-bin\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.512124 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.511635    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-host-cni-netd\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.512124 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.511663    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-host-var-lib-cni-bin\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.512124 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.511713    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/72709796-e935-4cba-a007-cc17fb595e10-hostroot\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.512433 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.512134    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-ovnkube-config\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.512946 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.512870    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-ovnkube-script-lib\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.515723 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.515696    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/782caf5b-542f-45e1-b011-263374137dcd-tmp\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.515829 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.515722    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-ovn-node-metrics-cert\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.518515 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:22.518495    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:08:22.518605 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:22.518524    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:08:22.518605 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:22.518538    2573 projected.go:196] Error preparing data for projected volume kube-api-access-w8zw6 for pod openshift-network-diagnostics/network-check-target-pd8zp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:22.518735 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:22.518606    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6 podName:fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae nodeName:}" failed. No retries permitted until 2026-07-17 21:08:23.018588056 +0000 UTC m=+2.982708666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w8zw6" (UniqueName: "kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6") pod "network-check-target-pd8zp" (UID: "fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:22.518806 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.518755    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/782caf5b-542f-45e1-b011-263374137dcd-etc-tuned\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.518864 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.518809    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvdrb\" (UniqueName: \"kubernetes.io/projected/4147724d-819d-4372-b2c8-c170b2d71411-kube-api-access-bvdrb\") pod \"network-metrics-daemon-87nv2\" (UID: \"4147724d-819d-4372-b2c8-c170b2d71411\") " pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:22.520829 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.520803    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxdbm\" (UniqueName: \"kubernetes.io/projected/cafe7b42-f3df-4284-8b7f-aa88520c1f28-kube-api-access-dxdbm\") pod \"iptables-alerter-cxpts\" (UID: \"cafe7b42-f3df-4284-8b7f-aa88520c1f28\") " pod="openshift-network-operator/iptables-alerter-cxpts"
Jul 17 21:08:22.521103 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.521046    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sjld\" (UniqueName: \"kubernetes.io/projected/72709796-e935-4cba-a007-cc17fb595e10-kube-api-access-2sjld\") pod \"multus-k9jkf\" (UID: \"72709796-e935-4cba-a007-cc17fb595e10\") " pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.521627 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.521494    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f682d\" (UniqueName: \"kubernetes.io/projected/782caf5b-542f-45e1-b011-263374137dcd-kube-api-access-f682d\") pod \"tuned-5tpb7\" (UID: \"782caf5b-542f-45e1-b011-263374137dcd\") " pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.521754 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.521731    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76rj4\" (UniqueName: \"kubernetes.io/projected/f2cd4bc8-a91f-4576-9945-534f6ca00bd6-kube-api-access-76rj4\") pod \"ovnkube-node-pkkqp\" (UID: \"f2cd4bc8-a91f-4576-9945-534f6ca00bd6\") " pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.523094 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.523060    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptp28\" (UniqueName: \"kubernetes.io/projected/a73e4522-94be-43de-87c3-e0db2654973f-kube-api-access-ptp28\") pod \"aws-ebs-csi-driver-node-92xmq\" (UID: \"a73e4522-94be-43de-87c3-e0db2654973f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.523906 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.523878    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfc9r\" (UniqueName: \"kubernetes.io/projected/2297d3f6-c109-4b3d-883c-1b0a411f9512-kube-api-access-vfc9r\") pod \"multus-additional-cni-plugins-99pvf\" (UID: \"2297d3f6-c109-4b3d-883c-1b0a411f9512\") " pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.529938 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.529911    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" event={"ID":"27af577f6839740a0585a4e6b9dd13dc","Type":"ContainerStarted","Data":"32edcf473d5c06738f5da143ad9f3eff9883dc60f262997a2c6f94c322471829"}
Jul 17 21:08:22.611659 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.611589    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d9d08404-9306-407f-b32a-a0e2d25473b1-serviceca\") pod \"node-ca-6vr5m\" (UID: \"d9d08404-9306-407f-b32a-a0e2d25473b1\") " pod="openshift-image-registry/node-ca-6vr5m"
Jul 17 21:08:22.611659 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.611664    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35528beb-a7c1-446b-bfae-6eb51d69a48c-hosts-file\") pod \"node-resolver-xgwzb\" (UID: \"35528beb-a7c1-446b-bfae-6eb51d69a48c\") " pod="openshift-dns/node-resolver-xgwzb"
Jul 17 21:08:22.611884 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.611694    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35528beb-a7c1-446b-bfae-6eb51d69a48c-tmp-dir\") pod \"node-resolver-xgwzb\" (UID: \"35528beb-a7c1-446b-bfae-6eb51d69a48c\") " pod="openshift-dns/node-resolver-xgwzb"
Jul 17 21:08:22.611884 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.611783    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/35528beb-a7c1-446b-bfae-6eb51d69a48c-hosts-file\") pod \"node-resolver-xgwzb\" (UID: \"35528beb-a7c1-446b-bfae-6eb51d69a48c\") " pod="openshift-dns/node-resolver-xgwzb"
Jul 17 21:08:22.611884 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.611822    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6205c591-20ac-4b53-aa08-656d2aecc409-konnectivity-ca\") pod \"konnectivity-agent-brktb\" (UID: \"6205c591-20ac-4b53-aa08-656d2aecc409\") " pod="kube-system/konnectivity-agent-brktb"
Jul 17 21:08:22.611884 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.611873    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxjmv\" (UniqueName: \"kubernetes.io/projected/35528beb-a7c1-446b-bfae-6eb51d69a48c-kube-api-access-vxjmv\") pod \"node-resolver-xgwzb\" (UID: \"35528beb-a7c1-446b-bfae-6eb51d69a48c\") " pod="openshift-dns/node-resolver-xgwzb"
Jul 17 21:08:22.612167 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.612144    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35528beb-a7c1-446b-bfae-6eb51d69a48c-tmp-dir\") pod \"node-resolver-xgwzb\" (UID: \"35528beb-a7c1-446b-bfae-6eb51d69a48c\") " pod="openshift-dns/node-resolver-xgwzb"
Jul 17 21:08:22.612240 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.612167    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9d08404-9306-407f-b32a-a0e2d25473b1-host\") pod \"node-ca-6vr5m\" (UID: \"d9d08404-9306-407f-b32a-a0e2d25473b1\") " pod="openshift-image-registry/node-ca-6vr5m"
Jul 17 21:08:22.612240 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.612211    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6205c591-20ac-4b53-aa08-656d2aecc409-agent-certs\") pod \"konnectivity-agent-brktb\" (UID: \"6205c591-20ac-4b53-aa08-656d2aecc409\") " pod="kube-system/konnectivity-agent-brktb"
Jul 17 21:08:22.612240 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.612214    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d9d08404-9306-407f-b32a-a0e2d25473b1-serviceca\") pod \"node-ca-6vr5m\" (UID: \"d9d08404-9306-407f-b32a-a0e2d25473b1\") " pod="openshift-image-registry/node-ca-6vr5m"
Jul 17 21:08:22.612413 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.612243    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82zqr\" (UniqueName: \"kubernetes.io/projected/d9d08404-9306-407f-b32a-a0e2d25473b1-kube-api-access-82zqr\") pod \"node-ca-6vr5m\" (UID: \"d9d08404-9306-407f-b32a-a0e2d25473b1\") " pod="openshift-image-registry/node-ca-6vr5m"
Jul 17 21:08:22.612413 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.612286    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9d08404-9306-407f-b32a-a0e2d25473b1-host\") pod \"node-ca-6vr5m\" (UID: \"d9d08404-9306-407f-b32a-a0e2d25473b1\") " pod="openshift-image-registry/node-ca-6vr5m"
Jul 17 21:08:22.612486 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.612443    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6205c591-20ac-4b53-aa08-656d2aecc409-konnectivity-ca\") pod \"konnectivity-agent-brktb\" (UID: \"6205c591-20ac-4b53-aa08-656d2aecc409\") " pod="kube-system/konnectivity-agent-brktb"
Jul 17 21:08:22.615057 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.615031    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6205c591-20ac-4b53-aa08-656d2aecc409-agent-certs\") pod \"konnectivity-agent-brktb\" (UID: \"6205c591-20ac-4b53-aa08-656d2aecc409\") " pod="kube-system/konnectivity-agent-brktb"
Jul 17 21:08:22.620992 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.620968    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxjmv\" (UniqueName: \"kubernetes.io/projected/35528beb-a7c1-446b-bfae-6eb51d69a48c-kube-api-access-vxjmv\") pod \"node-resolver-xgwzb\" (UID: \"35528beb-a7c1-446b-bfae-6eb51d69a48c\") " pod="openshift-dns/node-resolver-xgwzb"
Jul 17 21:08:22.621559 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.621534    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82zqr\" (UniqueName: \"kubernetes.io/projected/d9d08404-9306-407f-b32a-a0e2d25473b1-kube-api-access-82zqr\") pod \"node-ca-6vr5m\" (UID: \"d9d08404-9306-407f-b32a-a0e2d25473b1\") " pod="openshift-image-registry/node-ca-6vr5m"
Jul 17 21:08:22.696630 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.696602    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:22.703706 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.703679    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-99pvf"
Jul 17 21:08:22.713751 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.713590    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq"
Jul 17 21:08:22.721447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.721419    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5tpb7"
Jul 17 21:08:22.731217 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.731194    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cxpts"
Jul 17 21:08:22.739271 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.739241    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k9jkf"
Jul 17 21:08:22.748251 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.748232    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xgwzb"
Jul 17 21:08:22.755100 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.755079    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6vr5m"
Jul 17 21:08:22.762234 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:22.762214    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-brktb"
Jul 17 21:08:23.015767 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:23.015690    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs\") pod \"network-metrics-daemon-87nv2\" (UID: \"4147724d-819d-4372-b2c8-c170b2d71411\") " pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:23.015767 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:23.015752    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret\") pod \"global-pull-secret-syncer-6lpt4\" (UID: \"ce0a4720-2597-4f74-8e8a-012be51716e8\") " pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:23.015986 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:23.015856    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:23.015986 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:23.015942    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs podName:4147724d-819d-4372-b2c8-c170b2d71411 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:24.015919088 +0000 UTC m=+3.980039717 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs") pod "network-metrics-daemon-87nv2" (UID: "4147724d-819d-4372-b2c8-c170b2d71411") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:23.015986 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:23.015861    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:23.016150 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:23.016036    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret podName:ce0a4720-2597-4f74-8e8a-012be51716e8 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:24.016014826 +0000 UTC m=+3.980135422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret") pod "global-pull-secret-syncer-6lpt4" (UID: "ce0a4720-2597-4f74-8e8a-012be51716e8") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:23.116301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:23.116260    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zw6\" (UniqueName: \"kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6\") pod \"network-check-target-pd8zp\" (UID: \"fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae\") " pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:23.116481 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:23.116426    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:08:23.116481 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:23.116457    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:08:23.116481 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:23.116472    2573 projected.go:196] Error preparing data for projected volume kube-api-access-w8zw6 for pod openshift-network-diagnostics/network-check-target-pd8zp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:23.116599 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:23.116537    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6 podName:fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae nodeName:}" failed. No retries permitted until 2026-07-17 21:08:24.116520355 +0000 UTC m=+4.080640945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w8zw6" (UniqueName: "kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6") pod "network-check-target-pd8zp" (UID: "fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:23.441864 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:23.441761    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 21:03:22 +0000 UTC" deadline="2028-03-16 01:16:30.939404854 +0000 UTC"
Jul 17 21:08:23.441864 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:23.441801    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14572h8m7.497608065s"
Jul 17 21:08:23.536729 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:23.536697    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafe7b42_f3df_4284_8b7f_aa88520c1f28.slice/crio-a6c81d79d010307689de1c4c01635b043566ef3854ec361c572453853e8e070e WatchSource:0}: Error finding container a6c81d79d010307689de1c4c01635b043566ef3854ec361c572453853e8e070e: Status 404 returned error can't find the container with id a6c81d79d010307689de1c4c01635b043566ef3854ec361c572453853e8e070e
Jul 17 21:08:23.538238 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:23.538123    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782caf5b_542f_45e1_b011_263374137dcd.slice/crio-f880f0075c6be2f2a8c30cfdc130eb9368f8f9c7f7b6b3d1e4bd6eec142fed3f WatchSource:0}: Error finding container f880f0075c6be2f2a8c30cfdc130eb9368f8f9c7f7b6b3d1e4bd6eec142fed3f: Status 404 returned error can't find the container with id f880f0075c6be2f2a8c30cfdc130eb9368f8f9c7f7b6b3d1e4bd6eec142fed3f
Jul 17 21:08:23.540149 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:23.540105    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2cd4bc8_a91f_4576_9945_534f6ca00bd6.slice/crio-fcfe106721099278b9d03b1b71f694e84031895b6733cba819ce619fd279df46 WatchSource:0}: Error finding container fcfe106721099278b9d03b1b71f694e84031895b6733cba819ce619fd279df46: Status 404 returned error can't find the container with id fcfe106721099278b9d03b1b71f694e84031895b6733cba819ce619fd279df46
Jul 17 21:08:23.542831 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:23.542811    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d08404_9306_407f_b32a_a0e2d25473b1.slice/crio-1850d7ab36f4f62d8752151d496d7877f277fc40c63432d943a9849e38748e77 WatchSource:0}: Error finding container 1850d7ab36f4f62d8752151d496d7877f277fc40c63432d943a9849e38748e77: Status 404 returned error can't find the container with id 1850d7ab36f4f62d8752151d496d7877f277fc40c63432d943a9849e38748e77
Jul 17 21:08:23.544434 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:23.544354    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2297d3f6_c109_4b3d_883c_1b0a411f9512.slice/crio-2f0a28abbfb181afb684a1cc67fedd867d57cabccec66a94f25531b192d4f028 WatchSource:0}: Error finding container 2f0a28abbfb181afb684a1cc67fedd867d57cabccec66a94f25531b192d4f028: Status 404 returned error can't find the container with id 2f0a28abbfb181afb684a1cc67fedd867d57cabccec66a94f25531b192d4f028
Jul 17 21:08:23.546423 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:23.546132    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35528beb_a7c1_446b_bfae_6eb51d69a48c.slice/crio-d2dbcdfc263e5843911f2ca8b43c1c586b954cfb89774ae1989a99e13118dbda WatchSource:0}: Error finding container d2dbcdfc263e5843911f2ca8b43c1c586b954cfb89774ae1989a99e13118dbda: Status 404 returned error can't find the container with id d2dbcdfc263e5843911f2ca8b43c1c586b954cfb89774ae1989a99e13118dbda
Jul 17 21:08:23.547030 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:23.546914    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6205c591_20ac_4b53_aa08_656d2aecc409.slice/crio-bb05ba7642416e9f63584fac7b84a7938dbd4c14231bd5048614fd8d0d788dab WatchSource:0}: Error finding container bb05ba7642416e9f63584fac7b84a7938dbd4c14231bd5048614fd8d0d788dab: Status 404 returned error can't find the container with id bb05ba7642416e9f63584fac7b84a7938dbd4c14231bd5048614fd8d0d788dab
Jul 17 21:08:23.548550 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:23.548495    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72709796_e935_4cba_a007_cc17fb595e10.slice/crio-6bfeb635032e59020ef492466423a1cde45a39ed0bc1c71ddbabebfceaf19c2e WatchSource:0}: Error finding container 6bfeb635032e59020ef492466423a1cde45a39ed0bc1c71ddbabebfceaf19c2e: Status 404 returned error can't find the container with id 6bfeb635032e59020ef492466423a1cde45a39ed0bc1c71ddbabebfceaf19c2e
Jul 17 21:08:23.549655 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:08:23.549504    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73e4522_94be_43de_87c3_e0db2654973f.slice/crio-185da094d7d126361a535df8999de35c2b7e2f0330e68fe6ab9b6d59c9743040 WatchSource:0}: Error finding container 185da094d7d126361a535df8999de35c2b7e2f0330e68fe6ab9b6d59c9743040: Status 404 returned error can't find the container with id 185da094d7d126361a535df8999de35c2b7e2f0330e68fe6ab9b6d59c9743040
Jul 17 21:08:23.977500 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:23.977192    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:08:24.023122 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.023087    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret\") pod \"global-pull-secret-syncer-6lpt4\" (UID: \"ce0a4720-2597-4f74-8e8a-012be51716e8\") " pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:24.023305 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.023172    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs\") pod \"network-metrics-daemon-87nv2\" (UID: \"4147724d-819d-4372-b2c8-c170b2d71411\") " pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:24.023376 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:24.023335    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:24.023428 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:24.023414    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs podName:4147724d-819d-4372-b2c8-c170b2d71411 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:26.023391501 +0000 UTC m=+5.987512107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs") pod "network-metrics-daemon-87nv2" (UID: "4147724d-819d-4372-b2c8-c170b2d71411") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:24.023918 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:24.023859    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:24.023918 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:24.023919    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret podName:ce0a4720-2597-4f74-8e8a-012be51716e8 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:26.023902573 +0000 UTC m=+5.988023218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret") pod "global-pull-secret-syncer-6lpt4" (UID: "ce0a4720-2597-4f74-8e8a-012be51716e8") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:24.123780 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.123746    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zw6\" (UniqueName: \"kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6\") pod \"network-check-target-pd8zp\" (UID: \"fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae\") " pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:24.123974 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:24.123957    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:08:24.124053 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:24.123983    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:08:24.124053 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:24.123996    2573 projected.go:196] Error preparing data for projected volume kube-api-access-w8zw6 for pod openshift-network-diagnostics/network-check-target-pd8zp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:24.124161 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:24.124053    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6 podName:fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae nodeName:}" failed. No retries permitted until 2026-07-17 21:08:26.124034209 +0000 UTC m=+6.088154804 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w8zw6" (UniqueName: "kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6") pod "network-check-target-pd8zp" (UID: "fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:24.442729 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.442587    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 21:03:22 +0000 UTC" deadline="2028-01-30 02:50:01.18882235 +0000 UTC"
Jul 17 21:08:24.442729 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.442646    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13469h41m36.746181168s"
Jul 17 21:08:24.528414 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.527477    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:24.528414 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:24.527632    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lpt4" podUID="ce0a4720-2597-4f74-8e8a-012be51716e8"
Jul 17 21:08:24.528414 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.528059    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:24.528414 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:24.528192    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87nv2" podUID="4147724d-819d-4372-b2c8-c170b2d71411"
Jul 17 21:08:24.528414 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.528259    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:24.528414 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:24.528332    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd8zp" podUID="fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae"
Jul 17 21:08:24.539448 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.539417    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k9jkf" event={"ID":"72709796-e935-4cba-a007-cc17fb595e10","Type":"ContainerStarted","Data":"6bfeb635032e59020ef492466423a1cde45a39ed0bc1c71ddbabebfceaf19c2e"}
Jul 17 21:08:24.544193 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.544168    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-brktb" event={"ID":"6205c591-20ac-4b53-aa08-656d2aecc409","Type":"ContainerStarted","Data":"bb05ba7642416e9f63584fac7b84a7938dbd4c14231bd5048614fd8d0d788dab"}
Jul 17 21:08:24.549711 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.549683    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6vr5m" event={"ID":"d9d08404-9306-407f-b32a-a0e2d25473b1","Type":"ContainerStarted","Data":"1850d7ab36f4f62d8752151d496d7877f277fc40c63432d943a9849e38748e77"}
Jul 17 21:08:24.557764 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.557702    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5tpb7" event={"ID":"782caf5b-542f-45e1-b011-263374137dcd","Type":"ContainerStarted","Data":"f880f0075c6be2f2a8c30cfdc130eb9368f8f9c7f7b6b3d1e4bd6eec142fed3f"}
Jul 17 21:08:24.587824 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.587793    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" event={"ID":"a9ca977836d346c8c6c32a4fb71a6efd","Type":"ContainerStarted","Data":"4fd73b997c93094ff5ac2e92289cffcce37561210e32dc787e3e435930bec560"}
Jul 17 21:08:24.597018 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.596989    2573 generic.go:356] "Generic (PLEG): container finished" podID="27af577f6839740a0585a4e6b9dd13dc" containerID="ce0d4e72e459609072a26ed6dd04ecb9deb3704e608d701bba158152e82105b3" exitCode=0
Jul 17 21:08:24.597280 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.597258    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" event={"ID":"27af577f6839740a0585a4e6b9dd13dc","Type":"ContainerDied","Data":"ce0d4e72e459609072a26ed6dd04ecb9deb3704e608d701bba158152e82105b3"}
Jul 17 21:08:24.609383 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.608329    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xgwzb" event={"ID":"35528beb-a7c1-446b-bfae-6eb51d69a48c","Type":"ContainerStarted","Data":"d2dbcdfc263e5843911f2ca8b43c1c586b954cfb89774ae1989a99e13118dbda"}
Jul 17 21:08:24.610054 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.610026    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp" event={"ID":"f2cd4bc8-a91f-4576-9945-534f6ca00bd6","Type":"ContainerStarted","Data":"fcfe106721099278b9d03b1b71f694e84031895b6733cba819ce619fd279df46"}
Jul 17 21:08:24.613897 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.613852    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cxpts" event={"ID":"cafe7b42-f3df-4284-8b7f-aa88520c1f28","Type":"ContainerStarted","Data":"a6c81d79d010307689de1c4c01635b043566ef3854ec361c572453853e8e070e"}
Jul 17 21:08:24.615889 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.615864    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq" event={"ID":"a73e4522-94be-43de-87c3-e0db2654973f","Type":"ContainerStarted","Data":"185da094d7d126361a535df8999de35c2b7e2f0330e68fe6ab9b6d59c9743040"}
Jul 17 21:08:24.619268 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.619216    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-246.ec2.internal" podStartSLOduration=2.6191966239999998 podStartE2EDuration="2.619196624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:24.603743117 +0000 UTC m=+4.567863731" watchObservedRunningTime="2026-07-17 21:08:24.619196624 +0000 UTC m=+4.583317239"
Jul 17 21:08:24.623808 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:24.623783    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99pvf" event={"ID":"2297d3f6-c109-4b3d-883c-1b0a411f9512","Type":"ContainerStarted","Data":"2f0a28abbfb181afb684a1cc67fedd867d57cabccec66a94f25531b192d4f028"}
Jul 17 21:08:25.654853 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:25.654811    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" event={"ID":"27af577f6839740a0585a4e6b9dd13dc","Type":"ContainerStarted","Data":"444da9bac60a0c5ca72b01f6e8c2cbcf084d1aa79209962b55c54cfc69fc930a"}
Jul 17 21:08:26.039685 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:26.039627    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs\") pod \"network-metrics-daemon-87nv2\" (UID: \"4147724d-819d-4372-b2c8-c170b2d71411\") " pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:26.039876 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:26.039790    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret\") pod \"global-pull-secret-syncer-6lpt4\" (UID: \"ce0a4720-2597-4f74-8e8a-012be51716e8\") " pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:26.039945 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:26.039866    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:26.039945 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:26.039913    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:26.040052 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:26.039944    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs podName:4147724d-819d-4372-b2c8-c170b2d71411 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:30.03992241 +0000 UTC m=+10.004043018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs") pod "network-metrics-daemon-87nv2" (UID: "4147724d-819d-4372-b2c8-c170b2d71411") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:26.040052 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:26.039969    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret podName:ce0a4720-2597-4f74-8e8a-012be51716e8 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:30.039952438 +0000 UTC m=+10.004073030 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret") pod "global-pull-secret-syncer-6lpt4" (UID: "ce0a4720-2597-4f74-8e8a-012be51716e8") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:26.140481 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:26.140444    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zw6\" (UniqueName: \"kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6\") pod \"network-check-target-pd8zp\" (UID: \"fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae\") " pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:26.140689 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:26.140670    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:08:26.140767 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:26.140701    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:08:26.140767 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:26.140717    2573 projected.go:196] Error preparing data for projected volume kube-api-access-w8zw6 for pod openshift-network-diagnostics/network-check-target-pd8zp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:26.140878 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:26.140777    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6 podName:fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae nodeName:}" failed. No retries permitted until 2026-07-17 21:08:30.140756506 +0000 UTC m=+10.104877118 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w8zw6" (UniqueName: "kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6") pod "network-check-target-pd8zp" (UID: "fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:26.526207 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:26.526100    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:26.526399 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:26.526224    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lpt4" podUID="ce0a4720-2597-4f74-8e8a-012be51716e8"
Jul 17 21:08:26.526682 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:26.526662    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:26.526804 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:26.526777    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87nv2" podUID="4147724d-819d-4372-b2c8-c170b2d71411"
Jul 17 21:08:26.526872 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:26.526847    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:26.526930 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:26.526918    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd8zp" podUID="fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae"
Jul 17 21:08:28.524768 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:28.524739    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:28.525242 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:28.524877    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87nv2" podUID="4147724d-819d-4372-b2c8-c170b2d71411"
Jul 17 21:08:28.525242 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:28.524932    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:28.525242 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:28.525017    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd8zp" podUID="fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae"
Jul 17 21:08:28.525242 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:28.525056    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:28.525242 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:28.525119    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lpt4" podUID="ce0a4720-2597-4f74-8e8a-012be51716e8"
Jul 17 21:08:30.076641 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:30.074523    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs\") pod \"network-metrics-daemon-87nv2\" (UID: \"4147724d-819d-4372-b2c8-c170b2d71411\") " pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:30.076641 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:30.074608    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret\") pod \"global-pull-secret-syncer-6lpt4\" (UID: \"ce0a4720-2597-4f74-8e8a-012be51716e8\") " pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:30.076641 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:30.074755    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:30.076641 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:30.074757    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:30.076641 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:30.074835    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret podName:ce0a4720-2597-4f74-8e8a-012be51716e8 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:38.074813847 +0000 UTC m=+18.038934452 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret") pod "global-pull-secret-syncer-6lpt4" (UID: "ce0a4720-2597-4f74-8e8a-012be51716e8") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:30.076641 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:30.074924    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs podName:4147724d-819d-4372-b2c8-c170b2d71411 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:38.074904952 +0000 UTC m=+18.039025547 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs") pod "network-metrics-daemon-87nv2" (UID: "4147724d-819d-4372-b2c8-c170b2d71411") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:30.175783 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:30.175747    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zw6\" (UniqueName: \"kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6\") pod \"network-check-target-pd8zp\" (UID: \"fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae\") " pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:30.175943 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:30.175903    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:08:30.175943 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:30.175929    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:08:30.175943 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:30.175942    2573 projected.go:196] Error preparing data for projected volume kube-api-access-w8zw6 for pod openshift-network-diagnostics/network-check-target-pd8zp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:30.176100 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:30.176010    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6 podName:fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae nodeName:}" failed. No retries permitted until 2026-07-17 21:08:38.17598992 +0000 UTC m=+18.140110511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w8zw6" (UniqueName: "kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6") pod "network-check-target-pd8zp" (UID: "fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:30.526378 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:30.526120    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:30.526378 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:30.526232    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:30.526378 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:30.526242    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87nv2" podUID="4147724d-819d-4372-b2c8-c170b2d71411"
Jul 17 21:08:30.526378 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:30.526360    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:30.526771 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:30.526437    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lpt4" podUID="ce0a4720-2597-4f74-8e8a-012be51716e8"
Jul 17 21:08:30.526771 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:30.526486    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd8zp" podUID="fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae"
Jul 17 21:08:32.525407 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:32.524920    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:32.525407 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:32.525056    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lpt4" podUID="ce0a4720-2597-4f74-8e8a-012be51716e8"
Jul 17 21:08:32.525407 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:32.525067    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:32.525407 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:32.525097    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:32.525407 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:32.525183    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87nv2" podUID="4147724d-819d-4372-b2c8-c170b2d71411"
Jul 17 21:08:32.525407 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:32.525261    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd8zp" podUID="fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae"
Jul 17 21:08:34.524589 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:34.524504    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:34.524985 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:34.524637    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lpt4" podUID="ce0a4720-2597-4f74-8e8a-012be51716e8"
Jul 17 21:08:34.524985 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:34.524669    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:34.524985 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:34.524750    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd8zp" podUID="fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae"
Jul 17 21:08:34.524985 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:34.524773    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:34.524985 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:34.524846    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87nv2" podUID="4147724d-819d-4372-b2c8-c170b2d71411"
Jul 17 21:08:36.524819 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:36.524789    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:36.525271 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:36.524789    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:36.525271 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:36.524920    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lpt4" podUID="ce0a4720-2597-4f74-8e8a-012be51716e8"
Jul 17 21:08:36.525271 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:36.524969    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87nv2" podUID="4147724d-819d-4372-b2c8-c170b2d71411"
Jul 17 21:08:36.525271 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:36.524811    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:36.525271 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:36.525043    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd8zp" podUID="fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae"
Jul 17 21:08:38.136073 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:38.136030    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs\") pod \"network-metrics-daemon-87nv2\" (UID: \"4147724d-819d-4372-b2c8-c170b2d71411\") " pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:38.136636 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:38.136102    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret\") pod \"global-pull-secret-syncer-6lpt4\" (UID: \"ce0a4720-2597-4f74-8e8a-012be51716e8\") " pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:38.136636 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:38.136176    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:38.136636 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:38.136204    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:38.136636 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:38.136268    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret podName:ce0a4720-2597-4f74-8e8a-012be51716e8 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:54.136246127 +0000 UTC m=+34.100366765 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret") pod "global-pull-secret-syncer-6lpt4" (UID: "ce0a4720-2597-4f74-8e8a-012be51716e8") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:38.136636 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:38.136305    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs podName:4147724d-819d-4372-b2c8-c170b2d71411 nodeName:}" failed. No retries permitted until 2026-07-17 21:08:54.136277934 +0000 UTC m=+34.100398530 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs") pod "network-metrics-daemon-87nv2" (UID: "4147724d-819d-4372-b2c8-c170b2d71411") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:38.237264 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:38.237239    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zw6\" (UniqueName: \"kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6\") pod \"network-check-target-pd8zp\" (UID: \"fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae\") " pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:38.237405 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:38.237388    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:08:38.237450 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:38.237409    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:08:38.237450 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:38.237418    2573 projected.go:196] Error preparing data for projected volume kube-api-access-w8zw6 for pod openshift-network-diagnostics/network-check-target-pd8zp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:38.237517 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:38.237466    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6 podName:fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae nodeName:}" failed. No retries permitted until 2026-07-17 21:08:54.237449441 +0000 UTC m=+34.201570037 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w8zw6" (UniqueName: "kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6") pod "network-check-target-pd8zp" (UID: "fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:38.524830 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:38.524745    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:38.524830 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:38.524757    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:38.525034 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:38.524860    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:38.525034 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:38.524879    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lpt4" podUID="ce0a4720-2597-4f74-8e8a-012be51716e8"
Jul 17 21:08:38.525034 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:38.524961    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87nv2" podUID="4147724d-819d-4372-b2c8-c170b2d71411"
Jul 17 21:08:38.525124 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:38.525067    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd8zp" podUID="fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae"
Jul 17 21:08:40.528661 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:40.528627    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:40.529089 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:40.528720    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:40.529089 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:40.528747    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:40.529089 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:40.528782    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87nv2" podUID="4147724d-819d-4372-b2c8-c170b2d71411"
Jul 17 21:08:40.529089 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:40.528808    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd8zp" podUID="fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae"
Jul 17 21:08:40.529089 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:40.528860    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lpt4" podUID="ce0a4720-2597-4f74-8e8a-012be51716e8"
Jul 17 21:08:42.526937 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.526735    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:42.528621 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:42.527056    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lpt4" podUID="ce0a4720-2597-4f74-8e8a-012be51716e8"
Jul 17 21:08:42.528621 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.527127    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:42.528621 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:42.527215    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87nv2" podUID="4147724d-819d-4372-b2c8-c170b2d71411"
Jul 17 21:08:42.528621 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.527264    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:42.528621 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:42.527878    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd8zp" podUID="fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae"
Jul 17 21:08:42.690253 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.689932    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6vr5m" event={"ID":"d9d08404-9306-407f-b32a-a0e2d25473b1","Type":"ContainerStarted","Data":"5b2888af6fdc17586d61cebbf01b2d0211ffcfb610e6465a53309b89fdd2870b"}
Jul 17 21:08:42.691489 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.691466    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5tpb7" event={"ID":"782caf5b-542f-45e1-b011-263374137dcd","Type":"ContainerStarted","Data":"b9ba9334f906ac31384fba0792a914fa3bc813081987e016e1d9d14c1e16b6f4"}
Jul 17 21:08:42.692836 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.692813    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xgwzb" event={"ID":"35528beb-a7c1-446b-bfae-6eb51d69a48c","Type":"ContainerStarted","Data":"dd6210e199e4ac62e85f0ebd558b885e2c5d0d00fd5e899957bf730fb41b2e44"}
Jul 17 21:08:42.694892 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.694869    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp" event={"ID":"f2cd4bc8-a91f-4576-9945-534f6ca00bd6","Type":"ContainerStarted","Data":"f2c33cff9435f8a2626ccc937eacb95711b37ceca5a369e4dbef96b0044bad7d"}
Jul 17 21:08:42.694976 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.694935    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp" event={"ID":"f2cd4bc8-a91f-4576-9945-534f6ca00bd6","Type":"ContainerStarted","Data":"470ca1eab1f625ff6966d280d6b61c2138e5f99b4dcd5d20d7568d89ca2d4d8d"}
Jul 17 21:08:42.695063 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.694976    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp" event={"ID":"f2cd4bc8-a91f-4576-9945-534f6ca00bd6","Type":"ContainerStarted","Data":"a4bb9c343f80c5bd98623d3f08cba66a979710348857fa130db0c066e1bf6bc0"}
Jul 17 21:08:42.696324 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.696306    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq" event={"ID":"a73e4522-94be-43de-87c3-e0db2654973f","Type":"ContainerStarted","Data":"ff1f4beceb4c5b32631cd3affe3102a4af19bd1c1cbc81913f955614104c9887"}
Jul 17 21:08:42.697828 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.697803    2573 generic.go:356] "Generic (PLEG): container finished" podID="2297d3f6-c109-4b3d-883c-1b0a411f9512" containerID="8a0db9e1e5e3350d1346c2b300b7c4a17e1b0f9a0409e8688be611b4c8c9293f" exitCode=0
Jul 17 21:08:42.697899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.697878    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99pvf" event={"ID":"2297d3f6-c109-4b3d-883c-1b0a411f9512","Type":"ContainerDied","Data":"8a0db9e1e5e3350d1346c2b300b7c4a17e1b0f9a0409e8688be611b4c8c9293f"}
Jul 17 21:08:42.699579 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.699542    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k9jkf" event={"ID":"72709796-e935-4cba-a007-cc17fb595e10","Type":"ContainerStarted","Data":"91d8b5e34f8aa3113f844096474cac95b5a1ba8627d06353bb22ce10218af5df"}
Jul 17 21:08:42.702917 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.702899    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-brktb" event={"ID":"6205c591-20ac-4b53-aa08-656d2aecc409","Type":"ContainerStarted","Data":"c8b1489cfa6f4bd6f6da5e84e6e1f93b643e7ca87b2a331bed928ea8b0efc9dc"}
Jul 17 21:08:42.705842 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.705805    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6vr5m" podStartSLOduration=6.982818478 podStartE2EDuration="21.705793264s" totalImagesPullingTime="14.722974786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:42.705542341 +0000 UTC m=+22.669662956" watchObservedRunningTime="2026-07-17 21:08:42.705793264 +0000 UTC m=+22.669913873"
Jul 17 21:08:42.705932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.705886    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-246.ec2.internal" podStartSLOduration=20.705878123 podStartE2EDuration="20.705878123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:25.675005309 +0000 UTC m=+5.639125924" watchObservedRunningTime="2026-07-17 21:08:42.705878123 +0000 UTC m=+22.669998735"
Jul 17 21:08:42.737386 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.737339    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xgwzb" podStartSLOduration=2.908898615 podStartE2EDuration="21.737324619s" totalImagesPullingTime="18.828426004s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:42.736981544 +0000 UTC m=+22.701102155" watchObservedRunningTime="2026-07-17 21:08:42.737324619 +0000 UTC m=+22.701445228"
Jul 17 21:08:42.737597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.737572    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-k9jkf" podStartSLOduration=2.859142625 podStartE2EDuration="21.737563207s" totalImagesPullingTime="18.878420582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:42.721871298 +0000 UTC m=+22.685991904" watchObservedRunningTime="2026-07-17 21:08:42.737563207 +0000 UTC m=+22.701683843"
Jul 17 21:08:42.778093 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.778057    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5tpb7" podStartSLOduration=2.942106467 podStartE2EDuration="21.778043195s" totalImagesPullingTime="18.835936728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:42.777717359 +0000 UTC m=+22.741837968" watchObservedRunningTime="2026-07-17 21:08:42.778043195 +0000 UTC m=+22.742163834"
Jul 17 21:08:42.793920 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:42.793888    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-brktb" podStartSLOduration=2.98919158 podStartE2EDuration="21.793874769s" totalImagesPullingTime="18.804683189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:42.79386009 +0000 UTC m=+22.757980701" watchObservedRunningTime="2026-07-17 21:08:42.793874769 +0000 UTC m=+22.757995379"
Jul 17 21:08:43.708295 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:43.708200    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp" event={"ID":"f2cd4bc8-a91f-4576-9945-534f6ca00bd6","Type":"ContainerStarted","Data":"ca70dee81cbd3e51b155fc98f8946ec4d6fc14be937a71a428c599b98dba3484"}
Jul 17 21:08:43.709019 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:43.708348    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp" event={"ID":"f2cd4bc8-a91f-4576-9945-534f6ca00bd6","Type":"ContainerStarted","Data":"677f2c8efdb8f5eae89710ffd49a485176b5761a8a8c8b5c3a9a78e65a871719"}
Jul 17 21:08:43.709019 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:43.708400    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp" event={"ID":"f2cd4bc8-a91f-4576-9945-534f6ca00bd6","Type":"ContainerStarted","Data":"608de69adceb7db6e5a50696974518d588daa6eb1dd295516c155a374e6dd77c"}
Jul 17 21:08:43.709784 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:43.709757    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cxpts" event={"ID":"cafe7b42-f3df-4284-8b7f-aa88520c1f28","Type":"ContainerStarted","Data":"5aeb7fa216bd67a73f1504e45fef807a97be1c144df55534cf33be761c5b52c9"}
Jul 17 21:08:43.911037 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:43.911010    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 21:08:44.524219 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:44.524180    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:44.524646 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:44.524191    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:44.524646 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:44.524325    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lpt4" podUID="ce0a4720-2597-4f74-8e8a-012be51716e8"
Jul 17 21:08:44.524646 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:44.524424    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87nv2" podUID="4147724d-819d-4372-b2c8-c170b2d71411"
Jul 17 21:08:44.524646 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:44.524489    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:44.524646 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:44.524554    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd8zp" podUID="fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae"
Jul 17 21:08:44.607877 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:44.607780    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T21:08:43.911032339Z","UUID":"d4419718-7465-4bae-9edf-e3888319775d","Handler":null,"Name":"","Endpoint":""}
Jul 17 21:08:44.609422 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:44.609385    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 21:08:44.609422 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:44.609428    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 21:08:44.714112 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:44.714084    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq" event={"ID":"a73e4522-94be-43de-87c3-e0db2654973f","Type":"ContainerStarted","Data":"db31553ea9b754d57bc986c32abc6b83066cf34872a8e9708579bf044d45348c"}
Jul 17 21:08:45.457829 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:45.457621    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-brktb"
Jul 17 21:08:45.458317 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:45.458297    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-brktb"
Jul 17 21:08:45.472953 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:45.472918    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cxpts" podStartSLOduration=5.655946943 podStartE2EDuration="24.472905062s" totalImagesPullingTime="18.816958119s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:43.728302239 +0000 UTC m=+23.692422848" watchObservedRunningTime="2026-07-17 21:08:45.472905062 +0000 UTC m=+25.437025671"
Jul 17 21:08:45.722786 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:45.722704    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp" event={"ID":"f2cd4bc8-a91f-4576-9945-534f6ca00bd6","Type":"ContainerStarted","Data":"33ef37280348b2a4779d3bfceeb25220b6a8ec48b34ae0a33abc2ab361bbce10"}
Jul 17 21:08:45.724915 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:45.724884    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq" event={"ID":"a73e4522-94be-43de-87c3-e0db2654973f","Type":"ContainerStarted","Data":"abf09ffdd92dc86e513e78797d3c2f2231e4d92308bddba74ad45599f8517c87"}
Jul 17 21:08:45.725188 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:45.725166    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-brktb"
Jul 17 21:08:45.725843 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:45.725815    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-brktb"
Jul 17 21:08:45.741624 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:45.741572    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92xmq" podStartSLOduration=3.272237793 podStartE2EDuration="24.741557926s" totalImagesPullingTime="21.469320133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:45.741527391 +0000 UTC m=+25.705648002" watchObservedRunningTime="2026-07-17 21:08:45.741557926 +0000 UTC m=+25.705678532"
Jul 17 21:08:46.524344 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:46.524309    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:46.524344 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:46.524340    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:46.524612 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:46.524309    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:46.524612 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:46.524437    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lpt4" podUID="ce0a4720-2597-4f74-8e8a-012be51716e8"
Jul 17 21:08:46.524612 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:46.524511    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87nv2" podUID="4147724d-819d-4372-b2c8-c170b2d71411"
Jul 17 21:08:46.524759 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:46.524607    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd8zp" podUID="fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae"
Jul 17 21:08:47.737869 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:47.737692    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp" event={"ID":"f2cd4bc8-a91f-4576-9945-534f6ca00bd6","Type":"ContainerStarted","Data":"968c2168290b2326161b8bd7c47829a3635f8034ada552f29e3be92b37fc213a"}
Jul 17 21:08:47.738625 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:47.737919    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:47.738625 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:47.738025    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:47.738625 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:47.738092    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:47.739919 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:47.739890    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99pvf" event={"ID":"2297d3f6-c109-4b3d-883c-1b0a411f9512","Type":"ContainerStarted","Data":"66014336b8b7acfc81f7132b467301da317ecfece32160faa1a59ceabb1400be"}
Jul 17 21:08:47.754041 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:47.754023    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:47.754601 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:47.754583    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:08:47.778854 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:47.778817    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp" podStartSLOduration=7.866634901 podStartE2EDuration="26.778800392s" totalImagesPullingTime="18.912165491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:47.777702513 +0000 UTC m=+27.741823122" watchObservedRunningTime="2026-07-17 21:08:47.778800392 +0000 UTC m=+27.742921002"
Jul 17 21:08:48.524646 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:48.524611    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:48.524646 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:48.524639    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:48.524915 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:48.524611    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:48.524915 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:48.524711    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd8zp" podUID="fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae"
Jul 17 21:08:48.524915 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:48.524770    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lpt4" podUID="ce0a4720-2597-4f74-8e8a-012be51716e8"
Jul 17 21:08:48.524915 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:48.524834    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87nv2" podUID="4147724d-819d-4372-b2c8-c170b2d71411"
Jul 17 21:08:48.744331 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:48.744276    2573 generic.go:356] "Generic (PLEG): container finished" podID="2297d3f6-c109-4b3d-883c-1b0a411f9512" containerID="66014336b8b7acfc81f7132b467301da317ecfece32160faa1a59ceabb1400be" exitCode=0
Jul 17 21:08:48.744722 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:48.744364    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99pvf" event={"ID":"2297d3f6-c109-4b3d-883c-1b0a411f9512","Type":"ContainerDied","Data":"66014336b8b7acfc81f7132b467301da317ecfece32160faa1a59ceabb1400be"}
Jul 17 21:08:49.518551 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:49.518462    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6lpt4"]
Jul 17 21:08:49.518740 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:49.518569    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:49.518740 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:49.518677    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lpt4" podUID="ce0a4720-2597-4f74-8e8a-012be51716e8"
Jul 17 21:08:49.519084 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:49.519055    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pd8zp"]
Jul 17 21:08:49.519217 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:49.519136    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:49.519266 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:49.519226    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd8zp" podUID="fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae"
Jul 17 21:08:49.519805 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:49.519777    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-87nv2"]
Jul 17 21:08:49.519890 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:49.519857    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:49.519971 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:49.519946    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87nv2" podUID="4147724d-819d-4372-b2c8-c170b2d71411"
Jul 17 21:08:49.748374 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:49.748341    2573 generic.go:356] "Generic (PLEG): container finished" podID="2297d3f6-c109-4b3d-883c-1b0a411f9512" containerID="7648ca28369a93bce19350715681d57a1d3ff7e9bc6d7253585d95bf894b35b7" exitCode=0
Jul 17 21:08:49.748749 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:49.748420    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99pvf" event={"ID":"2297d3f6-c109-4b3d-883c-1b0a411f9512","Type":"ContainerDied","Data":"7648ca28369a93bce19350715681d57a1d3ff7e9bc6d7253585d95bf894b35b7"}
Jul 17 21:08:51.524545 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:51.524467    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:51.525133 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:51.524467    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:51.525133 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:51.524598    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd8zp" podUID="fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae"
Jul 17 21:08:51.525133 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:51.524467    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:51.525133 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:51.524655    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lpt4" podUID="ce0a4720-2597-4f74-8e8a-012be51716e8"
Jul 17 21:08:51.525133 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:51.524723    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87nv2" podUID="4147724d-819d-4372-b2c8-c170b2d71411"
Jul 17 21:08:51.755409 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:51.755376    2573 generic.go:356] "Generic (PLEG): container finished" podID="2297d3f6-c109-4b3d-883c-1b0a411f9512" containerID="bc9bd4e41cc45a55e3af351ba78e77a985972959f028f8cf568704467b3632c6" exitCode=0
Jul 17 21:08:51.755541 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:51.755423    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99pvf" event={"ID":"2297d3f6-c109-4b3d-883c-1b0a411f9512","Type":"ContainerDied","Data":"bc9bd4e41cc45a55e3af351ba78e77a985972959f028f8cf568704467b3632c6"}
Jul 17 21:08:53.525062 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:53.525030    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:53.525572 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:53.525117    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:53.525572 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:53.525123    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd8zp" podUID="fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae"
Jul 17 21:08:53.525572 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:53.525207    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lpt4" podUID="ce0a4720-2597-4f74-8e8a-012be51716e8"
Jul 17 21:08:53.525572 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:53.525125    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:53.525572 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:53.525322    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-87nv2" podUID="4147724d-819d-4372-b2c8-c170b2d71411"
Jul 17 21:08:53.763335 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:53.763268    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xgwzb_35528beb-a7c1-446b-bfae-6eb51d69a48c/dns-node-resolver/0.log"
Jul 17 21:08:54.161338 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:54.161309    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs\") pod \"network-metrics-daemon-87nv2\" (UID: \"4147724d-819d-4372-b2c8-c170b2d71411\") " pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:54.161499 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:54.161356    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret\") pod \"global-pull-secret-syncer-6lpt4\" (UID: \"ce0a4720-2597-4f74-8e8a-012be51716e8\") " pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:54.161499 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:54.161463    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:54.161499 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:54.161466    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:54.161648 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:54.161521    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret podName:ce0a4720-2597-4f74-8e8a-012be51716e8 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:26.161506843 +0000 UTC m=+66.125627432 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret") pod "global-pull-secret-syncer-6lpt4" (UID: "ce0a4720-2597-4f74-8e8a-012be51716e8") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:08:54.161648 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:54.161537    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs podName:4147724d-819d-4372-b2c8-c170b2d71411 nodeName:}" failed. No retries permitted until 2026-07-17 21:09:26.161530253 +0000 UTC m=+66.125650840 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs") pod "network-metrics-daemon-87nv2" (UID: "4147724d-819d-4372-b2c8-c170b2d71411") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:08:54.261846 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:54.261811    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zw6\" (UniqueName: \"kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6\") pod \"network-check-target-pd8zp\" (UID: \"fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae\") " pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:54.261998 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:54.261946    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:08:54.261998 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:54.261967    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:08:54.261998 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:54.261979    2573 projected.go:196] Error preparing data for projected volume kube-api-access-w8zw6 for pod openshift-network-diagnostics/network-check-target-pd8zp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:54.262157 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:08:54.262033    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6 podName:fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae nodeName:}" failed. No retries permitted until 2026-07-17 21:09:26.26201548 +0000 UTC m=+66.226136070 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-w8zw6" (UniqueName: "kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6") pod "network-check-target-pd8zp" (UID: "fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:08:54.331246 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:54.331219    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6vr5m_d9d08404-9306-407f-b32a-a0e2d25473b1/node-ca/0.log"
Jul 17 21:08:55.394881 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:55.394851    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeReady"
Jul 17 21:08:55.395378 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:55.394994    2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 21:08:55.524346 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:55.524313    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:08:55.524538 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:55.524320    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:08:55.524606 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:55.524326    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:08:55.528261 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:55.528228    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-szcqz\""
Jul 17 21:08:55.528261 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:55.528228    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8vkwx\""
Jul 17 21:08:55.528599 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:55.528548    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 21:08:55.528921 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:55.528777    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 21:08:55.528921 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:55.528788    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 21:08:55.528921 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:55.528850    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 21:08:56.771143 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:56.770975    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99pvf" event={"ID":"2297d3f6-c109-4b3d-883c-1b0a411f9512","Type":"ContainerStarted","Data":"94f939ab9bf861f4c231f375acf8a5d6e07ca203447347fc4fd048d3dd0e5692"}
Jul 17 21:08:57.775480 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:57.775443    2573 generic.go:356] "Generic (PLEG): container finished" podID="2297d3f6-c109-4b3d-883c-1b0a411f9512" containerID="94f939ab9bf861f4c231f375acf8a5d6e07ca203447347fc4fd048d3dd0e5692" exitCode=0
Jul 17 21:08:57.775937 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:57.775530    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99pvf" event={"ID":"2297d3f6-c109-4b3d-883c-1b0a411f9512","Type":"ContainerDied","Data":"94f939ab9bf861f4c231f375acf8a5d6e07ca203447347fc4fd048d3dd0e5692"}
Jul 17 21:08:58.780128 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:58.780096    2573 generic.go:356] "Generic (PLEG): container finished" podID="2297d3f6-c109-4b3d-883c-1b0a411f9512" containerID="4fed4213fff10b36f5b3a82cff12e11469377133db2ae16be425d2dcc47933ca" exitCode=0
Jul 17 21:08:58.780515 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:58.780134    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99pvf" event={"ID":"2297d3f6-c109-4b3d-883c-1b0a411f9512","Type":"ContainerDied","Data":"4fed4213fff10b36f5b3a82cff12e11469377133db2ae16be425d2dcc47933ca"}
Jul 17 21:08:59.786811 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:59.786775    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-99pvf" event={"ID":"2297d3f6-c109-4b3d-883c-1b0a411f9512","Type":"ContainerStarted","Data":"906751e45b3509d79b42c7da7acbf0c9b456f221c9455aeb48c258263b3bb4a9"}
Jul 17 21:08:59.813110 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:08:59.813071    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-99pvf" podStartSLOduration=11.983919876 podStartE2EDuration="38.813058301s" totalImagesPullingTime="26.829138425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:08:59.811631122 +0000 UTC m=+39.775751733" watchObservedRunningTime="2026-07-17 21:08:59.813058301 +0000 UTC m=+39.777178910"
Jul 17 21:09:17.083315 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.083260    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-749775c58-8ldvv"]
Jul 17 21:09:17.127691 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.127657    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd"]
Jul 17 21:09:17.127848 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.127773    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.132086 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.132060    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 17 21:09:17.132318 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.132297    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 17 21:09:17.132446 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.132399    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 17 21:09:17.133232 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.133213    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-b28fl\""
Jul 17 21:09:17.136686 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.136670    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 17 21:09:17.141661 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.141644    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"]
Jul 17 21:09:17.141740 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.141729    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd"
Jul 17 21:09:17.144723 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.144703    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 17 21:09:17.144809 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.144727    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 17 21:09:17.145023 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.145007    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 17 21:09:17.145191 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.145176    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 17 21:09:17.163671 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.163654    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j"]
Jul 17 21:09:17.163941 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.163919    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.168770 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.168751    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 17 21:09:17.168848 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.168787    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 17 21:09:17.175607 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.175576    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 17 21:09:17.175706 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.175691    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 17 21:09:17.178096 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.178080    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j"]
Jul 17 21:09:17.178181 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.178100    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dnp9k"]
Jul 17 21:09:17.178242 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.178210    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j"
Jul 17 21:09:17.183796 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.183781    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 17 21:09:17.183884 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.183855    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-kzfg4\""
Jul 17 21:09:17.192174 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.192154    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"]
Jul 17 21:09:17.192271 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.192206    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-749775c58-8ldvv"]
Jul 17 21:09:17.192271 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.192222    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd"]
Jul 17 21:09:17.192271 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.192236    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dnp9k"]
Jul 17 21:09:17.192454 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.192272    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dnp9k"
Jul 17 21:09:17.195212 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.195199    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 21:09:17.195637 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.195608    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sqgsr\""
Jul 17 21:09:17.195733 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.195668    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 21:09:17.195733 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.195672    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 21:09:17.195733 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.195695    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 21:09:17.288874 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.288856    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m6kll"]
Jul 17 21:09:17.308609 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.308589    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78f540df-428a-432e-ac03-09f58a3caa8c-registry-tls\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.308685 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.308619    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96edb730-a088-46cb-89e6-731a4d05f8cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnp9k\" (UID: \"96edb730-a088-46cb-89e6-731a4d05f8cc\") " pod="openshift-insights/insights-runtime-extractor-dnp9k"
Jul 17 21:09:17.308685 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.308649    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pwqh\" (UniqueName: \"kubernetes.io/projected/3cb998da-5aa4-41f7-9721-df07f47b793f-kube-api-access-8pwqh\") pod \"cluster-proxy-proxy-agent-596d656b9-bhfxp\" (UID: \"3cb998da-5aa4-41f7-9721-df07f47b793f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.308791 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.308706    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78f540df-428a-432e-ac03-09f58a3caa8c-installation-pull-secrets\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.308791 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.308728    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6x9x\" (UniqueName: \"kubernetes.io/projected/f8d1ffa9-81dd-461a-84b8-d0a69be2b619-kube-api-access-p6x9x\") pod \"klusterlet-addon-workmgr-5bd8c849bc-ml6kd\" (UID: \"f8d1ffa9-81dd-461a-84b8-d0a69be2b619\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd"
Jul 17 21:09:17.308791 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.308776    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96edb730-a088-46cb-89e6-731a4d05f8cc-data-volume\") pod \"insights-runtime-extractor-dnp9k\" (UID: \"96edb730-a088-46cb-89e6-731a4d05f8cc\") " pod="openshift-insights/insights-runtime-extractor-dnp9k"
Jul 17 21:09:17.308914 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.308816    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96edb730-a088-46cb-89e6-731a4d05f8cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnp9k\" (UID: \"96edb730-a088-46cb-89e6-731a4d05f8cc\") " pod="openshift-insights/insights-runtime-extractor-dnp9k"
Jul 17 21:09:17.308914 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.308843    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxpc6\" (UniqueName: \"kubernetes.io/projected/01a1222a-6dce-493f-bade-6f46be398b8e-kube-api-access-gxpc6\") pod \"managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j\" (UID: \"01a1222a-6dce-493f-bade-6f46be398b8e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j"
Jul 17 21:09:17.308914 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.308873    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01a1222a-6dce-493f-bade-6f46be398b8e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j\" (UID: \"01a1222a-6dce-493f-bade-6f46be398b8e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j"
Jul 17 21:09:17.308914 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.308909    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78f540df-428a-432e-ac03-09f58a3caa8c-ca-trust-extracted\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.309040 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.308928    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78f540df-428a-432e-ac03-09f58a3caa8c-trusted-ca\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.309040 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.308951    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78f540df-428a-432e-ac03-09f58a3caa8c-bound-sa-token\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.309040 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.308983    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3cb998da-5aa4-41f7-9721-df07f47b793f-ca\") pod \"cluster-proxy-proxy-agent-596d656b9-bhfxp\" (UID: \"3cb998da-5aa4-41f7-9721-df07f47b793f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.309040 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.309016    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78f540df-428a-432e-ac03-09f58a3caa8c-image-registry-private-configuration\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.309040 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.309030    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3cb998da-5aa4-41f7-9721-df07f47b793f-hub\") pod \"cluster-proxy-proxy-agent-596d656b9-bhfxp\" (UID: \"3cb998da-5aa4-41f7-9721-df07f47b793f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.309230 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.309046    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96edb730-a088-46cb-89e6-731a4d05f8cc-crio-socket\") pod \"insights-runtime-extractor-dnp9k\" (UID: \"96edb730-a088-46cb-89e6-731a4d05f8cc\") " pod="openshift-insights/insights-runtime-extractor-dnp9k"
Jul 17 21:09:17.309230 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.309080    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78f540df-428a-432e-ac03-09f58a3caa8c-registry-certificates\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.309230 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.309113    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwzw4\" (UniqueName: \"kubernetes.io/projected/96edb730-a088-46cb-89e6-731a4d05f8cc-kube-api-access-zwzw4\") pod \"insights-runtime-extractor-dnp9k\" (UID: \"96edb730-a088-46cb-89e6-731a4d05f8cc\") " pod="openshift-insights/insights-runtime-extractor-dnp9k"
Jul 17 21:09:17.309230 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.309163    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8d1ffa9-81dd-461a-84b8-d0a69be2b619-tmp\") pod \"klusterlet-addon-workmgr-5bd8c849bc-ml6kd\" (UID: \"f8d1ffa9-81dd-461a-84b8-d0a69be2b619\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd"
Jul 17 21:09:17.309230 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.309195    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f8d1ffa9-81dd-461a-84b8-d0a69be2b619-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bd8c849bc-ml6kd\" (UID: \"f8d1ffa9-81dd-461a-84b8-d0a69be2b619\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd"
Jul 17 21:09:17.309230 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.309217    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3cb998da-5aa4-41f7-9721-df07f47b793f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-596d656b9-bhfxp\" (UID: \"3cb998da-5aa4-41f7-9721-df07f47b793f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.309522 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.309242    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb998da-5aa4-41f7-9721-df07f47b793f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-596d656b9-bhfxp\" (UID: \"3cb998da-5aa4-41f7-9721-df07f47b793f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.309522 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.309267    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3cb998da-5aa4-41f7-9721-df07f47b793f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-596d656b9-bhfxp\" (UID: \"3cb998da-5aa4-41f7-9721-df07f47b793f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.309522 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.309340    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wmp8\" (UniqueName: \"kubernetes.io/projected/78f540df-428a-432e-ac03-09f58a3caa8c-kube-api-access-7wmp8\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.310405 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.310392    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m6kll"
Jul 17 21:09:17.313714 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.313498    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 21:09:17.313714 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.313525    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 21:09:17.313714 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.313605    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j8vps\""
Jul 17 21:09:17.313714 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.313687    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 21:09:17.314132 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.314116    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m6kll"]
Jul 17 21:09:17.378670 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.378647    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2m9sr"]
Jul 17 21:09:17.401353 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.401333    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2m9sr"]
Jul 17 21:09:17.401437 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.401409    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2m9sr"
Jul 17 21:09:17.404075 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.404053    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 21:09:17.404173 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.404087    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rdmx5\""
Jul 17 21:09:17.404317 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.404304    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 21:09:17.409910 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.409890    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8d1ffa9-81dd-461a-84b8-d0a69be2b619-tmp\") pod \"klusterlet-addon-workmgr-5bd8c849bc-ml6kd\" (UID: \"f8d1ffa9-81dd-461a-84b8-d0a69be2b619\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd"
Jul 17 21:09:17.409996 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.409927    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f8d1ffa9-81dd-461a-84b8-d0a69be2b619-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bd8c849bc-ml6kd\" (UID: \"f8d1ffa9-81dd-461a-84b8-d0a69be2b619\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd"
Jul 17 21:09:17.409996 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.409953    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3cb998da-5aa4-41f7-9721-df07f47b793f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-596d656b9-bhfxp\" (UID: \"3cb998da-5aa4-41f7-9721-df07f47b793f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.410072 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.409998    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb998da-5aa4-41f7-9721-df07f47b793f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-596d656b9-bhfxp\" (UID: \"3cb998da-5aa4-41f7-9721-df07f47b793f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.410072 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410023    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3cb998da-5aa4-41f7-9721-df07f47b793f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-596d656b9-bhfxp\" (UID: \"3cb998da-5aa4-41f7-9721-df07f47b793f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.410072 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410059    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wmp8\" (UniqueName: \"kubernetes.io/projected/78f540df-428a-432e-ac03-09f58a3caa8c-kube-api-access-7wmp8\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.410198 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410086    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78f540df-428a-432e-ac03-09f58a3caa8c-registry-tls\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.410198 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410128    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96edb730-a088-46cb-89e6-731a4d05f8cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnp9k\" (UID: \"96edb730-a088-46cb-89e6-731a4d05f8cc\") " pod="openshift-insights/insights-runtime-extractor-dnp9k"
Jul 17 21:09:17.410198 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410191    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pwqh\" (UniqueName: \"kubernetes.io/projected/3cb998da-5aa4-41f7-9721-df07f47b793f-kube-api-access-8pwqh\") pod \"cluster-proxy-proxy-agent-596d656b9-bhfxp\" (UID: \"3cb998da-5aa4-41f7-9721-df07f47b793f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.410365 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410226    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78f540df-428a-432e-ac03-09f58a3caa8c-installation-pull-secrets\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.410365 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410250    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6x9x\" (UniqueName: \"kubernetes.io/projected/f8d1ffa9-81dd-461a-84b8-d0a69be2b619-kube-api-access-p6x9x\") pod \"klusterlet-addon-workmgr-5bd8c849bc-ml6kd\" (UID: \"f8d1ffa9-81dd-461a-84b8-d0a69be2b619\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd"
Jul 17 21:09:17.410365 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410272    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96edb730-a088-46cb-89e6-731a4d05f8cc-data-volume\") pod \"insights-runtime-extractor-dnp9k\" (UID: \"96edb730-a088-46cb-89e6-731a4d05f8cc\") " pod="openshift-insights/insights-runtime-extractor-dnp9k"
Jul 17 21:09:17.410365 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410319    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96edb730-a088-46cb-89e6-731a4d05f8cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnp9k\" (UID: \"96edb730-a088-46cb-89e6-731a4d05f8cc\") " pod="openshift-insights/insights-runtime-extractor-dnp9k"
Jul 17 21:09:17.410365 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410343    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxpc6\" (UniqueName: \"kubernetes.io/projected/01a1222a-6dce-493f-bade-6f46be398b8e-kube-api-access-gxpc6\") pod \"managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j\" (UID: \"01a1222a-6dce-493f-bade-6f46be398b8e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j"
Jul 17 21:09:17.410365 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410365    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01a1222a-6dce-493f-bade-6f46be398b8e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j\" (UID: \"01a1222a-6dce-493f-bade-6f46be398b8e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j"
Jul 17 21:09:17.410653 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410389    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78f540df-428a-432e-ac03-09f58a3caa8c-ca-trust-extracted\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.410653 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410409    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78f540df-428a-432e-ac03-09f58a3caa8c-trusted-ca\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.410653 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410430    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78f540df-428a-432e-ac03-09f58a3caa8c-bound-sa-token\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.410653 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410453    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cf5abf4-6b79-4ed0-bf87-a45cddb80aa7-cert\") pod \"ingress-canary-m6kll\" (UID: \"8cf5abf4-6b79-4ed0-bf87-a45cddb80aa7\") " pod="openshift-ingress-canary/ingress-canary-m6kll"
Jul 17 21:09:17.410653 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410486    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3cb998da-5aa4-41f7-9721-df07f47b793f-ca\") pod \"cluster-proxy-proxy-agent-596d656b9-bhfxp\" (UID: \"3cb998da-5aa4-41f7-9721-df07f47b793f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.410653 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410521    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78f540df-428a-432e-ac03-09f58a3caa8c-image-registry-private-configuration\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.410653 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410545    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3cb998da-5aa4-41f7-9721-df07f47b793f-hub\") pod \"cluster-proxy-proxy-agent-596d656b9-bhfxp\" (UID: \"3cb998da-5aa4-41f7-9721-df07f47b793f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.410653 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410566    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96edb730-a088-46cb-89e6-731a4d05f8cc-crio-socket\") pod \"insights-runtime-extractor-dnp9k\" (UID: \"96edb730-a088-46cb-89e6-731a4d05f8cc\") " pod="openshift-insights/insights-runtime-extractor-dnp9k"
Jul 17 21:09:17.410653 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410589    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq9fg\" (UniqueName: \"kubernetes.io/projected/8cf5abf4-6b79-4ed0-bf87-a45cddb80aa7-kube-api-access-tq9fg\") pod \"ingress-canary-m6kll\" (UID: \"8cf5abf4-6b79-4ed0-bf87-a45cddb80aa7\") " pod="openshift-ingress-canary/ingress-canary-m6kll"
Jul 17 21:09:17.410653 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410632    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78f540df-428a-432e-ac03-09f58a3caa8c-registry-certificates\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.410653 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410656    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwzw4\" (UniqueName: \"kubernetes.io/projected/96edb730-a088-46cb-89e6-731a4d05f8cc-kube-api-access-zwzw4\") pod \"insights-runtime-extractor-dnp9k\" (UID: \"96edb730-a088-46cb-89e6-731a4d05f8cc\") " pod="openshift-insights/insights-runtime-extractor-dnp9k"
Jul 17 21:09:17.411185 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.410680    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96edb730-a088-46cb-89e6-731a4d05f8cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnp9k\" (UID: \"96edb730-a088-46cb-89e6-731a4d05f8cc\") " pod="openshift-insights/insights-runtime-extractor-dnp9k"
Jul 17 21:09:17.411185 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.411062    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96edb730-a088-46cb-89e6-731a4d05f8cc-data-volume\") pod \"insights-runtime-extractor-dnp9k\" (UID: \"96edb730-a088-46cb-89e6-731a4d05f8cc\") " pod="openshift-insights/insights-runtime-extractor-dnp9k"
Jul 17 21:09:17.411185 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.411126    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb998da-5aa4-41f7-9721-df07f47b793f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-596d656b9-bhfxp\" (UID: \"3cb998da-5aa4-41f7-9721-df07f47b793f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.411355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.411244    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96edb730-a088-46cb-89e6-731a4d05f8cc-crio-socket\") pod \"insights-runtime-extractor-dnp9k\" (UID: \"96edb730-a088-46cb-89e6-731a4d05f8cc\") " pod="openshift-insights/insights-runtime-extractor-dnp9k"
Jul 17 21:09:17.413462 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.413048    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78f540df-428a-432e-ac03-09f58a3caa8c-registry-certificates\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.413462 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.413390    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78f540df-428a-432e-ac03-09f58a3caa8c-ca-trust-extracted\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.414116 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.414096    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78f540df-428a-432e-ac03-09f58a3caa8c-trusted-ca\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.415493 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.415462    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78f540df-428a-432e-ac03-09f58a3caa8c-installation-pull-secrets\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.415579 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.415555    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96edb730-a088-46cb-89e6-731a4d05f8cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnp9k\" (UID: \"96edb730-a088-46cb-89e6-731a4d05f8cc\") " pod="openshift-insights/insights-runtime-extractor-dnp9k"
Jul 17 21:09:17.415687 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.415664    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3cb998da-5aa4-41f7-9721-df07f47b793f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-596d656b9-bhfxp\" (UID: \"3cb998da-5aa4-41f7-9721-df07f47b793f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.415780 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.415760    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78f540df-428a-432e-ac03-09f58a3caa8c-image-registry-private-configuration\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.416075 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.416034    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3cb998da-5aa4-41f7-9721-df07f47b793f-hub\") pod \"cluster-proxy-proxy-agent-596d656b9-bhfxp\" (UID: \"3cb998da-5aa4-41f7-9721-df07f47b793f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.416308 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.416266    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3cb998da-5aa4-41f7-9721-df07f47b793f-ca\") pod \"cluster-proxy-proxy-agent-596d656b9-bhfxp\" (UID: \"3cb998da-5aa4-41f7-9721-df07f47b793f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.416797 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.416774    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3cb998da-5aa4-41f7-9721-df07f47b793f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-596d656b9-bhfxp\" (UID: \"3cb998da-5aa4-41f7-9721-df07f47b793f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.416914 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.416793    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78f540df-428a-432e-ac03-09f58a3caa8c-registry-tls\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.417273 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.417249    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01a1222a-6dce-493f-bade-6f46be398b8e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j\" (UID: \"01a1222a-6dce-493f-bade-6f46be398b8e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j"
Jul 17 21:09:17.421368 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.421349    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8d1ffa9-81dd-461a-84b8-d0a69be2b619-tmp\") pod \"klusterlet-addon-workmgr-5bd8c849bc-ml6kd\" (UID: \"f8d1ffa9-81dd-461a-84b8-d0a69be2b619\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd"
Jul 17 21:09:17.421701 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.421683    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f8d1ffa9-81dd-461a-84b8-d0a69be2b619-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bd8c849bc-ml6kd\" (UID: \"f8d1ffa9-81dd-461a-84b8-d0a69be2b619\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd"
Jul 17 21:09:17.442588 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.442561    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78f540df-428a-432e-ac03-09f58a3caa8c-bound-sa-token\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.443150 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.443134    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwzw4\" (UniqueName: \"kubernetes.io/projected/96edb730-a088-46cb-89e6-731a4d05f8cc-kube-api-access-zwzw4\") pod \"insights-runtime-extractor-dnp9k\" (UID: \"96edb730-a088-46cb-89e6-731a4d05f8cc\") " pod="openshift-insights/insights-runtime-extractor-dnp9k"
Jul 17 21:09:17.443463 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.443444    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wmp8\" (UniqueName: \"kubernetes.io/projected/78f540df-428a-432e-ac03-09f58a3caa8c-kube-api-access-7wmp8\") pod \"image-registry-749775c58-8ldvv\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") " pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.446936 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.446910    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxpc6\" (UniqueName: \"kubernetes.io/projected/01a1222a-6dce-493f-bade-6f46be398b8e-kube-api-access-gxpc6\") pod \"managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j\" (UID: \"01a1222a-6dce-493f-bade-6f46be398b8e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j"
Jul 17 21:09:17.447840 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.447823    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pwqh\" (UniqueName: \"kubernetes.io/projected/3cb998da-5aa4-41f7-9721-df07f47b793f-kube-api-access-8pwqh\") pod \"cluster-proxy-proxy-agent-596d656b9-bhfxp\" (UID: \"3cb998da-5aa4-41f7-9721-df07f47b793f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.451257 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.451238    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6x9x\" (UniqueName: \"kubernetes.io/projected/f8d1ffa9-81dd-461a-84b8-d0a69be2b619-kube-api-access-p6x9x\") pod \"klusterlet-addon-workmgr-5bd8c849bc-ml6kd\" (UID: \"f8d1ffa9-81dd-461a-84b8-d0a69be2b619\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd"
Jul 17 21:09:17.481556 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.481537    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"
Jul 17 21:09:17.490643 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.490626    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j"
Jul 17 21:09:17.511436 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.511416    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq9fg\" (UniqueName: \"kubernetes.io/projected/8cf5abf4-6b79-4ed0-bf87-a45cddb80aa7-kube-api-access-tq9fg\") pod \"ingress-canary-m6kll\" (UID: \"8cf5abf4-6b79-4ed0-bf87-a45cddb80aa7\") " pod="openshift-ingress-canary/ingress-canary-m6kll"
Jul 17 21:09:17.511529 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.511448    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/febc06a9-decb-4fb6-9b31-443a339262b9-metrics-tls\") pod \"dns-default-2m9sr\" (UID: \"febc06a9-decb-4fb6-9b31-443a339262b9\") " pod="openshift-dns/dns-default-2m9sr"
Jul 17 21:09:17.511529 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.511470    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febc06a9-decb-4fb6-9b31-443a339262b9-tmp-dir\") pod \"dns-default-2m9sr\" (UID: \"febc06a9-decb-4fb6-9b31-443a339262b9\") " pod="openshift-dns/dns-default-2m9sr"
Jul 17 21:09:17.511615 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.511599    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cf5abf4-6b79-4ed0-bf87-a45cddb80aa7-cert\") pod \"ingress-canary-m6kll\" (UID: \"8cf5abf4-6b79-4ed0-bf87-a45cddb80aa7\") " pod="openshift-ingress-canary/ingress-canary-m6kll"
Jul 17 21:09:17.511661 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.511647    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc7p7\" (UniqueName: \"kubernetes.io/projected/febc06a9-decb-4fb6-9b31-443a339262b9-kube-api-access-zc7p7\") pod \"dns-default-2m9sr\" (UID: \"febc06a9-decb-4fb6-9b31-443a339262b9\") " pod="openshift-dns/dns-default-2m9sr"
Jul 17 21:09:17.511707 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.511678    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/febc06a9-decb-4fb6-9b31-443a339262b9-config-volume\") pod \"dns-default-2m9sr\" (UID: \"febc06a9-decb-4fb6-9b31-443a339262b9\") " pod="openshift-dns/dns-default-2m9sr"
Jul 17 21:09:17.513589 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.513567    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cf5abf4-6b79-4ed0-bf87-a45cddb80aa7-cert\") pod \"ingress-canary-m6kll\" (UID: \"8cf5abf4-6b79-4ed0-bf87-a45cddb80aa7\") " pod="openshift-ingress-canary/ingress-canary-m6kll"
Jul 17 21:09:17.518052 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.518033    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dnp9k"
Jul 17 21:09:17.520277 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.520256    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq9fg\" (UniqueName: \"kubernetes.io/projected/8cf5abf4-6b79-4ed0-bf87-a45cddb80aa7-kube-api-access-tq9fg\") pod \"ingress-canary-m6kll\" (UID: \"8cf5abf4-6b79-4ed0-bf87-a45cddb80aa7\") " pod="openshift-ingress-canary/ingress-canary-m6kll"
Jul 17 21:09:17.612333 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.612302    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/febc06a9-decb-4fb6-9b31-443a339262b9-metrics-tls\") pod \"dns-default-2m9sr\" (UID: \"febc06a9-decb-4fb6-9b31-443a339262b9\") " pod="openshift-dns/dns-default-2m9sr"
Jul 17 21:09:17.612449 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.612350    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febc06a9-decb-4fb6-9b31-443a339262b9-tmp-dir\") pod \"dns-default-2m9sr\" (UID: \"febc06a9-decb-4fb6-9b31-443a339262b9\") " pod="openshift-dns/dns-default-2m9sr"
Jul 17 21:09:17.612449 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.612443    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc7p7\" (UniqueName: \"kubernetes.io/projected/febc06a9-decb-4fb6-9b31-443a339262b9-kube-api-access-zc7p7\") pod \"dns-default-2m9sr\" (UID: \"febc06a9-decb-4fb6-9b31-443a339262b9\") " pod="openshift-dns/dns-default-2m9sr"
Jul 17 21:09:17.612514 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.612488    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/febc06a9-decb-4fb6-9b31-443a339262b9-config-volume\") pod \"dns-default-2m9sr\" (UID: \"febc06a9-decb-4fb6-9b31-443a339262b9\") " pod="openshift-dns/dns-default-2m9sr"
Jul 17 21:09:17.612978 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.612936    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febc06a9-decb-4fb6-9b31-443a339262b9-tmp-dir\") pod \"dns-default-2m9sr\" (UID: \"febc06a9-decb-4fb6-9b31-443a339262b9\") " pod="openshift-dns/dns-default-2m9sr"
Jul 17 21:09:17.613206 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.613184    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/febc06a9-decb-4fb6-9b31-443a339262b9-config-volume\") pod \"dns-default-2m9sr\" (UID: \"febc06a9-decb-4fb6-9b31-443a339262b9\") " pod="openshift-dns/dns-default-2m9sr"
Jul 17 21:09:17.615070 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.615047    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/febc06a9-decb-4fb6-9b31-443a339262b9-metrics-tls\") pod \"dns-default-2m9sr\" (UID: \"febc06a9-decb-4fb6-9b31-443a339262b9\") " pod="openshift-dns/dns-default-2m9sr"
Jul 17 21:09:17.618522 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.618503    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m6kll"
Jul 17 21:09:17.625794 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.625771    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc7p7\" (UniqueName: \"kubernetes.io/projected/febc06a9-decb-4fb6-9b31-443a339262b9-kube-api-access-zc7p7\") pod \"dns-default-2m9sr\" (UID: \"febc06a9-decb-4fb6-9b31-443a339262b9\") " pod="openshift-dns/dns-default-2m9sr"
Jul 17 21:09:17.673889 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.673864    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j"]
Jul 17 21:09:17.676769 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:09:17.676740    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01a1222a_6dce_493f_bade_6f46be398b8e.slice/crio-75b25b5cb43f26c5654205d5739cd919f030e552191bbdac19a661f3c0143556 WatchSource:0}: Error finding container 75b25b5cb43f26c5654205d5739cd919f030e552191bbdac19a661f3c0143556: Status 404 returned error can't find the container with id 75b25b5cb43f26c5654205d5739cd919f030e552191bbdac19a661f3c0143556
Jul 17 21:09:17.721768 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.721743    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2m9sr"
Jul 17 21:09:17.737983 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.737962    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:17.750026 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.750003    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd"
Jul 17 21:09:17.753730 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.753708    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m6kll"]
Jul 17 21:09:17.757369 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:09:17.757342    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf5abf4_6b79_4ed0_bf87_a45cddb80aa7.slice/crio-5bf6bcad2f74a1349c60f4409dce06da16417d3b27b26d698d0ec471559d1231 WatchSource:0}: Error finding container 5bf6bcad2f74a1349c60f4409dce06da16417d3b27b26d698d0ec471559d1231: Status 404 returned error can't find the container with id 5bf6bcad2f74a1349c60f4409dce06da16417d3b27b26d698d0ec471559d1231
Jul 17 21:09:17.829873 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.829799    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m6kll" event={"ID":"8cf5abf4-6b79-4ed0-bf87-a45cddb80aa7","Type":"ContainerStarted","Data":"5bf6bcad2f74a1349c60f4409dce06da16417d3b27b26d698d0ec471559d1231"}
Jul 17 21:09:17.831331 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.831300    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j" event={"ID":"01a1222a-6dce-493f-bade-6f46be398b8e","Type":"ContainerStarted","Data":"75b25b5cb43f26c5654205d5739cd919f030e552191bbdac19a661f3c0143556"}
Jul 17 21:09:17.880238 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:09:17.880204    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebc06a9_decb_4fb6_9b31_443a339262b9.slice/crio-26b8e3039336b35c9c91e1b2d0562893b9e77e06855230d889e2fe6ea951c4c3 WatchSource:0}: Error finding container 26b8e3039336b35c9c91e1b2d0562893b9e77e06855230d889e2fe6ea951c4c3: Status 404 returned error can't find the container with id 26b8e3039336b35c9c91e1b2d0562893b9e77e06855230d889e2fe6ea951c4c3
Jul 17 21:09:17.881426 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.881382    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2m9sr"]
Jul 17 21:09:17.895625 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.895466    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dnp9k"]
Jul 17 21:09:17.896576 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.896552    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp"]
Jul 17 21:09:17.897344 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.897323    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-749775c58-8ldvv"]
Jul 17 21:09:17.897833 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:09:17.897813    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f540df_428a_432e_ac03_09f58a3caa8c.slice/crio-86ceeaf8225a3e00bdf8bc1d1ec1a92ab6a0faf40d5d9f43c3988441869f7002 WatchSource:0}: Error finding container 86ceeaf8225a3e00bdf8bc1d1ec1a92ab6a0faf40d5d9f43c3988441869f7002: Status 404 returned error can't find the container with id 86ceeaf8225a3e00bdf8bc1d1ec1a92ab6a0faf40d5d9f43c3988441869f7002
Jul 17 21:09:17.898841 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:09:17.898809    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96edb730_a088_46cb_89e6_731a4d05f8cc.slice/crio-50da6dee0ded60057325a6c781da7d8ccddca81e623b11656a09076581ba7279 WatchSource:0}: Error finding container 50da6dee0ded60057325a6c781da7d8ccddca81e623b11656a09076581ba7279: Status 404 returned error can't find the container with id 50da6dee0ded60057325a6c781da7d8ccddca81e623b11656a09076581ba7279
Jul 17 21:09:17.899814 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:09:17.899778    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cb998da_5aa4_41f7_9721_df07f47b793f.slice/crio-9e30587763202664c3989c47cc0c0a5565d9290f57b6e92b1964f43ad4070dea WatchSource:0}: Error finding container 9e30587763202664c3989c47cc0c0a5565d9290f57b6e92b1964f43ad4070dea: Status 404 returned error can't find the container with id 9e30587763202664c3989c47cc0c0a5565d9290f57b6e92b1964f43ad4070dea
Jul 17 21:09:17.918034 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:17.917984    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd"]
Jul 17 21:09:17.922498 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:09:17.922207    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d1ffa9_81dd_461a_84b8_d0a69be2b619.slice/crio-1587c53fb9bccc7d1d967bc134a3c090dcf30d9987433dc2b4f4e2d5e0b1f6fa WatchSource:0}: Error finding container 1587c53fb9bccc7d1d967bc134a3c090dcf30d9987433dc2b4f4e2d5e0b1f6fa: Status 404 returned error can't find the container with id 1587c53fb9bccc7d1d967bc134a3c090dcf30d9987433dc2b4f4e2d5e0b1f6fa
Jul 17 21:09:18.839985 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:18.839888    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd" event={"ID":"f8d1ffa9-81dd-461a-84b8-d0a69be2b619","Type":"ContainerStarted","Data":"1587c53fb9bccc7d1d967bc134a3c090dcf30d9987433dc2b4f4e2d5e0b1f6fa"}
Jul 17 21:09:18.848686 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:18.848653    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp" event={"ID":"3cb998da-5aa4-41f7-9721-df07f47b793f","Type":"ContainerStarted","Data":"9e30587763202664c3989c47cc0c0a5565d9290f57b6e92b1964f43ad4070dea"}
Jul 17 21:09:18.854882 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:18.854787    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnp9k" event={"ID":"96edb730-a088-46cb-89e6-731a4d05f8cc","Type":"ContainerStarted","Data":"50da6dee0ded60057325a6c781da7d8ccddca81e623b11656a09076581ba7279"}
Jul 17 21:09:18.864741 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:18.864711    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2m9sr" event={"ID":"febc06a9-decb-4fb6-9b31-443a339262b9","Type":"ContainerStarted","Data":"26b8e3039336b35c9c91e1b2d0562893b9e77e06855230d889e2fe6ea951c4c3"}
Jul 17 21:09:18.869274 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:18.869152    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-749775c58-8ldvv" event={"ID":"78f540df-428a-432e-ac03-09f58a3caa8c","Type":"ContainerStarted","Data":"a18323c19676d9e0e39ce0e6a28006f5154bd9b3ca5b514e890388b5c88d8e0b"}
Jul 17 21:09:18.869274 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:18.869206    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-749775c58-8ldvv" event={"ID":"78f540df-428a-432e-ac03-09f58a3caa8c","Type":"ContainerStarted","Data":"86ceeaf8225a3e00bdf8bc1d1ec1a92ab6a0faf40d5d9f43c3988441869f7002"}
Jul 17 21:09:18.870009 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:18.869988    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:19.776664 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:19.776633    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pkkqp"
Jul 17 21:09:19.806365 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:19.805787    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-749775c58-8ldvv" podStartSLOduration=59.805768157 podStartE2EDuration="59.805768157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:18.911464526 +0000 UTC m=+58.875585139" watchObservedRunningTime="2026-07-17 21:09:19.805768157 +0000 UTC m=+59.769888768"
Jul 17 21:09:21.978836 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:21.977789    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ctqcz"]
Jul 17 21:09:21.980661 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:21.980116    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ctqcz"
Jul 17 21:09:21.984450 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:21.983574    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 17 21:09:21.984450 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:21.983806    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-sz7w2\""
Jul 17 21:09:21.991791 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:21.991540    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ctqcz"]
Jul 17 21:09:22.059064 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:22.059027    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ddbb92d2-c9c7-4e7c-ae12-28693c93de88-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-ctqcz\" (UID: \"ddbb92d2-c9c7-4e7c-ae12-28693c93de88\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ctqcz"
Jul 17 21:09:22.159553 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:22.159475    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ddbb92d2-c9c7-4e7c-ae12-28693c93de88-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-ctqcz\" (UID: \"ddbb92d2-c9c7-4e7c-ae12-28693c93de88\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ctqcz"
Jul 17 21:09:22.165932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:22.165877    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ddbb92d2-c9c7-4e7c-ae12-28693c93de88-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-ctqcz\" (UID: \"ddbb92d2-c9c7-4e7c-ae12-28693c93de88\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ctqcz"
Jul 17 21:09:22.295130 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:22.295035    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ctqcz"
Jul 17 21:09:26.195038 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:26.194993    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret\") pod \"global-pull-secret-syncer-6lpt4\" (UID: \"ce0a4720-2597-4f74-8e8a-012be51716e8\") " pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:09:26.195531 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:26.195158    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs\") pod \"network-metrics-daemon-87nv2\" (UID: \"4147724d-819d-4372-b2c8-c170b2d71411\") " pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:09:26.197342 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:26.197315    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 21:09:26.197462 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:26.197428    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 21:09:26.208497 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:26.208472    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4147724d-819d-4372-b2c8-c170b2d71411-metrics-certs\") pod \"network-metrics-daemon-87nv2\" (UID: \"4147724d-819d-4372-b2c8-c170b2d71411\") " pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:09:26.208612 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:26.208517    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ce0a4720-2597-4f74-8e8a-012be51716e8-original-pull-secret\") pod \"global-pull-secret-syncer-6lpt4\" (UID: \"ce0a4720-2597-4f74-8e8a-012be51716e8\") " pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:09:26.295988 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:26.295953    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zw6\" (UniqueName: \"kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6\") pod \"network-check-target-pd8zp\" (UID: \"fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae\") " pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:09:26.298745 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:26.298717    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 21:09:26.308730 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:26.308694    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 21:09:26.320031 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:26.320012    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8zw6\" (UniqueName: \"kubernetes.io/projected/fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae-kube-api-access-w8zw6\") pod \"network-check-target-pd8zp\" (UID: \"fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae\") " pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:09:26.439208 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:26.439177    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8vkwx\""
Jul 17 21:09:26.445167 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:26.445112    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-szcqz\""
Jul 17 21:09:26.446755 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:26.446728    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:09:26.448274 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:26.448255    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lpt4"
Jul 17 21:09:26.453966 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:26.453948    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-87nv2"
Jul 17 21:09:28.390215 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.389727    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ctqcz"]
Jul 17 21:09:28.443794 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.443722    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pd8zp"]
Jul 17 21:09:28.463175 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.463150    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-87nv2"]
Jul 17 21:09:28.464690 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.464570    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6lpt4"]
Jul 17 21:09:28.478207 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:09:28.478170    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0a4720_2597_4f74_8e8a_012be51716e8.slice/crio-d90463f0a7157867b468f953f1836650415f4d5bc561612205639eb193a03a1f WatchSource:0}: Error finding container d90463f0a7157867b468f953f1836650415f4d5bc561612205639eb193a03a1f: Status 404 returned error can't find the container with id d90463f0a7157867b468f953f1836650415f4d5bc561612205639eb193a03a1f
Jul 17 21:09:28.481743 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:09:28.481553    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4147724d_819d_4372_b2c8_c170b2d71411.slice/crio-d824a5fdd332bd8769a44488046defc2e87ed8afa39875402f80ab6610dfce44 WatchSource:0}: Error finding container d824a5fdd332bd8769a44488046defc2e87ed8afa39875402f80ab6610dfce44: Status 404 returned error can't find the container with id d824a5fdd332bd8769a44488046defc2e87ed8afa39875402f80ab6610dfce44
Jul 17 21:09:28.923898 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.923863    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd" event={"ID":"f8d1ffa9-81dd-461a-84b8-d0a69be2b619","Type":"ContainerStarted","Data":"f1e838b40d75e60b907facd2f9e51f994d732ee2d226fba7f8f076410fd7e49d"}
Jul 17 21:09:28.925414 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.924399    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd"
Jul 17 21:09:28.926106 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.926062    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd"
Jul 17 21:09:28.927682 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.927649    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp" event={"ID":"3cb998da-5aa4-41f7-9721-df07f47b793f","Type":"ContainerStarted","Data":"8c1ddd9a6f03f7e591ccdabd70f4e96f6f144ac027d4fc5544c4cddcc39fee05"}
Jul 17 21:09:28.927767 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.927686    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp" event={"ID":"3cb998da-5aa4-41f7-9721-df07f47b793f","Type":"ContainerStarted","Data":"1701126f694db745167c6826b4621fde131661340f793a983aeaacfb8f85eb53"}
Jul 17 21:09:28.927767 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.927704    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp" event={"ID":"3cb998da-5aa4-41f7-9721-df07f47b793f","Type":"ContainerStarted","Data":"be5b0fe2a602f197c76a93848d9f53ddc9fe71a5f1208c372bbd854dc9d0d013"}
Jul 17 21:09:28.929206 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.929186    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pd8zp" event={"ID":"fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae","Type":"ContainerStarted","Data":"cf201b101c7762a3f0c3d7472f8f2cd704d6d8e588c53ec4d92483c858b50068"}
Jul 17 21:09:28.932156 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.931638    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m6kll" event={"ID":"8cf5abf4-6b79-4ed0-bf87-a45cddb80aa7","Type":"ContainerStarted","Data":"f6b006c029b75b90d7aee28398330844308692db581b2aaabbaf6a1839e794d3"}
Jul 17 21:09:28.934594 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.934572    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnp9k" event={"ID":"96edb730-a088-46cb-89e6-731a4d05f8cc","Type":"ContainerStarted","Data":"1bb6b70915bc3f5617d040b4bbb12a6d1d2ff576ee6b9dbed0d7193f60c6c1ab"}
Jul 17 21:09:28.937958 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.937794    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2m9sr" event={"ID":"febc06a9-decb-4fb6-9b31-443a339262b9","Type":"ContainerStarted","Data":"4c60fd5cf1a4d708b4f00614c38526501a63833649acceea6ec0df5bbc54ef56"}
Jul 17 21:09:28.937958 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.937837    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2m9sr" event={"ID":"febc06a9-decb-4fb6-9b31-443a339262b9","Type":"ContainerStarted","Data":"623d7d31e73b3035c7c46b1beac0aa2afa5b7f3f7474e33f85b0497bcc7c4ea5"}
Jul 17 21:09:28.938270 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.938224    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2m9sr"
Jul 17 21:09:28.941420 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.941376    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd8c849bc-ml6kd" podStartSLOduration=1.672200516 podStartE2EDuration="11.941358977s" totalImagesPullingTime="10.269158461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:28.940278989 +0000 UTC m=+68.904399600" watchObservedRunningTime="2026-07-17 21:09:28.941358977 +0000 UTC m=+68.905479590"
Jul 17 21:09:28.942575 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.942549    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87nv2" event={"ID":"4147724d-819d-4372-b2c8-c170b2d71411","Type":"ContainerStarted","Data":"d824a5fdd332bd8769a44488046defc2e87ed8afa39875402f80ab6610dfce44"}
Jul 17 21:09:28.944009 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.943949    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ctqcz" event={"ID":"ddbb92d2-c9c7-4e7c-ae12-28693c93de88","Type":"ContainerStarted","Data":"d8ff6492992e9bc433ed7489cbe4c4e770872ed42281082454d571d20333752b"}
Jul 17 21:09:28.946969 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.946948    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j" event={"ID":"01a1222a-6dce-493f-bade-6f46be398b8e","Type":"ContainerStarted","Data":"bb178fc723a2ae454260ed692f60e409e0b98b178287be2ed5b0e68bfdae67dc"}
Jul 17 21:09:28.948918 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.948899    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6lpt4" event={"ID":"ce0a4720-2597-4f74-8e8a-012be51716e8","Type":"ContainerStarted","Data":"d90463f0a7157867b468f953f1836650415f4d5bc561612205639eb193a03a1f"}
Jul 17 21:09:28.973039 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:28.972016    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m6kll" podStartSLOduration=1.541461739 podStartE2EDuration="11.9719998s" totalImagesPullingTime="10.430538061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:28.956201823 +0000 UTC m=+68.920322435" watchObservedRunningTime="2026-07-17 21:09:28.9719998 +0000 UTC m=+68.936120408"
Jul 17 21:09:29.013207 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:29.013149    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-596d656b9-bhfxp" podStartSLOduration=1.718876072 podStartE2EDuration="12.013124643s" totalImagesPullingTime="10.294248571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:28.993038483 +0000 UTC m=+68.957159095" watchObservedRunningTime="2026-07-17 21:09:29.013124643 +0000 UTC m=+68.977245256"
Jul 17 21:09:29.014605 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:29.014088    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2m9sr" podStartSLOduration=1.707283775 podStartE2EDuration="12.014079335s" totalImagesPullingTime="10.30679556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:29.013959117 +0000 UTC m=+68.978079729" watchObservedRunningTime="2026-07-17 21:09:29.014079335 +0000 UTC m=+68.978199947"
Jul 17 21:09:29.030080 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:29.029618    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-776f6dbfb8-mkm4j" podStartSLOduration=1.512819543 podStartE2EDuration="12.029604598s" totalImagesPullingTime="10.516785055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:29.027941521 +0000 UTC m=+68.992225323" watchObservedRunningTime="2026-07-17 21:09:29.029604598 +0000 UTC m=+68.993725207"
Jul 17 21:09:35.978386 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:35.977931    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pd8zp" event={"ID":"fd9a2eb8-bfb7-46eb-89b9-f00543a1fcae","Type":"ContainerStarted","Data":"d70647c18b694b832fa2223005f624159afa48760a4c6ea1e1940b3f2f254ece"}
Jul 17 21:09:35.978386 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:35.978021    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:09:35.980012 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:35.979952    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnp9k" event={"ID":"96edb730-a088-46cb-89e6-731a4d05f8cc","Type":"ContainerStarted","Data":"e3af754e35f5005d711fd22a9bc74002d40954765640d5eb979e091e3814d320"}
Jul 17 21:09:35.981898 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:35.981868    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87nv2" event={"ID":"4147724d-819d-4372-b2c8-c170b2d71411","Type":"ContainerStarted","Data":"60c8e9050c7b28b51834f888aabab9a0672e028526558fa6a78348bd3b372104"}
Jul 17 21:09:35.982007 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:35.981912    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-87nv2" event={"ID":"4147724d-819d-4372-b2c8-c170b2d71411","Type":"ContainerStarted","Data":"c07a4fd289f65e8c04ebaea99f658b7fae25487013b0079a1ae7daaab821d9f9"}
Jul 17 21:09:35.983630 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:35.983595    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ctqcz" event={"ID":"ddbb92d2-c9c7-4e7c-ae12-28693c93de88","Type":"ContainerStarted","Data":"3bc67b21c96b082dd25bb8ce7b7eba43b40f653460679fbf5efc3d2192e373f3"}
Jul 17 21:09:35.983798 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:35.983773    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ctqcz"
Jul 17 21:09:35.985403 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:35.985380    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6lpt4" event={"ID":"ce0a4720-2597-4f74-8e8a-012be51716e8","Type":"ContainerStarted","Data":"b8680960ff31a549e89f62f8aa3b4b741a1f64926920b91ceeaa64224218d918"}
Jul 17 21:09:35.989465 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:35.989437    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ctqcz"
Jul 17 21:09:35.996509 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:35.996473    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pd8zp" podStartSLOduration=68.340053563 podStartE2EDuration="1m14.996462828s" totalImagesPullingTime="6.656409265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:35.994598123 +0000 UTC m=+75.958718733" watchObservedRunningTime="2026-07-17 21:09:35.996462828 +0000 UTC m=+75.960583437"
Jul 17 21:09:36.014002 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:36.013939    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-87nv2" podStartSLOduration=68.380857353 podStartE2EDuration="1m15.013923157s" totalImagesPullingTime="6.633065804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:36.013513597 +0000 UTC m=+75.977634207" watchObservedRunningTime="2026-07-17 21:09:36.013923157 +0000 UTC m=+75.978043768"
Jul 17 21:09:36.027715 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:36.027673    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6lpt4" podStartSLOduration=68.392263928 podStartE2EDuration="1m15.027657272s" totalImagesPullingTime="6.635393344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:08:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:36.026941083 +0000 UTC m=+75.991061697" watchObservedRunningTime="2026-07-17 21:09:36.027657272 +0000 UTC m=+75.991777883"
Jul 17 21:09:36.040766 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:36.040730    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-ctqcz" podStartSLOduration=8.323225634 podStartE2EDuration="15.040717234s" totalImagesPullingTime="6.7174916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:36.04032423 +0000 UTC m=+76.004444842" watchObservedRunningTime="2026-07-17 21:09:36.040717234 +0000 UTC m=+76.004837846"
Jul 17 21:09:37.743538 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:37.743352    2573 patch_prober.go:28] interesting pod/image-registry-749775c58-8ldvv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 21:09:37.743538 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:37.743422    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-749775c58-8ldvv" podUID="78f540df-428a-432e-ac03-09f58a3caa8c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 21:09:38.997714 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:38.997642    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnp9k" event={"ID":"96edb730-a088-46cb-89e6-731a4d05f8cc","Type":"ContainerStarted","Data":"43ac0f478919fd08f67b27c5d0e3b98e458c94d50597d0b7999b492663b2b9d3"}
Jul 17 21:09:39.015104 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:39.015057    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dnp9k" podStartSLOduration=1.427389662 podStartE2EDuration="22.015041517s" totalImagesPullingTime="20.587651855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:09:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:39.013415419 +0000 UTC m=+78.977536041" watchObservedRunningTime="2026-07-17 21:09:39.015041517 +0000 UTC m=+78.979162128"
Jul 17 21:09:39.960504 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:39.960471    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2m9sr"
Jul 17 21:09:40.893171 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:40.893142    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:09:42.496501 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.496472    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-njrz6"]
Jul 17 21:09:42.499789 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.499767    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.503536 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.503509    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 21:09:42.503904 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.503885    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6lcn7\""
Jul 17 21:09:42.503999 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.503917    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 21:09:42.504295 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.504262    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 21:09:42.504455 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.504428    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 21:09:42.504762 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.504746    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 21:09:42.504853 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.504780    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 21:09:42.636146 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.636122    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c558f3b-21ce-49a4-95ec-82537abae35a-metrics-client-ca\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.636258 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.636148    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qhdw\" (UniqueName: \"kubernetes.io/projected/3c558f3b-21ce-49a4-95ec-82537abae35a-kube-api-access-8qhdw\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.636258 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.636172    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c558f3b-21ce-49a4-95ec-82537abae35a-sys\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.636258 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.636188    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c558f3b-21ce-49a4-95ec-82537abae35a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.636258 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.636241    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3c558f3b-21ce-49a4-95ec-82537abae35a-node-exporter-wtmp\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.636426 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.636274    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3c558f3b-21ce-49a4-95ec-82537abae35a-node-exporter-tls\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.636426 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.636325    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3c558f3b-21ce-49a4-95ec-82537abae35a-node-exporter-textfile\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.636426 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.636385    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3c558f3b-21ce-49a4-95ec-82537abae35a-root\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.636525 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.636423    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3c558f3b-21ce-49a4-95ec-82537abae35a-node-exporter-accelerators-collector-config\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.737678 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.737655    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3c558f3b-21ce-49a4-95ec-82537abae35a-node-exporter-accelerators-collector-config\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.737762 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.737691    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c558f3b-21ce-49a4-95ec-82537abae35a-metrics-client-ca\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.737762 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.737713    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qhdw\" (UniqueName: \"kubernetes.io/projected/3c558f3b-21ce-49a4-95ec-82537abae35a-kube-api-access-8qhdw\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.737762 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.737748    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c558f3b-21ce-49a4-95ec-82537abae35a-sys\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.737891 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.737771    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c558f3b-21ce-49a4-95ec-82537abae35a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.737891 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.737796    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3c558f3b-21ce-49a4-95ec-82537abae35a-node-exporter-wtmp\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.737891 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.737851    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c558f3b-21ce-49a4-95ec-82537abae35a-sys\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.737891 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.737864    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3c558f3b-21ce-49a4-95ec-82537abae35a-node-exporter-tls\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.738080 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.737900    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3c558f3b-21ce-49a4-95ec-82537abae35a-node-exporter-textfile\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.738080 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.737930    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3c558f3b-21ce-49a4-95ec-82537abae35a-node-exporter-wtmp\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.738080 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.737933    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3c558f3b-21ce-49a4-95ec-82537abae35a-root\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.738080 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.737973    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3c558f3b-21ce-49a4-95ec-82537abae35a-root\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.738080 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:09:42.737988    2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 17 21:09:42.738080 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:09:42.738043    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c558f3b-21ce-49a4-95ec-82537abae35a-node-exporter-tls podName:3c558f3b-21ce-49a4-95ec-82537abae35a nodeName:}" failed. No retries permitted until 2026-07-17 21:09:43.238025189 +0000 UTC m=+83.202145777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3c558f3b-21ce-49a4-95ec-82537abae35a-node-exporter-tls") pod "node-exporter-njrz6" (UID: "3c558f3b-21ce-49a4-95ec-82537abae35a") : secret "node-exporter-tls" not found
Jul 17 21:09:42.738377 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.738297    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3c558f3b-21ce-49a4-95ec-82537abae35a-node-exporter-textfile\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.738377 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.738363    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c558f3b-21ce-49a4-95ec-82537abae35a-metrics-client-ca\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.738460 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.738386    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3c558f3b-21ce-49a4-95ec-82537abae35a-node-exporter-accelerators-collector-config\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.739992 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.739971    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c558f3b-21ce-49a4-95ec-82537abae35a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:42.746348 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:42.746327    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qhdw\" (UniqueName: \"kubernetes.io/projected/3c558f3b-21ce-49a4-95ec-82537abae35a-kube-api-access-8qhdw\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:43.150120 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:43.150089    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-749775c58-8ldvv"]
Jul 17 21:09:43.242212 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:43.242187    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3c558f3b-21ce-49a4-95ec-82537abae35a-node-exporter-tls\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:43.244252 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:43.244234    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3c558f3b-21ce-49a4-95ec-82537abae35a-node-exporter-tls\") pod \"node-exporter-njrz6\" (UID: \"3c558f3b-21ce-49a4-95ec-82537abae35a\") " pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:43.408654 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:43.408587    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-njrz6"
Jul 17 21:09:43.421316 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:09:43.421275    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c558f3b_21ce_49a4_95ec_82537abae35a.slice/crio-e88b298b9d032f37161dd9db67173116a5dc3ae8a2b2040002cdf8fbbadffc19 WatchSource:0}: Error finding container e88b298b9d032f37161dd9db67173116a5dc3ae8a2b2040002cdf8fbbadffc19: Status 404 returned error can't find the container with id e88b298b9d032f37161dd9db67173116a5dc3ae8a2b2040002cdf8fbbadffc19
Jul 17 21:09:44.020400 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:44.020346    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-njrz6" event={"ID":"3c558f3b-21ce-49a4-95ec-82537abae35a","Type":"ContainerStarted","Data":"e88b298b9d032f37161dd9db67173116a5dc3ae8a2b2040002cdf8fbbadffc19"}
Jul 17 21:09:45.025534 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:45.025498    2573 generic.go:356] "Generic (PLEG): container finished" podID="3c558f3b-21ce-49a4-95ec-82537abae35a" containerID="03e549c557de250f756b8053b77f6044b16b6ab423af80b73292a26e1c59263e" exitCode=0
Jul 17 21:09:45.025903 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:45.025570    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-njrz6" event={"ID":"3c558f3b-21ce-49a4-95ec-82537abae35a","Type":"ContainerDied","Data":"03e549c557de250f756b8053b77f6044b16b6ab423af80b73292a26e1c59263e"}
Jul 17 21:09:46.030842 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:46.030805    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-njrz6" event={"ID":"3c558f3b-21ce-49a4-95ec-82537abae35a","Type":"ContainerStarted","Data":"fe96502a114968a52aaeef88d4b68b866160389aaee2638309cc1a865912e59c"}
Jul 17 21:09:46.031271 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:46.030863    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-njrz6" event={"ID":"3c558f3b-21ce-49a4-95ec-82537abae35a","Type":"ContainerStarted","Data":"9a633ac04e44a971c809b1ebe883249cc078df704466b72f9b15e2d5e6c2e503"}
Jul 17 21:09:46.051521 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:09:46.051476    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-njrz6" podStartSLOduration=2.958390401 podStartE2EDuration="4.051462007s" totalImagesPullingTime="1.032200657s" totalInitContainerRuntime="60.870949ms" isStatefulPod=true podCreationTimestamp="2026-07-17 21:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:09:46.049699318 +0000 UTC m=+86.013819927" watchObservedRunningTime="2026-07-17 21:09:46.051462007 +0000 UTC m=+86.015582617"
Jul 17 21:10:06.991934 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:06.991904    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pd8zp"
Jul 17 21:10:08.174482 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:08.174419    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-749775c58-8ldvv" podUID="78f540df-428a-432e-ac03-09f58a3caa8c" containerName="registry" containerID="cri-o://a18323c19676d9e0e39ce0e6a28006f5154bd9b3ca5b514e890388b5c88d8e0b" gracePeriod=30
Jul 17 21:10:08.934644 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:08.934624    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:10:09.111082 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.111051    2573 generic.go:356] "Generic (PLEG): container finished" podID="78f540df-428a-432e-ac03-09f58a3caa8c" containerID="a18323c19676d9e0e39ce0e6a28006f5154bd9b3ca5b514e890388b5c88d8e0b" exitCode=0
Jul 17 21:10:09.111240 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.111100    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-749775c58-8ldvv"
Jul 17 21:10:09.111240 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.111115    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-749775c58-8ldvv" event={"ID":"78f540df-428a-432e-ac03-09f58a3caa8c","Type":"ContainerDied","Data":"a18323c19676d9e0e39ce0e6a28006f5154bd9b3ca5b514e890388b5c88d8e0b"}
Jul 17 21:10:09.111240 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.111154    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-749775c58-8ldvv" event={"ID":"78f540df-428a-432e-ac03-09f58a3caa8c","Type":"ContainerDied","Data":"86ceeaf8225a3e00bdf8bc1d1ec1a92ab6a0faf40d5d9f43c3988441869f7002"}
Jul 17 21:10:09.111240 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.111174    2573 scope.go:117] "RemoveContainer" containerID="a18323c19676d9e0e39ce0e6a28006f5154bd9b3ca5b514e890388b5c88d8e0b"
Jul 17 21:10:09.118622 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.118606    2573 scope.go:117] "RemoveContainer" containerID="a18323c19676d9e0e39ce0e6a28006f5154bd9b3ca5b514e890388b5c88d8e0b"
Jul 17 21:10:09.118706 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.118690    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78f540df-428a-432e-ac03-09f58a3caa8c-installation-pull-secrets\") pod \"78f540df-428a-432e-ac03-09f58a3caa8c\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") "
Jul 17 21:10:09.118749 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.118726    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78f540df-428a-432e-ac03-09f58a3caa8c-trusted-ca\") pod \"78f540df-428a-432e-ac03-09f58a3caa8c\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") "
Jul 17 21:10:09.118789 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.118752    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78f540df-428a-432e-ac03-09f58a3caa8c-ca-trust-extracted\") pod \"78f540df-428a-432e-ac03-09f58a3caa8c\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") "
Jul 17 21:10:09.118842 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.118814    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78f540df-428a-432e-ac03-09f58a3caa8c-registry-certificates\") pod \"78f540df-428a-432e-ac03-09f58a3caa8c\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") "
Jul 17 21:10:09.118891 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.118840    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78f540df-428a-432e-ac03-09f58a3caa8c-bound-sa-token\") pod \"78f540df-428a-432e-ac03-09f58a3caa8c\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") "
Jul 17 21:10:09.118891 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.118865    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78f540df-428a-432e-ac03-09f58a3caa8c-registry-tls\") pod \"78f540df-428a-432e-ac03-09f58a3caa8c\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") "
Jul 17 21:10:09.118891 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.118880    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wmp8\" (UniqueName: \"kubernetes.io/projected/78f540df-428a-432e-ac03-09f58a3caa8c-kube-api-access-7wmp8\") pod \"78f540df-428a-432e-ac03-09f58a3caa8c\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") "
Jul 17 21:10:09.119030 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.118900    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78f540df-428a-432e-ac03-09f58a3caa8c-image-registry-private-configuration\") pod \"78f540df-428a-432e-ac03-09f58a3caa8c\" (UID: \"78f540df-428a-432e-ac03-09f58a3caa8c\") "
Jul 17 21:10:09.119030 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:10:09.118895    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a18323c19676d9e0e39ce0e6a28006f5154bd9b3ca5b514e890388b5c88d8e0b\": container with ID starting with a18323c19676d9e0e39ce0e6a28006f5154bd9b3ca5b514e890388b5c88d8e0b not found: ID does not exist" containerID="a18323c19676d9e0e39ce0e6a28006f5154bd9b3ca5b514e890388b5c88d8e0b"
Jul 17 21:10:09.119128 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.118933    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18323c19676d9e0e39ce0e6a28006f5154bd9b3ca5b514e890388b5c88d8e0b"} err="failed to get container status \"a18323c19676d9e0e39ce0e6a28006f5154bd9b3ca5b514e890388b5c88d8e0b\": rpc error: code = NotFound desc = could not find container \"a18323c19676d9e0e39ce0e6a28006f5154bd9b3ca5b514e890388b5c88d8e0b\": container with ID starting with a18323c19676d9e0e39ce0e6a28006f5154bd9b3ca5b514e890388b5c88d8e0b not found: ID does not exist"
Jul 17 21:10:09.119421 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.119198    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78f540df-428a-432e-ac03-09f58a3caa8c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "78f540df-428a-432e-ac03-09f58a3caa8c" (UID: "78f540df-428a-432e-ac03-09f58a3caa8c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:10:09.119421 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.119224    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78f540df-428a-432e-ac03-09f58a3caa8c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "78f540df-428a-432e-ac03-09f58a3caa8c" (UID: "78f540df-428a-432e-ac03-09f58a3caa8c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:10:09.121444 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.121408    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f540df-428a-432e-ac03-09f58a3caa8c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "78f540df-428a-432e-ac03-09f58a3caa8c" (UID: "78f540df-428a-432e-ac03-09f58a3caa8c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:10:09.121695 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.121669    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f540df-428a-432e-ac03-09f58a3caa8c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "78f540df-428a-432e-ac03-09f58a3caa8c" (UID: "78f540df-428a-432e-ac03-09f58a3caa8c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:10:09.121695 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.121681    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78f540df-428a-432e-ac03-09f58a3caa8c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "78f540df-428a-432e-ac03-09f58a3caa8c" (UID: "78f540df-428a-432e-ac03-09f58a3caa8c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:10:09.121857 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.121724    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f540df-428a-432e-ac03-09f58a3caa8c-kube-api-access-7wmp8" (OuterVolumeSpecName: "kube-api-access-7wmp8") pod "78f540df-428a-432e-ac03-09f58a3caa8c" (UID: "78f540df-428a-432e-ac03-09f58a3caa8c"). InnerVolumeSpecName "kube-api-access-7wmp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:10:09.121857 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.121737    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78f540df-428a-432e-ac03-09f58a3caa8c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "78f540df-428a-432e-ac03-09f58a3caa8c" (UID: "78f540df-428a-432e-ac03-09f58a3caa8c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:10:09.127910 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.127885    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f540df-428a-432e-ac03-09f58a3caa8c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "78f540df-428a-432e-ac03-09f58a3caa8c" (UID: "78f540df-428a-432e-ac03-09f58a3caa8c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:10:09.220067 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.220041    2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78f540df-428a-432e-ac03-09f58a3caa8c-image-registry-private-configuration\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:10:09.220067 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.220062    2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78f540df-428a-432e-ac03-09f58a3caa8c-installation-pull-secrets\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:10:09.220397 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.220072    2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78f540df-428a-432e-ac03-09f58a3caa8c-trusted-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:10:09.220397 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.220082    2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78f540df-428a-432e-ac03-09f58a3caa8c-ca-trust-extracted\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:10:09.220397 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.220090    2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78f540df-428a-432e-ac03-09f58a3caa8c-registry-certificates\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:10:09.220397 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.220099    2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78f540df-428a-432e-ac03-09f58a3caa8c-bound-sa-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:10:09.220397 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.220107    2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78f540df-428a-432e-ac03-09f58a3caa8c-registry-tls\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:10:09.220397 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.220114    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wmp8\" (UniqueName: \"kubernetes.io/projected/78f540df-428a-432e-ac03-09f58a3caa8c-kube-api-access-7wmp8\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:10:09.435076 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.435048    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-749775c58-8ldvv"]
Jul 17 21:10:09.438747 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:09.438725    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-749775c58-8ldvv"]
Jul 17 21:10:10.528643 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:10:10.528610    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78f540df-428a-432e-ac03-09f58a3caa8c" path="/var/lib/kubelet/pods/78f540df-428a-432e-ac03-09f58a3caa8c/volumes"
Jul 17 21:13:20.399510 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:13:20.399487    2573 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 21:14:09.526770 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:09.526736    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-prcpv"]
Jul 17 21:14:09.527172 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:09.526956    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f540df-428a-432e-ac03-09f58a3caa8c" containerName="registry"
Jul 17 21:14:09.527172 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:09.526967    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f540df-428a-432e-ac03-09f58a3caa8c" containerName="registry"
Jul 17 21:14:09.527172 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:09.527014    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="78f540df-428a-432e-ac03-09f58a3caa8c" containerName="registry"
Jul 17 21:14:09.529621 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:09.529604    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-prcpv"
Jul 17 21:14:09.532258 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:09.532234    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 17 21:14:09.532427 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:09.532234    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 17 21:14:09.532427 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:09.532302    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-pckxk\""
Jul 17 21:14:09.541259 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:09.541239    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-prcpv"]
Jul 17 21:14:09.544445 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:09.544426    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqp5t\" (UniqueName: \"kubernetes.io/projected/0ffc219e-734a-49b3-a186-0b0cf91e9268-kube-api-access-vqp5t\") pod \"cert-manager-webhook-6486ffd997-prcpv\" (UID: \"0ffc219e-734a-49b3-a186-0b0cf91e9268\") " pod="cert-manager/cert-manager-webhook-6486ffd997-prcpv"
Jul 17 21:14:09.544532 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:09.544454    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ffc219e-734a-49b3-a186-0b0cf91e9268-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-prcpv\" (UID: \"0ffc219e-734a-49b3-a186-0b0cf91e9268\") " pod="cert-manager/cert-manager-webhook-6486ffd997-prcpv"
Jul 17 21:14:09.645401 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:09.645376    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqp5t\" (UniqueName: \"kubernetes.io/projected/0ffc219e-734a-49b3-a186-0b0cf91e9268-kube-api-access-vqp5t\") pod \"cert-manager-webhook-6486ffd997-prcpv\" (UID: \"0ffc219e-734a-49b3-a186-0b0cf91e9268\") " pod="cert-manager/cert-manager-webhook-6486ffd997-prcpv"
Jul 17 21:14:09.645508 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:09.645405    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ffc219e-734a-49b3-a186-0b0cf91e9268-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-prcpv\" (UID: \"0ffc219e-734a-49b3-a186-0b0cf91e9268\") " pod="cert-manager/cert-manager-webhook-6486ffd997-prcpv"
Jul 17 21:14:09.673636 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:09.673611    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0ffc219e-734a-49b3-a186-0b0cf91e9268-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-prcpv\" (UID: \"0ffc219e-734a-49b3-a186-0b0cf91e9268\") " pod="cert-manager/cert-manager-webhook-6486ffd997-prcpv"
Jul 17 21:14:09.675464 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:09.675443    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqp5t\" (UniqueName: \"kubernetes.io/projected/0ffc219e-734a-49b3-a186-0b0cf91e9268-kube-api-access-vqp5t\") pod \"cert-manager-webhook-6486ffd997-prcpv\" (UID: \"0ffc219e-734a-49b3-a186-0b0cf91e9268\") " pod="cert-manager/cert-manager-webhook-6486ffd997-prcpv"
Jul 17 21:14:09.837624 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:09.837603    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-prcpv"
Jul 17 21:14:09.969215 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:09.969156    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-prcpv"]
Jul 17 21:14:09.971391 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:14:09.971360    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ffc219e_734a_49b3_a186_0b0cf91e9268.slice/crio-ab8c16a18e2d115379cedbfdf2337768b87d6feaae56c7d06203cb34c45b72a2 WatchSource:0}: Error finding container ab8c16a18e2d115379cedbfdf2337768b87d6feaae56c7d06203cb34c45b72a2: Status 404 returned error can't find the container with id ab8c16a18e2d115379cedbfdf2337768b87d6feaae56c7d06203cb34c45b72a2
Jul 17 21:14:09.973315 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:09.973300    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 21:14:10.862250 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:10.862206    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-prcpv" event={"ID":"0ffc219e-734a-49b3-a186-0b0cf91e9268","Type":"ContainerStarted","Data":"ab8c16a18e2d115379cedbfdf2337768b87d6feaae56c7d06203cb34c45b72a2"}
Jul 17 21:14:15.881118 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:15.881087    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-prcpv" event={"ID":"0ffc219e-734a-49b3-a186-0b0cf91e9268","Type":"ContainerStarted","Data":"5dc55275796c5a39db196c7c1d27de95ba92877a5cb8ede5c59f42d83e68f7a2"}
Jul 17 21:14:15.881493 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:15.881156    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-prcpv"
Jul 17 21:14:15.896216 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:15.896175    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-prcpv" podStartSLOduration=1.474820176 podStartE2EDuration="6.896163997s" totalImagesPullingTime="5.421343821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:14:15.895834826 +0000 UTC m=+355.859955432" watchObservedRunningTime="2026-07-17 21:14:15.896163997 +0000 UTC m=+355.860284607"
Jul 17 21:14:21.886851 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:21.886823    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-prcpv"
Jul 17 21:14:29.697406 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:29.697372    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-658kp"]
Jul 17 21:14:29.702796 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:29.702777    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-658kp"
Jul 17 21:14:29.705118 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:29.705096    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ltpgv\""
Jul 17 21:14:29.710571 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:29.710550    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-658kp"]
Jul 17 21:14:29.785018 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:29.784997    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbf0d34f-eb6b-4d7a-a155-cf0eb149186d-bound-sa-token\") pod \"cert-manager-55b4d954d5-658kp\" (UID: \"cbf0d34f-eb6b-4d7a-a155-cf0eb149186d\") " pod="cert-manager/cert-manager-55b4d954d5-658kp"
Jul 17 21:14:29.785113 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:29.785038    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hqbw\" (UniqueName: \"kubernetes.io/projected/cbf0d34f-eb6b-4d7a-a155-cf0eb149186d-kube-api-access-8hqbw\") pod \"cert-manager-55b4d954d5-658kp\" (UID: \"cbf0d34f-eb6b-4d7a-a155-cf0eb149186d\") " pod="cert-manager/cert-manager-55b4d954d5-658kp"
Jul 17 21:14:29.885961 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:29.885936    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbf0d34f-eb6b-4d7a-a155-cf0eb149186d-bound-sa-token\") pod \"cert-manager-55b4d954d5-658kp\" (UID: \"cbf0d34f-eb6b-4d7a-a155-cf0eb149186d\") " pod="cert-manager/cert-manager-55b4d954d5-658kp"
Jul 17 21:14:29.886044 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:29.885978    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hqbw\" (UniqueName: \"kubernetes.io/projected/cbf0d34f-eb6b-4d7a-a155-cf0eb149186d-kube-api-access-8hqbw\") pod \"cert-manager-55b4d954d5-658kp\" (UID: \"cbf0d34f-eb6b-4d7a-a155-cf0eb149186d\") " pod="cert-manager/cert-manager-55b4d954d5-658kp"
Jul 17 21:14:29.893821 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:29.893799    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hqbw\" (UniqueName: \"kubernetes.io/projected/cbf0d34f-eb6b-4d7a-a155-cf0eb149186d-kube-api-access-8hqbw\") pod \"cert-manager-55b4d954d5-658kp\" (UID: \"cbf0d34f-eb6b-4d7a-a155-cf0eb149186d\") " pod="cert-manager/cert-manager-55b4d954d5-658kp"
Jul 17 21:14:29.893821 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:29.893803    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbf0d34f-eb6b-4d7a-a155-cf0eb149186d-bound-sa-token\") pod \"cert-manager-55b4d954d5-658kp\" (UID: \"cbf0d34f-eb6b-4d7a-a155-cf0eb149186d\") " pod="cert-manager/cert-manager-55b4d954d5-658kp"
Jul 17 21:14:30.011888 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:30.011831    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-658kp"
Jul 17 21:14:30.131848 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:30.131808    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-658kp"]
Jul 17 21:14:30.134495 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:14:30.134470    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbf0d34f_eb6b_4d7a_a155_cf0eb149186d.slice/crio-2a741e6920b1e6c9a7c7b91e7fc7aaa6b1fbb46725f4cf79717af409914cd878 WatchSource:0}: Error finding container 2a741e6920b1e6c9a7c7b91e7fc7aaa6b1fbb46725f4cf79717af409914cd878: Status 404 returned error can't find the container with id 2a741e6920b1e6c9a7c7b91e7fc7aaa6b1fbb46725f4cf79717af409914cd878
Jul 17 21:14:30.930896 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:30.930858    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-658kp" event={"ID":"cbf0d34f-eb6b-4d7a-a155-cf0eb149186d","Type":"ContainerStarted","Data":"690e72f0a4c9951a1a2f9179932ed77cbc03ef5247d9441175022f1e248b5151"}
Jul 17 21:14:30.930896 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:30.930899    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-658kp" event={"ID":"cbf0d34f-eb6b-4d7a-a155-cf0eb149186d","Type":"ContainerStarted","Data":"2a741e6920b1e6c9a7c7b91e7fc7aaa6b1fbb46725f4cf79717af409914cd878"}
Jul 17 21:14:30.945924 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:30.945876    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-658kp" podStartSLOduration=1.945859861 podStartE2EDuration="1.945859861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:14:30.944958618 +0000 UTC m=+370.909079230" watchObservedRunningTime="2026-07-17 21:14:30.945859861 +0000 UTC m=+370.909980471"
Jul 17 21:14:44.574907 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.574875    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"]
Jul 17 21:14:44.577875 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.577860    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:14:44.580865 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.580847    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 21:14:44.581043 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.581025    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 17 21:14:44.581112 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.581099    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rvgtv\""
Jul 17 21:14:44.581338 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.581317    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 21:14:44.588023 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.588005    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 21:14:44.592822 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.592799    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"]
Jul 17 21:14:44.691962 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.691935    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8501dc13-6ab5-4a53-84b0-bb02ef46e114-manifests\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-smxrt\" (UID: \"8501dc13-6ab5-4a53-84b0-bb02ef46e114\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:14:44.692073 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.691970    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8501dc13-6ab5-4a53-84b0-bb02ef46e114-tmp\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-smxrt\" (UID: \"8501dc13-6ab5-4a53-84b0-bb02ef46e114\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:14:44.692073 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.691989    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8501dc13-6ab5-4a53-84b0-bb02ef46e114-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-smxrt\" (UID: \"8501dc13-6ab5-4a53-84b0-bb02ef46e114\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:14:44.692073 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.692034    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r94c\" (UniqueName: \"kubernetes.io/projected/8501dc13-6ab5-4a53-84b0-bb02ef46e114-kube-api-access-7r94c\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-smxrt\" (UID: \"8501dc13-6ab5-4a53-84b0-bb02ef46e114\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:14:44.692219 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.692074    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8501dc13-6ab5-4a53-84b0-bb02ef46e114-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-smxrt\" (UID: \"8501dc13-6ab5-4a53-84b0-bb02ef46e114\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:14:44.792679 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.792651    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8501dc13-6ab5-4a53-84b0-bb02ef46e114-manifests\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-smxrt\" (UID: \"8501dc13-6ab5-4a53-84b0-bb02ef46e114\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:14:44.792751 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.792688    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8501dc13-6ab5-4a53-84b0-bb02ef46e114-tmp\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-smxrt\" (UID: \"8501dc13-6ab5-4a53-84b0-bb02ef46e114\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:14:44.792751 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.792714    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8501dc13-6ab5-4a53-84b0-bb02ef46e114-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-smxrt\" (UID: \"8501dc13-6ab5-4a53-84b0-bb02ef46e114\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:14:44.792842 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.792749    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r94c\" (UniqueName: \"kubernetes.io/projected/8501dc13-6ab5-4a53-84b0-bb02ef46e114-kube-api-access-7r94c\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-smxrt\" (UID: \"8501dc13-6ab5-4a53-84b0-bb02ef46e114\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:14:44.792842 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.792774    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8501dc13-6ab5-4a53-84b0-bb02ef46e114-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-smxrt\" (UID: \"8501dc13-6ab5-4a53-84b0-bb02ef46e114\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:14:44.793179 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.793146    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8501dc13-6ab5-4a53-84b0-bb02ef46e114-manifests\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-smxrt\" (UID: \"8501dc13-6ab5-4a53-84b0-bb02ef46e114\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:14:44.795005 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.794972    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8501dc13-6ab5-4a53-84b0-bb02ef46e114-tmp\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-smxrt\" (UID: \"8501dc13-6ab5-4a53-84b0-bb02ef46e114\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:14:44.795110 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.795007    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8501dc13-6ab5-4a53-84b0-bb02ef46e114-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-smxrt\" (UID: \"8501dc13-6ab5-4a53-84b0-bb02ef46e114\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:14:44.795110 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.795066    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8501dc13-6ab5-4a53-84b0-bb02ef46e114-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-smxrt\" (UID: \"8501dc13-6ab5-4a53-84b0-bb02ef46e114\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:14:44.801323 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.801278    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r94c\" (UniqueName: \"kubernetes.io/projected/8501dc13-6ab5-4a53-84b0-bb02ef46e114-kube-api-access-7r94c\") pod \"opendatahub-operator-controller-manager-6b8f6b79df-smxrt\" (UID: \"8501dc13-6ab5-4a53-84b0-bb02ef46e114\") " pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:14:44.888211 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:44.888180    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:14:45.016667 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:45.016645    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"]
Jul 17 21:14:45.017675 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:14:45.017649    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8501dc13_6ab5_4a53_84b0_bb02ef46e114.slice/crio-1de97171e12dff9e41c5a677d2a0e60405e8d86e7903f917c42b9b78fe7a3dfb WatchSource:0}: Error finding container 1de97171e12dff9e41c5a677d2a0e60405e8d86e7903f917c42b9b78fe7a3dfb: Status 404 returned error can't find the container with id 1de97171e12dff9e41c5a677d2a0e60405e8d86e7903f917c42b9b78fe7a3dfb
Jul 17 21:14:45.981848 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:45.981804    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt" event={"ID":"8501dc13-6ab5-4a53-84b0-bb02ef46e114","Type":"ContainerStarted","Data":"1de97171e12dff9e41c5a677d2a0e60405e8d86e7903f917c42b9b78fe7a3dfb"}
Jul 17 21:14:48.993701 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:48.993669    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt" event={"ID":"8501dc13-6ab5-4a53-84b0-bb02ef46e114","Type":"ContainerStarted","Data":"b6ac7d18f41bca75f45fe34bc884b86828452f482cbecc3e675d5c513755cade"}
Jul 17 21:14:49.998616 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:49.998577    2573 generic.go:356] "Generic (PLEG): container finished" podID="8501dc13-6ab5-4a53-84b0-bb02ef46e114" containerID="b6ac7d18f41bca75f45fe34bc884b86828452f482cbecc3e675d5c513755cade" exitCode=0
Jul 17 21:14:49.998985 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:49.998624    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt" event={"ID":"8501dc13-6ab5-4a53-84b0-bb02ef46e114","Type":"ContainerDied","Data":"b6ac7d18f41bca75f45fe34bc884b86828452f482cbecc3e675d5c513755cade"}
Jul 17 21:14:51.003313 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:51.003263    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt" event={"ID":"8501dc13-6ab5-4a53-84b0-bb02ef46e114","Type":"ContainerStarted","Data":"607e14326dcd66126188954bd2eadfd25f01297d626acb12e73b2c69c0496d8e"}
Jul 17 21:14:51.003763 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:51.003419    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:14:51.026198 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:14:51.026149    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt" podStartSLOduration=2.613683977 podStartE2EDuration="7.026137907s" totalImagesPullingTime="3.388987196s" totalInitContainerRuntime="1.023466734s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:14:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:14:51.024679619 +0000 UTC m=+390.988800227" watchObservedRunningTime="2026-07-17 21:14:51.026137907 +0000 UTC m=+390.990258517"
Jul 17 21:15:02.009360 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:02.009327    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b8f6b79df-smxrt"
Jul 17 21:15:11.663701 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.663666    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"]
Jul 17 21:15:11.672889 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.672866    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"
Jul 17 21:15:11.676199 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.676136    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 17 21:15:11.676423 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.676406    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 17 21:15:11.676607 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.676587    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:15:11.676708 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.676621    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 17 21:15:11.676708 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.676695    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cmtnp\""
Jul 17 21:15:11.676810 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.676585    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 17 21:15:11.677958 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.677935    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"]
Jul 17 21:15:11.778364 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.778336    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-772wl\" (UniqueName: \"kubernetes.io/projected/6b7f6c48-59a8-4a21-9002-1ad27a25cef5-kube-api-access-772wl\") pod \"lws-controller-manager-7f487bdccb-fj54j\" (UID: \"6b7f6c48-59a8-4a21-9002-1ad27a25cef5\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"
Jul 17 21:15:11.778486 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.778373    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b7f6c48-59a8-4a21-9002-1ad27a25cef5-metrics-cert\") pod \"lws-controller-manager-7f487bdccb-fj54j\" (UID: \"6b7f6c48-59a8-4a21-9002-1ad27a25cef5\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"
Jul 17 21:15:11.778486 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.778400    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b7f6c48-59a8-4a21-9002-1ad27a25cef5-cert\") pod \"lws-controller-manager-7f487bdccb-fj54j\" (UID: \"6b7f6c48-59a8-4a21-9002-1ad27a25cef5\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"
Jul 17 21:15:11.778486 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.778466    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6b7f6c48-59a8-4a21-9002-1ad27a25cef5-manager-config\") pod \"lws-controller-manager-7f487bdccb-fj54j\" (UID: \"6b7f6c48-59a8-4a21-9002-1ad27a25cef5\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"
Jul 17 21:15:11.879217 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.879190    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b7f6c48-59a8-4a21-9002-1ad27a25cef5-cert\") pod \"lws-controller-manager-7f487bdccb-fj54j\" (UID: \"6b7f6c48-59a8-4a21-9002-1ad27a25cef5\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"
Jul 17 21:15:11.879317 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.879228    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6b7f6c48-59a8-4a21-9002-1ad27a25cef5-manager-config\") pod \"lws-controller-manager-7f487bdccb-fj54j\" (UID: \"6b7f6c48-59a8-4a21-9002-1ad27a25cef5\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"
Jul 17 21:15:11.879317 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.879260    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-772wl\" (UniqueName: \"kubernetes.io/projected/6b7f6c48-59a8-4a21-9002-1ad27a25cef5-kube-api-access-772wl\") pod \"lws-controller-manager-7f487bdccb-fj54j\" (UID: \"6b7f6c48-59a8-4a21-9002-1ad27a25cef5\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"
Jul 17 21:15:11.879388 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.879315    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b7f6c48-59a8-4a21-9002-1ad27a25cef5-metrics-cert\") pod \"lws-controller-manager-7f487bdccb-fj54j\" (UID: \"6b7f6c48-59a8-4a21-9002-1ad27a25cef5\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"
Jul 17 21:15:11.879846 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.879822    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6b7f6c48-59a8-4a21-9002-1ad27a25cef5-manager-config\") pod \"lws-controller-manager-7f487bdccb-fj54j\" (UID: \"6b7f6c48-59a8-4a21-9002-1ad27a25cef5\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"
Jul 17 21:15:11.881630 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.881611    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b7f6c48-59a8-4a21-9002-1ad27a25cef5-metrics-cert\") pod \"lws-controller-manager-7f487bdccb-fj54j\" (UID: \"6b7f6c48-59a8-4a21-9002-1ad27a25cef5\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"
Jul 17 21:15:11.881717 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.881678    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b7f6c48-59a8-4a21-9002-1ad27a25cef5-cert\") pod \"lws-controller-manager-7f487bdccb-fj54j\" (UID: \"6b7f6c48-59a8-4a21-9002-1ad27a25cef5\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"
Jul 17 21:15:11.886925 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.886904    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-772wl\" (UniqueName: \"kubernetes.io/projected/6b7f6c48-59a8-4a21-9002-1ad27a25cef5-kube-api-access-772wl\") pod \"lws-controller-manager-7f487bdccb-fj54j\" (UID: \"6b7f6c48-59a8-4a21-9002-1ad27a25cef5\") " pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"
Jul 17 21:15:11.982525 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:11.982467    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"
Jul 17 21:15:12.102783 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:12.102706    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"]
Jul 17 21:15:12.104842 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:15:12.104812    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7f6c48_59a8_4a21_9002_1ad27a25cef5.slice/crio-0c5d167695c02706195d5e9095920b9fd7f6a824c4c1737e55dd7b755b823fff WatchSource:0}: Error finding container 0c5d167695c02706195d5e9095920b9fd7f6a824c4c1737e55dd7b755b823fff: Status 404 returned error can't find the container with id 0c5d167695c02706195d5e9095920b9fd7f6a824c4c1737e55dd7b755b823fff
Jul 17 21:15:12.876978 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:12.876945    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"]
Jul 17 21:15:12.879875 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:12.879861    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:12.884701 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:12.884667    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 21:15:12.884829 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:12.884770    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 21:15:12.884905 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:12.884883    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 21:15:12.885585 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:12.885567    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-x4mt8\""
Jul 17 21:15:12.902753 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:12.902734    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"]
Jul 17 21:15:12.988245 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:12.988217    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:12.988379 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:12.988255    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:12.988379 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:12.988303    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:12.988379 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:12.988328    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x266p\" (UniqueName: \"kubernetes.io/projected/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-kube-api-access-x266p\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:12.988379 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:12.988354    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:12.988546 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:12.988390    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:12.988546 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:12.988439    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:12.988546 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:12.988495    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:12.988546 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:12.988532    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.081831 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.081809    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j" event={"ID":"6b7f6c48-59a8-4a21-9002-1ad27a25cef5","Type":"ContainerStarted","Data":"0c5d167695c02706195d5e9095920b9fd7f6a824c4c1737e55dd7b755b823fff"}
Jul 17 21:15:13.089038 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.089019    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.089111 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.089069    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.089111 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.089094    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.089208 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.089116    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.089208 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.089138    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.089208 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.089158    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.089208 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.089179    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.089430 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.089326    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x266p\" (UniqueName: \"kubernetes.io/projected/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-kube-api-access-x266p\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.089430 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.089359    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.089584 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.089563    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.089680 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.089660    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.089739 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.089722    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.089802 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.089762    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.089948 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.089928    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.091707 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.091685    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.091786 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.091716    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.099940 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.099919    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.100842 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.100822    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x266p\" (UniqueName: \"kubernetes.io/projected/3ea85e23-ab25-4b26-83f4-cd0b3dfffe06-kube-api-access-x266p\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vlswt\" (UID: \"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.190120 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.190065    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:13.323731 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.323705    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"]
Jul 17 21:15:13.327906 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.327890    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.330513 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.330489    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 17 21:15:13.331045 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.331028    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 17 21:15:13.331135 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.331051    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 17 21:15:13.331135 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.331034    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-d9p27\""
Jul 17 21:15:13.340136 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.340115    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"]
Jul 17 21:15:13.342823 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:15:13.342801    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea85e23_ab25_4b26_83f4_cd0b3dfffe06.slice/crio-4263fbdf4fbb82387797b809f454f949164020b90fd08c0b404bb504d8c7da83 WatchSource:0}: Error finding container 4263fbdf4fbb82387797b809f454f949164020b90fd08c0b404bb504d8c7da83: Status 404 returned error can't find the container with id 4263fbdf4fbb82387797b809f454f949164020b90fd08c0b404bb504d8c7da83
Jul 17 21:15:13.347812 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.347792    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"]
Jul 17 21:15:13.492521 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.492458    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/054e5029-b4c2-465f-ad91-2dd5e116f847-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.492521 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.492488    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/054e5029-b4c2-465f-ad91-2dd5e116f847-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.492521 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.492505    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/054e5029-b4c2-465f-ad91-2dd5e116f847-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.492743 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.492591    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054e5029-b4c2-465f-ad91-2dd5e116f847-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.492743 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.492624    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/054e5029-b4c2-465f-ad91-2dd5e116f847-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.492743 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.492640    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/054e5029-b4c2-465f-ad91-2dd5e116f847-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.492743 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.492682    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r79kn\" (UniqueName: \"kubernetes.io/projected/054e5029-b4c2-465f-ad91-2dd5e116f847-kube-api-access-r79kn\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.593873 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.593838    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054e5029-b4c2-465f-ad91-2dd5e116f847-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.593873 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.593877    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/054e5029-b4c2-465f-ad91-2dd5e116f847-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.594002 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.593896    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/054e5029-b4c2-465f-ad91-2dd5e116f847-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.594002 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.593939    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r79kn\" (UniqueName: \"kubernetes.io/projected/054e5029-b4c2-465f-ad91-2dd5e116f847-kube-api-access-r79kn\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.594002 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.593974    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/054e5029-b4c2-465f-ad91-2dd5e116f847-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.594132 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.594003    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/054e5029-b4c2-465f-ad91-2dd5e116f847-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.594132 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.594023    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/054e5029-b4c2-465f-ad91-2dd5e116f847-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.594788 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.594768    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/054e5029-b4c2-465f-ad91-2dd5e116f847-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.596359 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.596328    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/054e5029-b4c2-465f-ad91-2dd5e116f847-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.596441 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.596391    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/054e5029-b4c2-465f-ad91-2dd5e116f847-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.596483 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.596473    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/054e5029-b4c2-465f-ad91-2dd5e116f847-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.596595 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.596579    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/054e5029-b4c2-465f-ad91-2dd5e116f847-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.605615 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.605595    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/054e5029-b4c2-465f-ad91-2dd5e116f847-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.605926 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.605906    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r79kn\" (UniqueName: \"kubernetes.io/projected/054e5029-b4c2-465f-ad91-2dd5e116f847-kube-api-access-r79kn\") pod \"istiod-openshift-gateway-94bb8fbfd-ghm49\" (UID: \"054e5029-b4c2-465f-ad91-2dd5e116f847\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.636879 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.636860    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:13.760233 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:13.760209    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"]
Jul 17 21:15:13.762197 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:15:13.762170    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054e5029_b4c2_465f_ad91_2dd5e116f847.slice/crio-3b2e675da8d8d1840fa9cee8a6e15260079a4771e14093c56c1aaa3636beed3e WatchSource:0}: Error finding container 3b2e675da8d8d1840fa9cee8a6e15260079a4771e14093c56c1aaa3636beed3e: Status 404 returned error can't find the container with id 3b2e675da8d8d1840fa9cee8a6e15260079a4771e14093c56c1aaa3636beed3e
Jul 17 21:15:14.087743 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:14.087714    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49" event={"ID":"054e5029-b4c2-465f-ad91-2dd5e116f847","Type":"ContainerStarted","Data":"3b2e675da8d8d1840fa9cee8a6e15260079a4771e14093c56c1aaa3636beed3e"}
Jul 17 21:15:14.088858 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:14.088831    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt" event={"ID":"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06","Type":"ContainerStarted","Data":"4263fbdf4fbb82387797b809f454f949164020b90fd08c0b404bb504d8c7da83"}
Jul 17 21:15:17.126699 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:17.126657    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:15:17.127009 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:17.126732    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:15:17.127009 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:17.126761    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:15:18.106439 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:18.106404    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j" event={"ID":"6b7f6c48-59a8-4a21-9002-1ad27a25cef5","Type":"ContainerStarted","Data":"e2a2c8d800983078522c80ea00255ec15b009ed291dabcbf4eddc555259b1d67"}
Jul 17 21:15:18.106646 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:18.106504    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"
Jul 17 21:15:18.108079 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:18.108051    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt" event={"ID":"3ea85e23-ab25-4b26-83f4-cd0b3dfffe06","Type":"ContainerStarted","Data":"5bcdb6bf894d0b09c4a0fea3f8ea3900a8d3a24f7e7499bb81a542afbb9c2d45"}
Jul 17 21:15:18.124367 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:18.124321    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j" podStartSLOduration=2.106350885 podStartE2EDuration="7.124305665s" totalImagesPullingTime="5.01795478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:15:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:15:18.123453371 +0000 UTC m=+418.087573980" watchObservedRunningTime="2026-07-17 21:15:18.124305665 +0000 UTC m=+418.088426273"
Jul 17 21:15:18.149211 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:18.149160    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt" podStartSLOduration=2.367547871 podStartE2EDuration="6.149149155s" totalImagesPullingTime="3.781601284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:15:18.148556019 +0000 UTC m=+418.112676630" watchObservedRunningTime="2026-07-17 21:15:18.149149155 +0000 UTC m=+418.113269768"
Jul 17 21:15:18.191124 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:18.191101    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:18.196147 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:18.196125    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:19.113463 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:19.113422    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:19.114646 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:19.114622    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vlswt"
Jul 17 21:15:19.410205 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:19.410169    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:15:19.410468 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:19.410240    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:15:20.119548 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:20.119503    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49" event={"ID":"054e5029-b4c2-465f-ad91-2dd5e116f847","Type":"ContainerStarted","Data":"b4241c58733b08b94ddc7bc87bec878d57fb1a74f5aa771cf37a75732cb58773"}
Jul 17 21:15:20.119728 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:20.119643    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:20.121633 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:20.121608    2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-ghm49 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 21:15:20.121747 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:20.121665    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49" podUID="054e5029-b4c2-465f-ad91-2dd5e116f847" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 21:15:20.168567 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:20.168515    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49" podStartSLOduration=1.522741466 podStartE2EDuration="7.16850261s" totalImagesPullingTime="5.645761144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:15:20.167853841 +0000 UTC m=+420.131974452" watchObservedRunningTime="2026-07-17 21:15:20.16850261 +0000 UTC m=+420.132623220"
Jul 17 21:15:21.123785 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:21.123755    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-ghm49"
Jul 17 21:15:29.115692 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:15:29.115663    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7f487bdccb-fj54j"
Jul 17 21:19:32.008305 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.008258    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"]
Jul 17 21:19:32.011917 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.011898    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.014395 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.014366    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 17 21:19:32.014508 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.014428    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 17 21:19:32.015585 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.015567    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 17 21:19:32.015693 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.015601    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-q6bg6\""
Jul 17 21:19:32.021879 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.021859    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"]
Jul 17 21:19:32.127974 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.127938    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/933175c9-8925-4726-aec4-6ab6c52e0a5e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hzrq7\" (UID: \"933175c9-8925-4726-aec4-6ab6c52e0a5e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.128151 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.127982    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr62m\" (UniqueName: \"kubernetes.io/projected/933175c9-8925-4726-aec4-6ab6c52e0a5e-kube-api-access-qr62m\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hzrq7\" (UID: \"933175c9-8925-4726-aec4-6ab6c52e0a5e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.128151 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.128026    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/933175c9-8925-4726-aec4-6ab6c52e0a5e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hzrq7\" (UID: \"933175c9-8925-4726-aec4-6ab6c52e0a5e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.128151 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.128084    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933175c9-8925-4726-aec4-6ab6c52e0a5e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hzrq7\" (UID: \"933175c9-8925-4726-aec4-6ab6c52e0a5e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.128314 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.128150    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933175c9-8925-4726-aec4-6ab6c52e0a5e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hzrq7\" (UID: \"933175c9-8925-4726-aec4-6ab6c52e0a5e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.128314 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.128174    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/933175c9-8925-4726-aec4-6ab6c52e0a5e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hzrq7\" (UID: \"933175c9-8925-4726-aec4-6ab6c52e0a5e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.229440 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.229394    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/933175c9-8925-4726-aec4-6ab6c52e0a5e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hzrq7\" (UID: \"933175c9-8925-4726-aec4-6ab6c52e0a5e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.229630 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.229458    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933175c9-8925-4726-aec4-6ab6c52e0a5e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hzrq7\" (UID: \"933175c9-8925-4726-aec4-6ab6c52e0a5e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.229630 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.229535    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933175c9-8925-4726-aec4-6ab6c52e0a5e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hzrq7\" (UID: \"933175c9-8925-4726-aec4-6ab6c52e0a5e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.229630 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.229568    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/933175c9-8925-4726-aec4-6ab6c52e0a5e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hzrq7\" (UID: \"933175c9-8925-4726-aec4-6ab6c52e0a5e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.229630 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.229615    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/933175c9-8925-4726-aec4-6ab6c52e0a5e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hzrq7\" (UID: \"933175c9-8925-4726-aec4-6ab6c52e0a5e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.229843 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.229645    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr62m\" (UniqueName: \"kubernetes.io/projected/933175c9-8925-4726-aec4-6ab6c52e0a5e-kube-api-access-qr62m\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hzrq7\" (UID: \"933175c9-8925-4726-aec4-6ab6c52e0a5e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.229934 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.229912    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/933175c9-8925-4726-aec4-6ab6c52e0a5e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hzrq7\" (UID: \"933175c9-8925-4726-aec4-6ab6c52e0a5e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.229997 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.229978    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/933175c9-8925-4726-aec4-6ab6c52e0a5e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hzrq7\" (UID: \"933175c9-8925-4726-aec4-6ab6c52e0a5e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.230077 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.230052    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/933175c9-8925-4726-aec4-6ab6c52e0a5e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hzrq7\" (UID: \"933175c9-8925-4726-aec4-6ab6c52e0a5e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.231785 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.231767    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/933175c9-8925-4726-aec4-6ab6c52e0a5e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hzrq7\" (UID: \"933175c9-8925-4726-aec4-6ab6c52e0a5e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.231942 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.231925    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/933175c9-8925-4726-aec4-6ab6c52e0a5e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hzrq7\" (UID: \"933175c9-8925-4726-aec4-6ab6c52e0a5e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.238715 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.238694    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr62m\" (UniqueName: \"kubernetes.io/projected/933175c9-8925-4726-aec4-6ab6c52e0a5e-kube-api-access-qr62m\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hzrq7\" (UID: \"933175c9-8925-4726-aec4-6ab6c52e0a5e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.324200 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.324147    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:32.449457 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.449432    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"]
Jul 17 21:19:32.451166 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:19:32.451139    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933175c9_8925_4726_aec4_6ab6c52e0a5e.slice/crio-0d82061c931ca52b78dd28db3c2396cd32cbea97606ff4909d47df16ae9b3aba WatchSource:0}: Error finding container 0d82061c931ca52b78dd28db3c2396cd32cbea97606ff4909d47df16ae9b3aba: Status 404 returned error can't find the container with id 0d82061c931ca52b78dd28db3c2396cd32cbea97606ff4909d47df16ae9b3aba
Jul 17 21:19:32.452452 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:32.452435    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 21:19:33.056828 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:33.056795    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7" event={"ID":"933175c9-8925-4726-aec4-6ab6c52e0a5e","Type":"ContainerStarted","Data":"0d82061c931ca52b78dd28db3c2396cd32cbea97606ff4909d47df16ae9b3aba"}
Jul 17 21:19:34.061966 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:34.061935    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7" event={"ID":"933175c9-8925-4726-aec4-6ab6c52e0a5e","Type":"ContainerStarted","Data":"a11ad6f2af30cd15889c985119702cae75249074ee721c9e8de94896d8afbbab"}
Jul 17 21:19:34.078864 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:34.078821    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7" podStartSLOduration=2.064639076 podStartE2EDuration="3.078803648s" totalImagesPullingTime="1.014164572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:19:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:19:34.078557932 +0000 UTC m=+674.042678543" watchObservedRunningTime="2026-07-17 21:19:34.078803648 +0000 UTC m=+674.042924260"
Jul 17 21:19:42.324430 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:42.324383    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:42.324831 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:42.324471    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:42.336623 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:42.336600    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:43.105447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:43.105416    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hzrq7"
Jul 17 21:19:59.418514 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.418485    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"]
Jul 17 21:19:59.423929 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.423911    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.426254 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.426231    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 17 21:19:59.429133 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.429098    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"]
Jul 17 21:19:59.535031 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.535004    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2aed99-b52e-4278-8c33-87ced547b65a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ckqnw\" (UID: \"aa2aed99-b52e-4278-8c33-87ced547b65a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.535031 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.535036    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2aed99-b52e-4278-8c33-87ced547b65a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ckqnw\" (UID: \"aa2aed99-b52e-4278-8c33-87ced547b65a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.535234 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.535061    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2aed99-b52e-4278-8c33-87ced547b65a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ckqnw\" (UID: \"aa2aed99-b52e-4278-8c33-87ced547b65a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.535234 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.535088    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2aed99-b52e-4278-8c33-87ced547b65a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ckqnw\" (UID: \"aa2aed99-b52e-4278-8c33-87ced547b65a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.535234 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.535160    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wps6\" (UniqueName: \"kubernetes.io/projected/aa2aed99-b52e-4278-8c33-87ced547b65a-kube-api-access-6wps6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ckqnw\" (UID: \"aa2aed99-b52e-4278-8c33-87ced547b65a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.535234 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.535200    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2aed99-b52e-4278-8c33-87ced547b65a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ckqnw\" (UID: \"aa2aed99-b52e-4278-8c33-87ced547b65a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.636375 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.636343    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2aed99-b52e-4278-8c33-87ced547b65a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ckqnw\" (UID: \"aa2aed99-b52e-4278-8c33-87ced547b65a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.636534 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.636380    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2aed99-b52e-4278-8c33-87ced547b65a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ckqnw\" (UID: \"aa2aed99-b52e-4278-8c33-87ced547b65a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.636534 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.636427    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wps6\" (UniqueName: \"kubernetes.io/projected/aa2aed99-b52e-4278-8c33-87ced547b65a-kube-api-access-6wps6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ckqnw\" (UID: \"aa2aed99-b52e-4278-8c33-87ced547b65a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.636534 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.636450    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2aed99-b52e-4278-8c33-87ced547b65a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ckqnw\" (UID: \"aa2aed99-b52e-4278-8c33-87ced547b65a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.636534 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.636483    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2aed99-b52e-4278-8c33-87ced547b65a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ckqnw\" (UID: \"aa2aed99-b52e-4278-8c33-87ced547b65a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.636534 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.636499    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2aed99-b52e-4278-8c33-87ced547b65a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ckqnw\" (UID: \"aa2aed99-b52e-4278-8c33-87ced547b65a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.636824 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.636800    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2aed99-b52e-4278-8c33-87ced547b65a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ckqnw\" (UID: \"aa2aed99-b52e-4278-8c33-87ced547b65a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.636885 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.636813    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2aed99-b52e-4278-8c33-87ced547b65a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ckqnw\" (UID: \"aa2aed99-b52e-4278-8c33-87ced547b65a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.636930 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.636892    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2aed99-b52e-4278-8c33-87ced547b65a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ckqnw\" (UID: \"aa2aed99-b52e-4278-8c33-87ced547b65a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.638672 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.638651    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2aed99-b52e-4278-8c33-87ced547b65a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ckqnw\" (UID: \"aa2aed99-b52e-4278-8c33-87ced547b65a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.638921 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.638903    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2aed99-b52e-4278-8c33-87ced547b65a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ckqnw\" (UID: \"aa2aed99-b52e-4278-8c33-87ced547b65a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.645160 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.645137    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wps6\" (UniqueName: \"kubernetes.io/projected/aa2aed99-b52e-4278-8c33-87ced547b65a-kube-api-access-6wps6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ckqnw\" (UID: \"aa2aed99-b52e-4278-8c33-87ced547b65a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.735174 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.735108    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:19:59.860501 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:19:59.860474    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"]
Jul 17 21:20:00.157816 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:20:00.157786    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw" event={"ID":"aa2aed99-b52e-4278-8c33-87ced547b65a","Type":"ContainerStarted","Data":"a3d197b55714bf1e1298fc96edaf2da0618a6f2977b05b0b9a4c1cd4e7977e93"}
Jul 17 21:20:00.157935 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:20:00.157843    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw" event={"ID":"aa2aed99-b52e-4278-8c33-87ced547b65a","Type":"ContainerStarted","Data":"f80f1ac0aecc8d66bed0a4f66180c44868cce4b5aed1f18ac08dda145f82fc64"}
Jul 17 21:20:00.174869 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:20:00.174696    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw" podStartSLOduration=0.943939418 podStartE2EDuration="1.174677974s" totalImagesPullingTime="230.738556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:20:00.174131926 +0000 UTC m=+700.138252536" watchObservedRunningTime="2026-07-17 21:20:00.174677974 +0000 UTC m=+700.138798586"
Jul 17 21:20:09.735621 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:20:09.735591    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:20:09.736113 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:20:09.735681    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:20:09.747517 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:20:09.747492    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:20:10.206765 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:20:10.206732    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ckqnw"
Jul 17 21:36:03.549968 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.549941    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"]
Jul 17 21:36:03.552900 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.552885    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.561275 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.561257    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-52039111-openshift-default-dockercfg-q6z7q\""
Jul 17 21:36:03.637631 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.637608    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"]
Jul 17 21:36:03.691537 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.691510    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-workload-certs\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.691670 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.691558    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-podinfo\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.691670 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.691608    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpxvg\" (UniqueName: \"kubernetes.io/projected/0fbae19e-40e7-44b6-ac7f-ddf261a96486-kube-api-access-rpxvg\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.691670 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.691666    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-workload-socket\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.691803 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.691763    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istiod-ca-cert\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.691867 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.691843    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-data\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.691931 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.691915    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-token\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.691968 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.691953    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-envoy\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.692028 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.692010    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-credential-socket\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.792853 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.792775    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-envoy\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.792853 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.792822    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-credential-socket\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.793046 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.792943    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-workload-certs\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.793046 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.792989    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-podinfo\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.793046 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.793023    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpxvg\" (UniqueName: \"kubernetes.io/projected/0fbae19e-40e7-44b6-ac7f-ddf261a96486-kube-api-access-rpxvg\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.793204 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.793060    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-workload-socket\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.793204 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.793090    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istiod-ca-cert\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.793204 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.793136    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-data\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.793204 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.793189    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-token\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.793493 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.793333    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-credential-socket\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.793493 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.793472    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-workload-socket\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.793606 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.793521    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-workload-certs\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.793661 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.793624    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-data\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.793818 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.793800    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istiod-ca-cert\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.795026 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.795000    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-envoy\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.795431 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.795415    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-podinfo\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.801075 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.801049    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-token\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.801215 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.801195    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpxvg\" (UniqueName: \"kubernetes.io/projected/0fbae19e-40e7-44b6-ac7f-ddf261a96486-kube-api-access-rpxvg\") pod \"e2e-ait-52039111-openshift-default-68dbc44c-zqzg4\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") " pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.861860 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.861834    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:03.988721 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.988701    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"]
Jul 17 21:36:03.990764 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:36:03.990736    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbae19e_40e7_44b6_ac7f_ddf261a96486.slice/crio-57336e4ed4d29301f32321070e8af64c49a1bd77217600b6196efc037a67455d WatchSource:0}: Error finding container 57336e4ed4d29301f32321070e8af64c49a1bd77217600b6196efc037a67455d: Status 404 returned error can't find the container with id 57336e4ed4d29301f32321070e8af64c49a1bd77217600b6196efc037a67455d
Jul 17 21:36:03.992926 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.992894    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:36:03.993021 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.992977    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:36:03.993081 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:03.993023    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:36:04.845259 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:04.845224    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4" event={"ID":"0fbae19e-40e7-44b6-ac7f-ddf261a96486","Type":"ContainerStarted","Data":"efcc82d4610f38503cd01e346f776ffc7ae62c414c4056bcb1dafca402dedbff"}
Jul 17 21:36:04.845673 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:04.845276    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4" event={"ID":"0fbae19e-40e7-44b6-ac7f-ddf261a96486","Type":"ContainerStarted","Data":"57336e4ed4d29301f32321070e8af64c49a1bd77217600b6196efc037a67455d"}
Jul 17 21:36:04.861968 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:04.861940    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:04.867190 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:04.867140    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4" podStartSLOduration=1.867122772 podStartE2EDuration="1.867122772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:36:04.866732032 +0000 UTC m=+1664.830852637" watchObservedRunningTime="2026-07-17 21:36:04.867122772 +0000 UTC m=+1664.831243383"
Jul 17 21:36:05.867210 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:05.867182    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:06.853434 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:06.853403    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:06.854526 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:06.854502    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:16.900178 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:16.900107    2573 generic.go:356] "Generic (PLEG): container finished" podID="0fbae19e-40e7-44b6-ac7f-ddf261a96486" containerID="efcc82d4610f38503cd01e346f776ffc7ae62c414c4056bcb1dafca402dedbff" exitCode=0
Jul 17 21:36:16.900580 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:16.900176    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4" event={"ID":"0fbae19e-40e7-44b6-ac7f-ddf261a96486","Type":"ContainerDied","Data":"efcc82d4610f38503cd01e346f776ffc7ae62c414c4056bcb1dafca402dedbff"}
Jul 17 21:36:16.900640 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:16.900601    2573 scope.go:117] "RemoveContainer" containerID="efcc82d4610f38503cd01e346f776ffc7ae62c414c4056bcb1dafca402dedbff"
Jul 17 21:36:16.901590 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:16.901559    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:36:16.901706 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:16.901642    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:36:16.901706 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:16.901693    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:36:17.905683 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:17.905651    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4" event={"ID":"0fbae19e-40e7-44b6-ac7f-ddf261a96486","Type":"ContainerStarted","Data":"36e42fa88a6ef6be98f351740e121aedcc719beadbe78db83e59a7141498fde0"}
Jul 17 21:36:18.862133 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:18.862092    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:18.863193 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:18.863169    2573 patch_prober.go:28] interesting pod/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Jul 17 21:36:18.863304 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:18.863217    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4" podUID="0fbae19e-40e7-44b6-ac7f-ddf261a96486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Jul 17 21:36:19.862566 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:19.862534    2573 patch_prober.go:28] interesting pod/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Jul 17 21:36:19.862913 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:19.862589    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4" podUID="0fbae19e-40e7-44b6-ac7f-ddf261a96486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Jul 17 21:36:20.863198 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:20.863153    2573 patch_prober.go:28] interesting pod/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body=
Jul 17 21:36:20.863585 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:20.863219    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4" podUID="0fbae19e-40e7-44b6-ac7f-ddf261a96486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused"
Jul 17 21:36:21.853996 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:21.853966    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:21.866273 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:21.866249    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:36:21.924066 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:36:21.924045    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:39:09.801578 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:09.801541    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"]
Jul 17 21:39:09.802048 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:09.801732    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4" podUID="0fbae19e-40e7-44b6-ac7f-ddf261a96486" containerName="istio-proxy" containerID="cri-o://36e42fa88a6ef6be98f351740e121aedcc719beadbe78db83e59a7141498fde0" gracePeriod=30
Jul 17 21:39:15.047000 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.046976    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:39:15.202925 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.202897    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-envoy\") pod \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") "
Jul 17 21:39:15.203061 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.202937    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-podinfo\") pod \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") "
Jul 17 21:39:15.203061 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.202968    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istiod-ca-cert\") pod \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") "
Jul 17 21:39:15.203061 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.203008    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-workload-socket\") pod \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") "
Jul 17 21:39:15.203061 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.203023    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-credential-socket\") pod \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") "
Jul 17 21:39:15.203061 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.203064    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-workload-certs\") pod \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") "
Jul 17 21:39:15.203366 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.203090    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-data\") pod \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") "
Jul 17 21:39:15.203366 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.203113    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-token\") pod \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") "
Jul 17 21:39:15.203366 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.203133    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpxvg\" (UniqueName: \"kubernetes.io/projected/0fbae19e-40e7-44b6-ac7f-ddf261a96486-kube-api-access-rpxvg\") pod \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\" (UID: \"0fbae19e-40e7-44b6-ac7f-ddf261a96486\") "
Jul 17 21:39:15.203366 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.203312    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0fbae19e-40e7-44b6-ac7f-ddf261a96486" (UID: "0fbae19e-40e7-44b6-ac7f-ddf261a96486"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:39:15.203579 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.203506    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0fbae19e-40e7-44b6-ac7f-ddf261a96486" (UID: "0fbae19e-40e7-44b6-ac7f-ddf261a96486"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:39:15.203579 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.203563    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0fbae19e-40e7-44b6-ac7f-ddf261a96486" (UID: "0fbae19e-40e7-44b6-ac7f-ddf261a96486"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:39:15.203686 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.203592    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0fbae19e-40e7-44b6-ac7f-ddf261a96486" (UID: "0fbae19e-40e7-44b6-ac7f-ddf261a96486"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:39:15.204278 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.204239    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-data" (OuterVolumeSpecName: "istio-data") pod "0fbae19e-40e7-44b6-ac7f-ddf261a96486" (UID: "0fbae19e-40e7-44b6-ac7f-ddf261a96486"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:39:15.205198 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.205172    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0fbae19e-40e7-44b6-ac7f-ddf261a96486" (UID: "0fbae19e-40e7-44b6-ac7f-ddf261a96486"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 21:39:15.205301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.205213    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0fbae19e-40e7-44b6-ac7f-ddf261a96486" (UID: "0fbae19e-40e7-44b6-ac7f-ddf261a96486"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:39:15.205301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.205231    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-token" (OuterVolumeSpecName: "istio-token") pod "0fbae19e-40e7-44b6-ac7f-ddf261a96486" (UID: "0fbae19e-40e7-44b6-ac7f-ddf261a96486"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:39:15.205398 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.205320    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbae19e-40e7-44b6-ac7f-ddf261a96486-kube-api-access-rpxvg" (OuterVolumeSpecName: "kube-api-access-rpxvg") pod "0fbae19e-40e7-44b6-ac7f-ddf261a96486" (UID: "0fbae19e-40e7-44b6-ac7f-ddf261a96486"). InnerVolumeSpecName "kube-api-access-rpxvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:39:15.304577 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.304544    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:39:15.304577 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.304570    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:39:15.304577 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.304579    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:39:15.304868 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.304588    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:39:15.304868 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.304596    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:39:15.304868 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.304604    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpxvg\" (UniqueName: \"kubernetes.io/projected/0fbae19e-40e7-44b6-ac7f-ddf261a96486-kube-api-access-rpxvg\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:39:15.304868 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.304611    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:39:15.304868 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.304619    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:39:15.304868 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.304627    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbae19e-40e7-44b6-ac7f-ddf261a96486-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:39:15.608502 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.608470    2573 generic.go:356] "Generic (PLEG): container finished" podID="0fbae19e-40e7-44b6-ac7f-ddf261a96486" containerID="36e42fa88a6ef6be98f351740e121aedcc719beadbe78db83e59a7141498fde0" exitCode=0
Jul 17 21:39:15.608688 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.608520    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"
Jul 17 21:39:15.608688 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.608560    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4" event={"ID":"0fbae19e-40e7-44b6-ac7f-ddf261a96486","Type":"ContainerDied","Data":"36e42fa88a6ef6be98f351740e121aedcc719beadbe78db83e59a7141498fde0"}
Jul 17 21:39:15.608688 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.608618    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4" event={"ID":"0fbae19e-40e7-44b6-ac7f-ddf261a96486","Type":"ContainerDied","Data":"57336e4ed4d29301f32321070e8af64c49a1bd77217600b6196efc037a67455d"}
Jul 17 21:39:15.608688 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.608648    2573 scope.go:117] "RemoveContainer" containerID="36e42fa88a6ef6be98f351740e121aedcc719beadbe78db83e59a7141498fde0"
Jul 17 21:39:15.617222 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.617205    2573 scope.go:117] "RemoveContainer" containerID="efcc82d4610f38503cd01e346f776ffc7ae62c414c4056bcb1dafca402dedbff"
Jul 17 21:39:15.624214 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.624197    2573 scope.go:117] "RemoveContainer" containerID="36e42fa88a6ef6be98f351740e121aedcc719beadbe78db83e59a7141498fde0"
Jul 17 21:39:15.624459 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:39:15.624439    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36e42fa88a6ef6be98f351740e121aedcc719beadbe78db83e59a7141498fde0\": container with ID starting with 36e42fa88a6ef6be98f351740e121aedcc719beadbe78db83e59a7141498fde0 not found: ID does not exist" containerID="36e42fa88a6ef6be98f351740e121aedcc719beadbe78db83e59a7141498fde0"
Jul 17 21:39:15.624521 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.624465    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e42fa88a6ef6be98f351740e121aedcc719beadbe78db83e59a7141498fde0"} err="failed to get container status \"36e42fa88a6ef6be98f351740e121aedcc719beadbe78db83e59a7141498fde0\": rpc error: code = NotFound desc = could not find container \"36e42fa88a6ef6be98f351740e121aedcc719beadbe78db83e59a7141498fde0\": container with ID starting with 36e42fa88a6ef6be98f351740e121aedcc719beadbe78db83e59a7141498fde0 not found: ID does not exist"
Jul 17 21:39:15.624521 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.624481    2573 scope.go:117] "RemoveContainer" containerID="efcc82d4610f38503cd01e346f776ffc7ae62c414c4056bcb1dafca402dedbff"
Jul 17 21:39:15.624719 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:39:15.624695    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efcc82d4610f38503cd01e346f776ffc7ae62c414c4056bcb1dafca402dedbff\": container with ID starting with efcc82d4610f38503cd01e346f776ffc7ae62c414c4056bcb1dafca402dedbff not found: ID does not exist" containerID="efcc82d4610f38503cd01e346f776ffc7ae62c414c4056bcb1dafca402dedbff"
Jul 17 21:39:15.624761 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.624728    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efcc82d4610f38503cd01e346f776ffc7ae62c414c4056bcb1dafca402dedbff"} err="failed to get container status \"efcc82d4610f38503cd01e346f776ffc7ae62c414c4056bcb1dafca402dedbff\": rpc error: code = NotFound desc = could not find container \"efcc82d4610f38503cd01e346f776ffc7ae62c414c4056bcb1dafca402dedbff\": container with ID starting with efcc82d4610f38503cd01e346f776ffc7ae62c414c4056bcb1dafca402dedbff not found: ID does not exist"
Jul 17 21:39:15.649849 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.649817    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"]
Jul 17 21:39:15.660737 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:15.660715    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-52039111-openshift-default-68dbc44c-zqzg4"]
Jul 17 21:39:16.529352 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:16.529321    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fbae19e-40e7-44b6-ac7f-ddf261a96486" path="/var/lib/kubelet/pods/0fbae19e-40e7-44b6-ac7f-ddf261a96486/volumes"
Jul 17 21:39:22.885336 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:22.882089    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"]
Jul 17 21:39:22.885336 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:22.882693    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fbae19e-40e7-44b6-ac7f-ddf261a96486" containerName="istio-proxy"
Jul 17 21:39:22.885336 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:22.882714    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fbae19e-40e7-44b6-ac7f-ddf261a96486" containerName="istio-proxy"
Jul 17 21:39:22.885336 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:22.882863    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fbae19e-40e7-44b6-ac7f-ddf261a96486" containerName="istio-proxy"
Jul 17 21:39:22.885336 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:22.882881    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fbae19e-40e7-44b6-ac7f-ddf261a96486" containerName="istio-proxy"
Jul 17 21:39:22.886694 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:22.886670    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:22.889123 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:22.889100    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-cc1422e3-gw-openshift-default-dockercfg-fc5wd\""
Jul 17 21:39:22.897322 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:22.897299    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"]
Jul 17 21:39:23.062601 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.062564    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66bb5c90-be1e-4a11-bff5-94c166add057-istio-token\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.062601 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.062603    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bb5c90-be1e-4a11-bff5-94c166add057-istio-podinfo\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.062828 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.062628    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-istio-envoy\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.062828 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.062725    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-workload-certs\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.062828 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.062758    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-workload-socket\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.062828 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.062781    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bb5c90-be1e-4a11-bff5-94c166add057-istiod-ca-cert\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.063018 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.062832    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-istio-data\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.063018 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.062883    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-credential-socket\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.063018 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.062901    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5nvj\" (UniqueName: \"kubernetes.io/projected/66bb5c90-be1e-4a11-bff5-94c166add057-kube-api-access-w5nvj\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.163459 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.163396    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66bb5c90-be1e-4a11-bff5-94c166add057-istio-token\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.163459 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.163424    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bb5c90-be1e-4a11-bff5-94c166add057-istio-podinfo\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.163459 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.163448    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-istio-envoy\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.163764 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.163470    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-workload-certs\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.163764 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.163488    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-workload-socket\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.163764 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.163519    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bb5c90-be1e-4a11-bff5-94c166add057-istiod-ca-cert\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.163764 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.163565    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-istio-data\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.163764 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.163610    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-credential-socket\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.163764 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.163721    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5nvj\" (UniqueName: \"kubernetes.io/projected/66bb5c90-be1e-4a11-bff5-94c166add057-kube-api-access-w5nvj\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.164094 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.163925    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-workload-socket\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.164094 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.164022    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-credential-socket\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.164094 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.164080    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-istio-data\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.164254 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.164151    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-workload-certs\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.164392 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.164371    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bb5c90-be1e-4a11-bff5-94c166add057-istiod-ca-cert\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.165710 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.165690    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-istio-envoy\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.165935 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.165918    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bb5c90-be1e-4a11-bff5-94c166add057-istio-podinfo\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.175994 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.175971    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5nvj\" (UniqueName: \"kubernetes.io/projected/66bb5c90-be1e-4a11-bff5-94c166add057-kube-api-access-w5nvj\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.176070 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.176025    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66bb5c90-be1e-4a11-bff5-94c166add057-istio-token\") pod \"e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") " pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.198014 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.197992    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:23.328573 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.328544    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"]
Jul 17 21:39:23.331110 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:39:23.331084    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66bb5c90_be1e_4a11_bff5_94c166add057.slice/crio-c94205e45c02b28693baa2c31a0e883492c804e3c58006c5c10e7f4e58b1aa9f WatchSource:0}: Error finding container c94205e45c02b28693baa2c31a0e883492c804e3c58006c5c10e7f4e58b1aa9f: Status 404 returned error can't find the container with id c94205e45c02b28693baa2c31a0e883492c804e3c58006c5c10e7f4e58b1aa9f
Jul 17 21:39:23.333199 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.333169    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:39:23.333270 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.333232    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:39:23.333270 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.333265    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:39:23.641475 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.641440    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff" event={"ID":"66bb5c90-be1e-4a11-bff5-94c166add057","Type":"ContainerStarted","Data":"0fd4018f3d7a801833670b93ea62a0bf2bb7bcaf9a1089c285d6d9245ae8dd79"}
Jul 17 21:39:23.641654 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.641487    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff" event={"ID":"66bb5c90-be1e-4a11-bff5-94c166add057","Type":"ContainerStarted","Data":"c94205e45c02b28693baa2c31a0e883492c804e3c58006c5c10e7f4e58b1aa9f"}
Jul 17 21:39:23.664686 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:23.664636    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff" podStartSLOduration=1.664612481 podStartE2EDuration="1.664612481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:39:23.663234544 +0000 UTC m=+1863.627355148" watchObservedRunningTime="2026-07-17 21:39:23.664612481 +0000 UTC m=+1863.628733158"
Jul 17 21:39:24.198730 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:24.198697    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:24.203346 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:24.203323    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:24.645981 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:24.645950    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:24.646958 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:24.646930    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:39.647901 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:39.647869    2573 patch_prober.go:28] interesting pod/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Jul 17 21:39:39.648303 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:39.647937    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff" podUID="66bb5c90-be1e-4a11-bff5-94c166add057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Jul 17 21:39:39.708620 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:39.708593    2573 generic.go:356] "Generic (PLEG): container finished" podID="66bb5c90-be1e-4a11-bff5-94c166add057" containerID="0fd4018f3d7a801833670b93ea62a0bf2bb7bcaf9a1089c285d6d9245ae8dd79" exitCode=0
Jul 17 21:39:39.708746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:39.708669    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff" event={"ID":"66bb5c90-be1e-4a11-bff5-94c166add057","Type":"ContainerDied","Data":"0fd4018f3d7a801833670b93ea62a0bf2bb7bcaf9a1089c285d6d9245ae8dd79"}
Jul 17 21:39:39.709130 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:39.709114    2573 scope.go:117] "RemoveContainer" containerID="0fd4018f3d7a801833670b93ea62a0bf2bb7bcaf9a1089c285d6d9245ae8dd79"
Jul 17 21:39:39.710246 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:39.710215    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:39:39.710334 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:39.710310    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:39:39.710378 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:39.710345    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:39:40.713815 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:40.713780    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff" event={"ID":"66bb5c90-be1e-4a11-bff5-94c166add057","Type":"ContainerStarted","Data":"cf39eb758e2b5f3a1e8d0eb8f56fa7d7f0638e6b4eff1dfe7034ec093d466408"}
Jul 17 21:39:41.198736 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:41.198698    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:41.199000 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:41.198974    2573 patch_prober.go:28] interesting pod/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Jul 17 21:39:41.199053 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:41.199032    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff" podUID="66bb5c90-be1e-4a11-bff5-94c166add057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Jul 17 21:39:42.199343 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:42.199279    2573 patch_prober.go:28] interesting pod/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Jul 17 21:39:42.199746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:42.199363    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff" podUID="66bb5c90-be1e-4a11-bff5-94c166add057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Jul 17 21:39:43.199273 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:43.199189    2573 patch_prober.go:28] interesting pod/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Jul 17 21:39:43.199273 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:43.199257    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff" podUID="66bb5c90-be1e-4a11-bff5-94c166add057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Jul 17 21:39:44.199408 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:44.199374    2573 patch_prober.go:28] interesting pod/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Jul 17 21:39:44.199588 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:44.199456    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff" podUID="66bb5c90-be1e-4a11-bff5-94c166add057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Jul 17 21:39:45.202810 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:45.202776    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:45.203320 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:45.203294    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:39:45.203892 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:39:45.203870    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:40:28.794399 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:28.794368    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"]
Jul 17 21:40:28.794888 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:28.794574    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff" podUID="66bb5c90-be1e-4a11-bff5-94c166add057" containerName="istio-proxy" containerID="cri-o://cf39eb758e2b5f3a1e8d0eb8f56fa7d7f0638e6b4eff1dfe7034ec093d466408" gracePeriod=30
Jul 17 21:40:30.205516 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:30.205472    2573 patch_prober.go:28] interesting pod/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Jul 17 21:40:30.205981 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:30.205534    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff" podUID="66bb5c90-be1e-4a11-bff5-94c166add057" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 21:40:33.926558 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:33.926516    2573 generic.go:356] "Generic (PLEG): container finished" podID="66bb5c90-be1e-4a11-bff5-94c166add057" containerID="cf39eb758e2b5f3a1e8d0eb8f56fa7d7f0638e6b4eff1dfe7034ec093d466408" exitCode=0
Jul 17 21:40:33.926987 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:33.926584    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff" event={"ID":"66bb5c90-be1e-4a11-bff5-94c166add057","Type":"ContainerDied","Data":"cf39eb758e2b5f3a1e8d0eb8f56fa7d7f0638e6b4eff1dfe7034ec093d466408"}
Jul 17 21:40:33.926987 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:33.926653    2573 scope.go:117] "RemoveContainer" containerID="0fd4018f3d7a801833670b93ea62a0bf2bb7bcaf9a1089c285d6d9245ae8dd79"
Jul 17 21:40:34.037617 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.037594    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:40:34.209696 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.209669    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66bb5c90-be1e-4a11-bff5-94c166add057-istio-token\") pod \"66bb5c90-be1e-4a11-bff5-94c166add057\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") "
Jul 17 21:40:34.209963 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.209728    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bb5c90-be1e-4a11-bff5-94c166add057-istio-podinfo\") pod \"66bb5c90-be1e-4a11-bff5-94c166add057\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") "
Jul 17 21:40:34.209963 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.209757    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-istio-data\") pod \"66bb5c90-be1e-4a11-bff5-94c166add057\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") "
Jul 17 21:40:34.209963 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.209795    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-credential-socket\") pod \"66bb5c90-be1e-4a11-bff5-94c166add057\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") "
Jul 17 21:40:34.209963 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.209824    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5nvj\" (UniqueName: \"kubernetes.io/projected/66bb5c90-be1e-4a11-bff5-94c166add057-kube-api-access-w5nvj\") pod \"66bb5c90-be1e-4a11-bff5-94c166add057\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") "
Jul 17 21:40:34.209963 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.209858    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bb5c90-be1e-4a11-bff5-94c166add057-istiod-ca-cert\") pod \"66bb5c90-be1e-4a11-bff5-94c166add057\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") "
Jul 17 21:40:34.209963 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.209894    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-workload-certs\") pod \"66bb5c90-be1e-4a11-bff5-94c166add057\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") "
Jul 17 21:40:34.209963 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.209929    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-istio-envoy\") pod \"66bb5c90-be1e-4a11-bff5-94c166add057\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") "
Jul 17 21:40:34.209963 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.209962    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-workload-socket\") pod \"66bb5c90-be1e-4a11-bff5-94c166add057\" (UID: \"66bb5c90-be1e-4a11-bff5-94c166add057\") "
Jul 17 21:40:34.210391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.210092    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "66bb5c90-be1e-4a11-bff5-94c166add057" (UID: "66bb5c90-be1e-4a11-bff5-94c166add057"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:40:34.210391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.210243    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:40:34.210391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.210255    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "66bb5c90-be1e-4a11-bff5-94c166add057" (UID: "66bb5c90-be1e-4a11-bff5-94c166add057"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:40:34.210391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.210366    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "66bb5c90-be1e-4a11-bff5-94c166add057" (UID: "66bb5c90-be1e-4a11-bff5-94c166add057"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:40:34.210391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.210265    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66bb5c90-be1e-4a11-bff5-94c166add057-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "66bb5c90-be1e-4a11-bff5-94c166add057" (UID: "66bb5c90-be1e-4a11-bff5-94c166add057"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:40:34.210998 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.210971    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-istio-data" (OuterVolumeSpecName: "istio-data") pod "66bb5c90-be1e-4a11-bff5-94c166add057" (UID: "66bb5c90-be1e-4a11-bff5-94c166add057"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:40:34.211971 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.211944    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66bb5c90-be1e-4a11-bff5-94c166add057-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "66bb5c90-be1e-4a11-bff5-94c166add057" (UID: "66bb5c90-be1e-4a11-bff5-94c166add057"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 21:40:34.212061 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.211944    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66bb5c90-be1e-4a11-bff5-94c166add057-istio-token" (OuterVolumeSpecName: "istio-token") pod "66bb5c90-be1e-4a11-bff5-94c166add057" (UID: "66bb5c90-be1e-4a11-bff5-94c166add057"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:40:34.212061 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.212019    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66bb5c90-be1e-4a11-bff5-94c166add057-kube-api-access-w5nvj" (OuterVolumeSpecName: "kube-api-access-w5nvj") pod "66bb5c90-be1e-4a11-bff5-94c166add057" (UID: "66bb5c90-be1e-4a11-bff5-94c166add057"). InnerVolumeSpecName "kube-api-access-w5nvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:40:34.212168 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.212149    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "66bb5c90-be1e-4a11-bff5-94c166add057" (UID: "66bb5c90-be1e-4a11-bff5-94c166add057"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:40:34.310616 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.310572    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5nvj\" (UniqueName: \"kubernetes.io/projected/66bb5c90-be1e-4a11-bff5-94c166add057-kube-api-access-w5nvj\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:40:34.310616 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.310611    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66bb5c90-be1e-4a11-bff5-94c166add057-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:40:34.310616 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.310626    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:40:34.310814 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.310638    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:40:34.310814 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.310649    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:40:34.310814 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.310660    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66bb5c90-be1e-4a11-bff5-94c166add057-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:40:34.310814 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.310672    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66bb5c90-be1e-4a11-bff5-94c166add057-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:40:34.310814 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.310682    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66bb5c90-be1e-4a11-bff5-94c166add057-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:40:34.931600 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.931568    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff" event={"ID":"66bb5c90-be1e-4a11-bff5-94c166add057","Type":"ContainerDied","Data":"c94205e45c02b28693baa2c31a0e883492c804e3c58006c5c10e7f4e58b1aa9f"}
Jul 17 21:40:34.932012 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.931622    2573 scope.go:117] "RemoveContainer" containerID="cf39eb758e2b5f3a1e8d0eb8f56fa7d7f0638e6b4eff1dfe7034ec093d466408"
Jul 17 21:40:34.932012 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.931584    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"
Jul 17 21:40:34.949048 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.949027    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"]
Jul 17 21:40:34.952468 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:34.952446    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-cc1422e3-gw-openshift-default-fcffb9f59-h8vff"]
Jul 17 21:40:36.534437 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:36.532446    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66bb5c90-be1e-4a11-bff5-94c166add057" path="/var/lib/kubelet/pods/66bb5c90-be1e-4a11-bff5-94c166add057/volumes"
Jul 17 21:40:40.652542 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.652513    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"]
Jul 17 21:40:40.652954 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.652785    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fbae19e-40e7-44b6-ac7f-ddf261a96486" containerName="istio-proxy"
Jul 17 21:40:40.652954 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.652795    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fbae19e-40e7-44b6-ac7f-ddf261a96486" containerName="istio-proxy"
Jul 17 21:40:40.652954 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.652802    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66bb5c90-be1e-4a11-bff5-94c166add057" containerName="istio-proxy"
Jul 17 21:40:40.652954 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.652807    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="66bb5c90-be1e-4a11-bff5-94c166add057" containerName="istio-proxy"
Jul 17 21:40:40.652954 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.652817    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66bb5c90-be1e-4a11-bff5-94c166add057" containerName="istio-proxy"
Jul 17 21:40:40.652954 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.652822    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="66bb5c90-be1e-4a11-bff5-94c166add057" containerName="istio-proxy"
Jul 17 21:40:40.652954 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.652864    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="66bb5c90-be1e-4a11-bff5-94c166add057" containerName="istio-proxy"
Jul 17 21:40:40.652954 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.652872    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="66bb5c90-be1e-4a11-bff5-94c166add057" containerName="istio-proxy"
Jul 17 21:40:40.655433 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.655413    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.658187 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.658167    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-db4eb08b-openshift-default-dockercfg-8dskk\""
Jul 17 21:40:40.671267 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.671246    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"]
Jul 17 21:40:40.757057 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.757036    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istiod-ca-cert\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.757158 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.757090    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-podinfo\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.757158 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.757115    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-workload-socket\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.757241 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.757164    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-workload-certs\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.757241 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.757191    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-envoy\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.757241 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.757214    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-token\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.757241 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.757236    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-credential-socket\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.757410 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.757278    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgfnt\" (UniqueName: \"kubernetes.io/projected/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-kube-api-access-mgfnt\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.757410 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.757325    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-data\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.858056 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.858010    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-workload-certs\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.858056 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.858055    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-envoy\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.858310 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.858075    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-token\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.858310 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.858098    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-credential-socket\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.858310 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.858120    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgfnt\" (UniqueName: \"kubernetes.io/projected/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-kube-api-access-mgfnt\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.858310 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.858247    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-data\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.858516 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.858344    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istiod-ca-cert\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.858516 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.858427    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-podinfo\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.858516 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.858468    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-workload-socket\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.858516 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.858483    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-workload-certs\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.858718 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.858583    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-credential-socket\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.858718 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.858617    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-data\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.858826 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.858793    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-workload-socket\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.859099 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.859073    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istiod-ca-cert\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.860405 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.860377    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-envoy\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.860746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.860729    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-podinfo\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.866403 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.866384    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-token\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.866616 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.866593    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgfnt\" (UniqueName: \"kubernetes.io/projected/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-kube-api-access-mgfnt\") pod \"e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") " pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:40.967300 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:40.967226    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:41.093535 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:41.093510    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"]
Jul 17 21:40:41.095267 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:40:41.095237    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d4ad95_76d4_4bbf_805a_dce69fbf77b9.slice/crio-3017428cd0395d835d8e73dcefc318382e4ba7915caf59cab72b1847826da3f6 WatchSource:0}: Error finding container 3017428cd0395d835d8e73dcefc318382e4ba7915caf59cab72b1847826da3f6: Status 404 returned error can't find the container with id 3017428cd0395d835d8e73dcefc318382e4ba7915caf59cab72b1847826da3f6
Jul 17 21:40:41.097318 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:41.097263    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:40:41.097403 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:41.097353    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:40:41.097403 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:41.097384    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:40:41.962530 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:41.962490    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv" event={"ID":"86d4ad95-76d4-4bbf-805a-dce69fbf77b9","Type":"ContainerStarted","Data":"671f83d56d012b3caded4ab3c1dc107ff7a24c78c4648523b9e3d93b0cecbffe"}
Jul 17 21:40:41.962950 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:41.962555    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv" event={"ID":"86d4ad95-76d4-4bbf-805a-dce69fbf77b9","Type":"ContainerStarted","Data":"3017428cd0395d835d8e73dcefc318382e4ba7915caf59cab72b1847826da3f6"}
Jul 17 21:40:41.967527 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:41.967490    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:41.985571 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:41.985529    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv" podStartSLOduration=1.985511803 podStartE2EDuration="1.985511803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:40:41.984725012 +0000 UTC m=+1941.948845624" watchObservedRunningTime="2026-07-17 21:40:41.985511803 +0000 UTC m=+1941.949632413"
Jul 17 21:40:42.972059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:42.972034    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:43.971364 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:43.971331    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:43.972267 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:43.972243    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:53.010480 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:53.010448    2573 generic.go:356] "Generic (PLEG): container finished" podID="86d4ad95-76d4-4bbf-805a-dce69fbf77b9" containerID="671f83d56d012b3caded4ab3c1dc107ff7a24c78c4648523b9e3d93b0cecbffe" exitCode=0
Jul 17 21:40:53.010888 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:53.010527    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv" event={"ID":"86d4ad95-76d4-4bbf-805a-dce69fbf77b9","Type":"ContainerDied","Data":"671f83d56d012b3caded4ab3c1dc107ff7a24c78c4648523b9e3d93b0cecbffe"}
Jul 17 21:40:53.010951 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:53.010932    2573 scope.go:117] "RemoveContainer" containerID="671f83d56d012b3caded4ab3c1dc107ff7a24c78c4648523b9e3d93b0cecbffe"
Jul 17 21:40:53.012021 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:53.011992    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:40:53.012103 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:53.012059    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:40:53.012103 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:53.012092    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:40:54.016704 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:54.016671    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv" event={"ID":"86d4ad95-76d4-4bbf-805a-dce69fbf77b9","Type":"ContainerStarted","Data":"7bcc8fb641c46b6dd0445d9467dcc36257e8dd703451d7d45b4a937250636b53"}
Jul 17 21:40:54.968186 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:54.968157    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:54.969166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:54.969140    2573 patch_prober.go:28] interesting pod/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Jul 17 21:40:54.969279 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:54.969244    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv" podUID="86d4ad95-76d4-4bbf-805a-dce69fbf77b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Jul 17 21:40:55.968479 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:55.968443    2573 patch_prober.go:28] interesting pod/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Jul 17 21:40:55.968877 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:55.968499    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv" podUID="86d4ad95-76d4-4bbf-805a-dce69fbf77b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Jul 17 21:40:56.968519 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:56.968479    2573 patch_prober.go:28] interesting pod/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Jul 17 21:40:56.968998 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:56.968550    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv" podUID="86d4ad95-76d4-4bbf-805a-dce69fbf77b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Jul 17 21:40:57.967574 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:57.967539    2573 patch_prober.go:28] interesting pod/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body=
Jul 17 21:40:57.967741 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:57.967598    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv" podUID="86d4ad95-76d4-4bbf-805a-dce69fbf77b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused"
Jul 17 21:40:58.971397 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:58.971362    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:58.971858 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:58.971760    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:40:59.041197 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:40:59.041170    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:45:17.130232 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:17.130198    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn"]
Jul 17 21:45:17.132185 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:17.132170    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn"
Jul 17 21:45:17.134857 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:17.134832    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:45:17.134974 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:17.134863    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:45:17.141204 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:17.141180    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn"]
Jul 17 21:45:17.229365 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:17.229337    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7881f5f5-2354-4338-8a4d-26db0728a188-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn\" (UID: \"7881f5f5-2354-4338-8a4d-26db0728a188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn"
Jul 17 21:45:17.330586 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:17.330551    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7881f5f5-2354-4338-8a4d-26db0728a188-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn\" (UID: \"7881f5f5-2354-4338-8a4d-26db0728a188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn"
Jul 17 21:45:17.331149 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:17.331129    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7881f5f5-2354-4338-8a4d-26db0728a188-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn\" (UID: \"7881f5f5-2354-4338-8a4d-26db0728a188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn"
Jul 17 21:45:17.443664 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:17.443608    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn"
Jul 17 21:45:17.569352 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:17.569313    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn"]
Jul 17 21:45:17.572278 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:45:17.572255    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7881f5f5_2354_4338_8a4d_26db0728a188.slice/crio-cb542145783b06fffe947c95559550c6f1e3fb8206838762c5de9f5beee5a01e WatchSource:0}: Error finding container cb542145783b06fffe947c95559550c6f1e3fb8206838762c5de9f5beee5a01e: Status 404 returned error can't find the container with id cb542145783b06fffe947c95559550c6f1e3fb8206838762c5de9f5beee5a01e
Jul 17 21:45:17.574030 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:17.574012    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 21:45:18.044109 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:18.044079    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn" event={"ID":"7881f5f5-2354-4338-8a4d-26db0728a188","Type":"ContainerStarted","Data":"cb542145783b06fffe947c95559550c6f1e3fb8206838762c5de9f5beee5a01e"}
Jul 17 21:45:20.054243 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:20.054216    2573 generic.go:356] "Generic (PLEG): container finished" podID="7881f5f5-2354-4338-8a4d-26db0728a188" containerID="5dccf714f2a579eeb51a410f93e86b8406209151c4cdc525cfc1b4f3e9eba148" exitCode=7
Jul 17 21:45:20.054578 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:20.054257    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn" event={"ID":"7881f5f5-2354-4338-8a4d-26db0728a188","Type":"ContainerDied","Data":"5dccf714f2a579eeb51a410f93e86b8406209151c4cdc525cfc1b4f3e9eba148"}
Jul 17 21:45:20.054578 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:20.054490    2573 scope.go:117] "RemoveContainer" containerID="5dccf714f2a579eeb51a410f93e86b8406209151c4cdc525cfc1b4f3e9eba148"
Jul 17 21:45:21.060626 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:21.060539    2573 generic.go:356] "Generic (PLEG): container finished" podID="7881f5f5-2354-4338-8a4d-26db0728a188" containerID="d37cc7d0a391a91eb9249b5ee8b0906e2b13fb917289230fda4722041848902e" exitCode=7
Jul 17 21:45:21.061003 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:21.060630    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn" event={"ID":"7881f5f5-2354-4338-8a4d-26db0728a188","Type":"ContainerDied","Data":"d37cc7d0a391a91eb9249b5ee8b0906e2b13fb917289230fda4722041848902e"}
Jul 17 21:45:21.061003 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:21.060693    2573 scope.go:117] "RemoveContainer" containerID="5dccf714f2a579eeb51a410f93e86b8406209151c4cdc525cfc1b4f3e9eba148"
Jul 17 21:45:21.061003 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:21.060785    2573 scope.go:117] "RemoveContainer" containerID="d37cc7d0a391a91eb9249b5ee8b0906e2b13fb917289230fda4722041848902e"
Jul 17 21:45:21.061003 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:45:21.060987    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn_odh-ai-gateway-infra(7881f5f5-2354-4338-8a4d-26db0728a188)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn" podUID="7881f5f5-2354-4338-8a4d-26db0728a188"
Jul 17 21:45:22.066787 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:22.066756    2573 scope.go:117] "RemoveContainer" containerID="d37cc7d0a391a91eb9249b5ee8b0906e2b13fb917289230fda4722041848902e"
Jul 17 21:45:22.067162 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:45:22.066943    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn_odh-ai-gateway-infra(7881f5f5-2354-4338-8a4d-26db0728a188)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn" podUID="7881f5f5-2354-4338-8a4d-26db0728a188"
Jul 17 21:45:30.194550 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:30.194509    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm"]
Jul 17 21:45:30.197515 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:30.197496    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm"
Jul 17 21:45:30.205552 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:30.205528    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm"]
Jul 17 21:45:30.333875 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:30.333830    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-5sntm\" (UID: \"4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm"
Jul 17 21:45:30.436318 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:30.435833    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-5sntm\" (UID: \"4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm"
Jul 17 21:45:30.439975 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:30.436945    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-5sntm\" (UID: \"4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm"
Jul 17 21:45:30.508526 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:30.508440    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm"
Jul 17 21:45:30.632565 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:30.632532    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm"]
Jul 17 21:45:30.635615 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:45:30.635587    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d66cc9e_12c5_4a16_93ab_40e8a76ec4a8.slice/crio-9154473f0e497ccf8b22b130499ddfbc54b19dfb28b68cfe9e4f2f99932e1507 WatchSource:0}: Error finding container 9154473f0e497ccf8b22b130499ddfbc54b19dfb28b68cfe9e4f2f99932e1507: Status 404 returned error can't find the container with id 9154473f0e497ccf8b22b130499ddfbc54b19dfb28b68cfe9e4f2f99932e1507
Jul 17 21:45:31.104989 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:31.104901    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm" event={"ID":"4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8","Type":"ContainerStarted","Data":"dc1706561e046d1d99cae52aead78854f8bbd3c1aea8ff64f0b3082623615fa9"}
Jul 17 21:45:31.104989 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:31.104955    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm" event={"ID":"4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8","Type":"ContainerStarted","Data":"9154473f0e497ccf8b22b130499ddfbc54b19dfb28b68cfe9e4f2f99932e1507"}
Jul 17 21:45:32.110485 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:32.110449    2573 generic.go:356] "Generic (PLEG): container finished" podID="4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8" containerID="dc1706561e046d1d99cae52aead78854f8bbd3c1aea8ff64f0b3082623615fa9" exitCode=6
Jul 17 21:45:32.110892 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:32.110520    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm" event={"ID":"4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8","Type":"ContainerDied","Data":"dc1706561e046d1d99cae52aead78854f8bbd3c1aea8ff64f0b3082623615fa9"}
Jul 17 21:45:32.110892 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:32.110680    2573 scope.go:117] "RemoveContainer" containerID="dc1706561e046d1d99cae52aead78854f8bbd3c1aea8ff64f0b3082623615fa9"
Jul 17 21:45:33.115969 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:33.115944    2573 generic.go:356] "Generic (PLEG): container finished" podID="4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8" containerID="48fa84f7cc9b63b21736a877c604f29b80330dc1659425f7472ce2f20eccf0ac" exitCode=6
Jul 17 21:45:33.116374 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:33.116016    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm" event={"ID":"4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8","Type":"ContainerDied","Data":"48fa84f7cc9b63b21736a877c604f29b80330dc1659425f7472ce2f20eccf0ac"}
Jul 17 21:45:33.116374 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:33.116067    2573 scope.go:117] "RemoveContainer" containerID="dc1706561e046d1d99cae52aead78854f8bbd3c1aea8ff64f0b3082623615fa9"
Jul 17 21:45:33.116374 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:33.116241    2573 scope.go:117] "RemoveContainer" containerID="48fa84f7cc9b63b21736a877c604f29b80330dc1659425f7472ce2f20eccf0ac"
Jul 17 21:45:33.116514 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:45:33.116443    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-5sntm_odh-ai-gateway-infra(4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm" podUID="4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8"
Jul 17 21:45:34.121142 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:34.121113    2573 scope.go:117] "RemoveContainer" containerID="48fa84f7cc9b63b21736a877c604f29b80330dc1659425f7472ce2f20eccf0ac"
Jul 17 21:45:34.121509 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:45:34.121322    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-5sntm_odh-ai-gateway-infra(4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm" podUID="4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8"
Jul 17 21:45:35.525086 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:35.525057    2573 scope.go:117] "RemoveContainer" containerID="d37cc7d0a391a91eb9249b5ee8b0906e2b13fb917289230fda4722041848902e"
Jul 17 21:45:36.130150 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:36.130113    2573 generic.go:356] "Generic (PLEG): container finished" podID="7881f5f5-2354-4338-8a4d-26db0728a188" containerID="fc10c883a5ac9d9c9fc44522a33dd3c41704e400eab5855cb72c67195096ba35" exitCode=7
Jul 17 21:45:36.130321 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:36.130184    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn" event={"ID":"7881f5f5-2354-4338-8a4d-26db0728a188","Type":"ContainerDied","Data":"fc10c883a5ac9d9c9fc44522a33dd3c41704e400eab5855cb72c67195096ba35"}
Jul 17 21:45:36.130321 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:36.130278    2573 scope.go:117] "RemoveContainer" containerID="d37cc7d0a391a91eb9249b5ee8b0906e2b13fb917289230fda4722041848902e"
Jul 17 21:45:36.130610 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:36.130594    2573 scope.go:117] "RemoveContainer" containerID="fc10c883a5ac9d9c9fc44522a33dd3c41704e400eab5855cb72c67195096ba35"
Jul 17 21:45:36.130855 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:45:36.130827    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn_odh-ai-gateway-infra(7881f5f5-2354-4338-8a4d-26db0728a188)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn" podUID="7881f5f5-2354-4338-8a4d-26db0728a188"
Jul 17 21:45:37.156328 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:37.156292    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn"]
Jul 17 21:45:37.286001 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:37.285974    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn"
Jul 17 21:45:37.387959 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:37.387929    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7881f5f5-2354-4338-8a4d-26db0728a188-maas-api-service-ca\") pod \"7881f5f5-2354-4338-8a4d-26db0728a188\" (UID: \"7881f5f5-2354-4338-8a4d-26db0728a188\") "
Jul 17 21:45:37.388301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:37.388260    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7881f5f5-2354-4338-8a4d-26db0728a188-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7881f5f5-2354-4338-8a4d-26db0728a188" (UID: "7881f5f5-2354-4338-8a4d-26db0728a188"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:45:37.489366 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:37.489275    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7881f5f5-2354-4338-8a4d-26db0728a188-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:45:38.140627 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:38.140599    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn"
Jul 17 21:45:38.140627 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:38.140609    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn" event={"ID":"7881f5f5-2354-4338-8a4d-26db0728a188","Type":"ContainerDied","Data":"cb542145783b06fffe947c95559550c6f1e3fb8206838762c5de9f5beee5a01e"}
Jul 17 21:45:38.140873 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:38.140663    2573 scope.go:117] "RemoveContainer" containerID="fc10c883a5ac9d9c9fc44522a33dd3c41704e400eab5855cb72c67195096ba35"
Jul 17 21:45:38.160492 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:38.160472    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn"]
Jul 17 21:45:38.166050 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:38.166030    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-zzzrn"]
Jul 17 21:45:38.529084 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:38.529008    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7881f5f5-2354-4338-8a4d-26db0728a188" path="/var/lib/kubelet/pods/7881f5f5-2354-4338-8a4d-26db0728a188/volumes"
Jul 17 21:45:47.524584 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:47.524496    2573 scope.go:117] "RemoveContainer" containerID="48fa84f7cc9b63b21736a877c604f29b80330dc1659425f7472ce2f20eccf0ac"
Jul 17 21:45:48.185043 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:48.184960    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm" event={"ID":"4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8","Type":"ContainerStarted","Data":"a0bd9864518e4ef8e9008e13699db1d954d2b0ed05822b7d4632e43c17478986"}
Jul 17 21:45:48.200226 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:48.200185    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm" podStartSLOduration=18.200173539 podStartE2EDuration="18.200173539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:45:31.119547944 +0000 UTC m=+2231.083668555" watchObservedRunningTime="2026-07-17 21:45:48.200173539 +0000 UTC m=+2248.164294149"
Jul 17 21:45:49.019712 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.019679    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr"]
Jul 17 21:45:49.020143 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.019968    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7881f5f5-2354-4338-8a4d-26db0728a188" containerName="revoke-keys"
Jul 17 21:45:49.020143 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.019979    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7881f5f5-2354-4338-8a4d-26db0728a188" containerName="revoke-keys"
Jul 17 21:45:49.020143 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.019997    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7881f5f5-2354-4338-8a4d-26db0728a188" containerName="revoke-keys"
Jul 17 21:45:49.020143 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.020003    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7881f5f5-2354-4338-8a4d-26db0728a188" containerName="revoke-keys"
Jul 17 21:45:49.020143 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.020053    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7881f5f5-2354-4338-8a4d-26db0728a188" containerName="revoke-keys"
Jul 17 21:45:49.020143 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.020060    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7881f5f5-2354-4338-8a4d-26db0728a188" containerName="revoke-keys"
Jul 17 21:45:49.022817 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.022801    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr"
Jul 17 21:45:49.033813 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.032534    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr"]
Jul 17 21:45:49.175746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.175713    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a044a1d8-7550-4474-92ac-d2d100ba0b64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr\" (UID: \"a044a1d8-7550-4474-92ac-d2d100ba0b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr"
Jul 17 21:45:49.190924 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.190898    2573 generic.go:356] "Generic (PLEG): container finished" podID="4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8" containerID="a0bd9864518e4ef8e9008e13699db1d954d2b0ed05822b7d4632e43c17478986" exitCode=6
Jul 17 21:45:49.191038 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.190973    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm" event={"ID":"4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8","Type":"ContainerDied","Data":"a0bd9864518e4ef8e9008e13699db1d954d2b0ed05822b7d4632e43c17478986"}
Jul 17 21:45:49.191038 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.191032    2573 scope.go:117] "RemoveContainer" containerID="48fa84f7cc9b63b21736a877c604f29b80330dc1659425f7472ce2f20eccf0ac"
Jul 17 21:45:49.191367 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.191345    2573 scope.go:117] "RemoveContainer" containerID="a0bd9864518e4ef8e9008e13699db1d954d2b0ed05822b7d4632e43c17478986"
Jul 17 21:45:49.191573 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:45:49.191552    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-5sntm_odh-ai-gateway-infra(4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm" podUID="4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8"
Jul 17 21:45:49.214549 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.214513    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm"]
Jul 17 21:45:49.276607 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.276544    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a044a1d8-7550-4474-92ac-d2d100ba0b64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr\" (UID: \"a044a1d8-7550-4474-92ac-d2d100ba0b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr"
Jul 17 21:45:49.277160 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.277141    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a044a1d8-7550-4474-92ac-d2d100ba0b64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr\" (UID: \"a044a1d8-7550-4474-92ac-d2d100ba0b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr"
Jul 17 21:45:49.338819 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.338798    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr"
Jul 17 21:45:49.461585 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:49.461557    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr"]
Jul 17 21:45:49.463907 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:45:49.463882    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda044a1d8_7550_4474_92ac_d2d100ba0b64.slice/crio-f091a0cabffceed74bf336dfc4b91c062a1d5550c2ec7dfe62947e69ffdffef8 WatchSource:0}: Error finding container f091a0cabffceed74bf336dfc4b91c062a1d5550c2ec7dfe62947e69ffdffef8: Status 404 returned error can't find the container with id f091a0cabffceed74bf336dfc4b91c062a1d5550c2ec7dfe62947e69ffdffef8
Jul 17 21:45:50.197935 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:50.197905    2573 generic.go:356] "Generic (PLEG): container finished" podID="a044a1d8-7550-4474-92ac-d2d100ba0b64" containerID="81c091c0c0281d71ae75432e9e5993569b71c8c44bddf367c5475379febaf7e6" exitCode=6
Jul 17 21:45:50.198223 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:50.197944    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr" event={"ID":"a044a1d8-7550-4474-92ac-d2d100ba0b64","Type":"ContainerDied","Data":"81c091c0c0281d71ae75432e9e5993569b71c8c44bddf367c5475379febaf7e6"}
Jul 17 21:45:50.198223 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:50.197985    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr" event={"ID":"a044a1d8-7550-4474-92ac-d2d100ba0b64","Type":"ContainerStarted","Data":"f091a0cabffceed74bf336dfc4b91c062a1d5550c2ec7dfe62947e69ffdffef8"}
Jul 17 21:45:50.198223 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:50.198171    2573 scope.go:117] "RemoveContainer" containerID="81c091c0c0281d71ae75432e9e5993569b71c8c44bddf367c5475379febaf7e6"
Jul 17 21:45:50.315273 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:50.315250    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm"
Jul 17 21:45:50.385987 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:50.385962    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8-maas-api-service-ca\") pod \"4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8\" (UID: \"4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8\") "
Jul 17 21:45:50.386328 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:50.386307    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8" (UID: "4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:45:50.486942 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:50.486912    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:45:51.203270 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:51.203182    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm"
Jul 17 21:45:51.203772 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:51.203188    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm" event={"ID":"4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8","Type":"ContainerDied","Data":"9154473f0e497ccf8b22b130499ddfbc54b19dfb28b68cfe9e4f2f99932e1507"}
Jul 17 21:45:51.203772 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:51.203340    2573 scope.go:117] "RemoveContainer" containerID="a0bd9864518e4ef8e9008e13699db1d954d2b0ed05822b7d4632e43c17478986"
Jul 17 21:45:51.204979 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:51.204956    2573 generic.go:356] "Generic (PLEG): container finished" podID="a044a1d8-7550-4474-92ac-d2d100ba0b64" containerID="dd576cac624db099b9b0680d78ff50ceb8bf68270eee28a5a81b479e54295e6e" exitCode=6
Jul 17 21:45:51.205081 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:51.205020    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr" event={"ID":"a044a1d8-7550-4474-92ac-d2d100ba0b64","Type":"ContainerDied","Data":"dd576cac624db099b9b0680d78ff50ceb8bf68270eee28a5a81b479e54295e6e"}
Jul 17 21:45:51.205273 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:51.205250    2573 scope.go:117] "RemoveContainer" containerID="dd576cac624db099b9b0680d78ff50ceb8bf68270eee28a5a81b479e54295e6e"
Jul 17 21:45:51.205519 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:45:51.205498    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr_odh-ai-gateway-infra(a044a1d8-7550-4474-92ac-d2d100ba0b64)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr" podUID="a044a1d8-7550-4474-92ac-d2d100ba0b64"
Jul 17 21:45:51.211711 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:51.211697    2573 scope.go:117] "RemoveContainer" containerID="81c091c0c0281d71ae75432e9e5993569b71c8c44bddf367c5475379febaf7e6"
Jul 17 21:45:51.218838 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:51.218814    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm"]
Jul 17 21:45:51.225031 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:51.225011    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-5sntm"]
Jul 17 21:45:52.211424 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:52.211396    2573 scope.go:117] "RemoveContainer" containerID="dd576cac624db099b9b0680d78ff50ceb8bf68270eee28a5a81b479e54295e6e"
Jul 17 21:45:52.211789 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:45:52.211566    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr_odh-ai-gateway-infra(a044a1d8-7550-4474-92ac-d2d100ba0b64)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr" podUID="a044a1d8-7550-4474-92ac-d2d100ba0b64"
Jul 17 21:45:52.529106 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:45:52.529021    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8" path="/var/lib/kubelet/pods/4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8/volumes"
Jul 17 21:46:06.524418 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:06.524392    2573 scope.go:117] "RemoveContainer" containerID="dd576cac624db099b9b0680d78ff50ceb8bf68270eee28a5a81b479e54295e6e"
Jul 17 21:46:07.273935 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:07.273905    2573 generic.go:356] "Generic (PLEG): container finished" podID="a044a1d8-7550-4474-92ac-d2d100ba0b64" containerID="0ed5d1c30b8d313df1760a17e9da5e59d05142a68c842f5fa71c16536ed16da1" exitCode=6
Jul 17 21:46:07.274067 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:07.273974    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr" event={"ID":"a044a1d8-7550-4474-92ac-d2d100ba0b64","Type":"ContainerDied","Data":"0ed5d1c30b8d313df1760a17e9da5e59d05142a68c842f5fa71c16536ed16da1"}
Jul 17 21:46:07.274067 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:07.274030    2573 scope.go:117] "RemoveContainer" containerID="dd576cac624db099b9b0680d78ff50ceb8bf68270eee28a5a81b479e54295e6e"
Jul 17 21:46:07.274351 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:07.274334    2573 scope.go:117] "RemoveContainer" containerID="0ed5d1c30b8d313df1760a17e9da5e59d05142a68c842f5fa71c16536ed16da1"
Jul 17 21:46:07.274580 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:46:07.274560    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr_odh-ai-gateway-infra(a044a1d8-7550-4474-92ac-d2d100ba0b64)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr" podUID="a044a1d8-7550-4474-92ac-d2d100ba0b64"
Jul 17 21:46:08.301163 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:08.301125    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr"]
Jul 17 21:46:08.427923 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:08.427898    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr"
Jul 17 21:46:08.522509 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:08.522478    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a044a1d8-7550-4474-92ac-d2d100ba0b64-maas-api-service-ca\") pod \"a044a1d8-7550-4474-92ac-d2d100ba0b64\" (UID: \"a044a1d8-7550-4474-92ac-d2d100ba0b64\") "
Jul 17 21:46:08.522857 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:08.522832    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a044a1d8-7550-4474-92ac-d2d100ba0b64-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a044a1d8-7550-4474-92ac-d2d100ba0b64" (UID: "a044a1d8-7550-4474-92ac-d2d100ba0b64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:46:08.623330 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:08.623299    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a044a1d8-7550-4474-92ac-d2d100ba0b64-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:46:09.284225 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:09.284187    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr" event={"ID":"a044a1d8-7550-4474-92ac-d2d100ba0b64","Type":"ContainerDied","Data":"f091a0cabffceed74bf336dfc4b91c062a1d5550c2ec7dfe62947e69ffdffef8"}
Jul 17 21:46:09.284433 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:09.284255    2573 scope.go:117] "RemoveContainer" containerID="0ed5d1c30b8d313df1760a17e9da5e59d05142a68c842f5fa71c16536ed16da1"
Jul 17 21:46:09.284433 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:09.284197    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr"
Jul 17 21:46:09.299625 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:09.299604    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr"]
Jul 17 21:46:09.305581 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:09.305561    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8kzpr"]
Jul 17 21:46:10.529305 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:10.529255    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a044a1d8-7550-4474-92ac-d2d100ba0b64" path="/var/lib/kubelet/pods/a044a1d8-7550-4474-92ac-d2d100ba0b64/volumes"
Jul 17 21:46:30.238728 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.238695    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp"]
Jul 17 21:46:30.239368 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239133    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8" containerName="revoke-keys"
Jul 17 21:46:30.239368 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239152    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8" containerName="revoke-keys"
Jul 17 21:46:30.239368 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239166    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8" containerName="revoke-keys"
Jul 17 21:46:30.239368 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239175    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8" containerName="revoke-keys"
Jul 17 21:46:30.239368 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239195    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8" containerName="revoke-keys"
Jul 17 21:46:30.239368 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239203    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8" containerName="revoke-keys"
Jul 17 21:46:30.239368 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239214    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a044a1d8-7550-4474-92ac-d2d100ba0b64" containerName="revoke-keys"
Jul 17 21:46:30.239368 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239222    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a044a1d8-7550-4474-92ac-d2d100ba0b64" containerName="revoke-keys"
Jul 17 21:46:30.239368 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239233    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a044a1d8-7550-4474-92ac-d2d100ba0b64" containerName="revoke-keys"
Jul 17 21:46:30.239368 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239242    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a044a1d8-7550-4474-92ac-d2d100ba0b64" containerName="revoke-keys"
Jul 17 21:46:30.239368 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239252    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7881f5f5-2354-4338-8a4d-26db0728a188" containerName="revoke-keys"
Jul 17 21:46:30.239368 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239260    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7881f5f5-2354-4338-8a4d-26db0728a188" containerName="revoke-keys"
Jul 17 21:46:30.239368 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239273    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a044a1d8-7550-4474-92ac-d2d100ba0b64" containerName="revoke-keys"
Jul 17 21:46:30.239368 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239299    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a044a1d8-7550-4474-92ac-d2d100ba0b64" containerName="revoke-keys"
Jul 17 21:46:30.240017 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239404    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a044a1d8-7550-4474-92ac-d2d100ba0b64" containerName="revoke-keys"
Jul 17 21:46:30.240017 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239414    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8" containerName="revoke-keys"
Jul 17 21:46:30.240017 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239422    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8" containerName="revoke-keys"
Jul 17 21:46:30.240017 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239432    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a044a1d8-7550-4474-92ac-d2d100ba0b64" containerName="revoke-keys"
Jul 17 21:46:30.240017 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.239441    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7881f5f5-2354-4338-8a4d-26db0728a188" containerName="revoke-keys"
Jul 17 21:46:30.242442 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.242424    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp"
Jul 17 21:46:30.244855 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.244833    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:46:30.244949 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.244903    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:46:30.249368 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.249048    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp"]
Jul 17 21:46:30.286158 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.286125    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/849e6be6-9fac-445a-a58c-8dc6ba6cd5a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-6b9bp\" (UID: \"849e6be6-9fac-445a-a58c-8dc6ba6cd5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp"
Jul 17 21:46:30.387044 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.387010    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/849e6be6-9fac-445a-a58c-8dc6ba6cd5a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-6b9bp\" (UID: \"849e6be6-9fac-445a-a58c-8dc6ba6cd5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp"
Jul 17 21:46:30.387729 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.387701    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/849e6be6-9fac-445a-a58c-8dc6ba6cd5a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-6b9bp\" (UID: \"849e6be6-9fac-445a-a58c-8dc6ba6cd5a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp"
Jul 17 21:46:30.553833 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.553753    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp"
Jul 17 21:46:30.679259 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:30.679233    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp"]
Jul 17 21:46:30.681483 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:46:30.681458    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849e6be6_9fac_445a_a58c_8dc6ba6cd5a3.slice/crio-955b09b81342652f0d89dc457929d8cdba60428b9911ae67d659635367988bd9 WatchSource:0}: Error finding container 955b09b81342652f0d89dc457929d8cdba60428b9911ae67d659635367988bd9: Status 404 returned error can't find the container with id 955b09b81342652f0d89dc457929d8cdba60428b9911ae67d659635367988bd9
Jul 17 21:46:31.372769 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:31.372731    2573 generic.go:356] "Generic (PLEG): container finished" podID="849e6be6-9fac-445a-a58c-8dc6ba6cd5a3" containerID="7d35b9632a1eb2e65979303f84dc7861b924d24eafa3c585ba0c48a53b4e581b" exitCode=6
Jul 17 21:46:31.373117 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:31.372786    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp" event={"ID":"849e6be6-9fac-445a-a58c-8dc6ba6cd5a3","Type":"ContainerDied","Data":"7d35b9632a1eb2e65979303f84dc7861b924d24eafa3c585ba0c48a53b4e581b"}
Jul 17 21:46:31.373117 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:31.372840    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp" event={"ID":"849e6be6-9fac-445a-a58c-8dc6ba6cd5a3","Type":"ContainerStarted","Data":"955b09b81342652f0d89dc457929d8cdba60428b9911ae67d659635367988bd9"}
Jul 17 21:46:31.373117 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:31.372958    2573 scope.go:117] "RemoveContainer" containerID="7d35b9632a1eb2e65979303f84dc7861b924d24eafa3c585ba0c48a53b4e581b"
Jul 17 21:46:32.379239 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:32.379204    2573 generic.go:356] "Generic (PLEG): container finished" podID="849e6be6-9fac-445a-a58c-8dc6ba6cd5a3" containerID="fca889706c9b95b9ceeca4d51032c15983a9d885e057d4bbcf6dbdf48d14e268" exitCode=6
Jul 17 21:46:32.379696 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:32.379255    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp" event={"ID":"849e6be6-9fac-445a-a58c-8dc6ba6cd5a3","Type":"ContainerDied","Data":"fca889706c9b95b9ceeca4d51032c15983a9d885e057d4bbcf6dbdf48d14e268"}
Jul 17 21:46:32.379696 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:32.379321    2573 scope.go:117] "RemoveContainer" containerID="7d35b9632a1eb2e65979303f84dc7861b924d24eafa3c585ba0c48a53b4e581b"
Jul 17 21:46:32.379696 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:32.379474    2573 scope.go:117] "RemoveContainer" containerID="fca889706c9b95b9ceeca4d51032c15983a9d885e057d4bbcf6dbdf48d14e268"
Jul 17 21:46:32.379811 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:46:32.379708    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-6b9bp_odh-ai-gateway-infra(849e6be6-9fac-445a-a58c-8dc6ba6cd5a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp" podUID="849e6be6-9fac-445a-a58c-8dc6ba6cd5a3"
Jul 17 21:46:33.384796 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:33.384767    2573 scope.go:117] "RemoveContainer" containerID="fca889706c9b95b9ceeca4d51032c15983a9d885e057d4bbcf6dbdf48d14e268"
Jul 17 21:46:33.385174 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:46:33.384937    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-6b9bp_odh-ai-gateway-infra(849e6be6-9fac-445a-a58c-8dc6ba6cd5a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp" podUID="849e6be6-9fac-445a-a58c-8dc6ba6cd5a3"
Jul 17 21:46:45.524589 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:45.524561    2573 scope.go:117] "RemoveContainer" containerID="fca889706c9b95b9ceeca4d51032c15983a9d885e057d4bbcf6dbdf48d14e268"
Jul 17 21:46:46.440856 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:46.440778    2573 generic.go:356] "Generic (PLEG): container finished" podID="849e6be6-9fac-445a-a58c-8dc6ba6cd5a3" containerID="b66243c9e6466c24824b222ef0fbf4b11e09598ab5c0a4debfcb474d8600e83b" exitCode=6
Jul 17 21:46:46.440980 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:46.440857    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp" event={"ID":"849e6be6-9fac-445a-a58c-8dc6ba6cd5a3","Type":"ContainerDied","Data":"b66243c9e6466c24824b222ef0fbf4b11e09598ab5c0a4debfcb474d8600e83b"}
Jul 17 21:46:46.440980 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:46.440921    2573 scope.go:117] "RemoveContainer" containerID="fca889706c9b95b9ceeca4d51032c15983a9d885e057d4bbcf6dbdf48d14e268"
Jul 17 21:46:46.441265 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:46.441249    2573 scope.go:117] "RemoveContainer" containerID="b66243c9e6466c24824b222ef0fbf4b11e09598ab5c0a4debfcb474d8600e83b"
Jul 17 21:46:46.441511 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:46:46.441485    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-6b9bp_odh-ai-gateway-infra(849e6be6-9fac-445a-a58c-8dc6ba6cd5a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp" podUID="849e6be6-9fac-445a-a58c-8dc6ba6cd5a3"
Jul 17 21:46:47.232451 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:47.232412    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"]
Jul 17 21:46:47.232928 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:47.232721    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv" podUID="86d4ad95-76d4-4bbf-805a-dce69fbf77b9" containerName="istio-proxy" containerID="cri-o://7bcc8fb641c46b6dd0445d9467dcc36257e8dd703451d7d45b4a937250636b53" gracePeriod=30
Jul 17 21:46:47.467508 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:47.467473    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp"]
Jul 17 21:46:47.595129 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:47.595105    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp"
Jul 17 21:46:47.721300 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:47.721252    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/849e6be6-9fac-445a-a58c-8dc6ba6cd5a3-maas-api-service-ca\") pod \"849e6be6-9fac-445a-a58c-8dc6ba6cd5a3\" (UID: \"849e6be6-9fac-445a-a58c-8dc6ba6cd5a3\") "
Jul 17 21:46:47.721599 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:47.721576    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/849e6be6-9fac-445a-a58c-8dc6ba6cd5a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "849e6be6-9fac-445a-a58c-8dc6ba6cd5a3" (UID: "849e6be6-9fac-445a-a58c-8dc6ba6cd5a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:46:47.822068 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:47.822002    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/849e6be6-9fac-445a-a58c-8dc6ba6cd5a3-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:46:48.451178 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:48.451140    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp" event={"ID":"849e6be6-9fac-445a-a58c-8dc6ba6cd5a3","Type":"ContainerDied","Data":"955b09b81342652f0d89dc457929d8cdba60428b9911ae67d659635367988bd9"}
Jul 17 21:46:48.451673 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:48.451199    2573 scope.go:117] "RemoveContainer" containerID="b66243c9e6466c24824b222ef0fbf4b11e09598ab5c0a4debfcb474d8600e83b"
Jul 17 21:46:48.451673 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:48.451154    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp"
Jul 17 21:46:48.471683 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:48.471661    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp"]
Jul 17 21:46:48.478615 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:48.478596    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-6b9bp"]
Jul 17 21:46:48.529461 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:48.529439    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="849e6be6-9fac-445a-a58c-8dc6ba6cd5a3" path="/var/lib/kubelet/pods/849e6be6-9fac-445a-a58c-8dc6ba6cd5a3/volumes"
Jul 17 21:46:49.074930 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:49.074896    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v"]
Jul 17 21:46:49.075182 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:49.075170    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="849e6be6-9fac-445a-a58c-8dc6ba6cd5a3" containerName="revoke-keys"
Jul 17 21:46:49.075236 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:49.075185    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="849e6be6-9fac-445a-a58c-8dc6ba6cd5a3" containerName="revoke-keys"
Jul 17 21:46:49.075236 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:49.075200    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="849e6be6-9fac-445a-a58c-8dc6ba6cd5a3" containerName="revoke-keys"
Jul 17 21:46:49.075236 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:49.075207    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="849e6be6-9fac-445a-a58c-8dc6ba6cd5a3" containerName="revoke-keys"
Jul 17 21:46:49.075374 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:49.075253    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="849e6be6-9fac-445a-a58c-8dc6ba6cd5a3" containerName="revoke-keys"
Jul 17 21:46:49.075374 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:49.075261    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a044a1d8-7550-4474-92ac-d2d100ba0b64" containerName="revoke-keys"
Jul 17 21:46:49.075374 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:49.075267    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d66cc9e-12c5-4a16-93ab-40e8a76ec4a8" containerName="revoke-keys"
Jul 17 21:46:49.075374 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:49.075274    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="849e6be6-9fac-445a-a58c-8dc6ba6cd5a3" containerName="revoke-keys"
Jul 17 21:46:49.079369 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:49.079348    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v"
Jul 17 21:46:49.081685 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:49.081661    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:46:49.081815 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:49.081660    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:46:49.085951 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:49.085925    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v"]
Jul 17 21:46:49.131636 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:49.131612    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v\" (UID: \"005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v"
Jul 17 21:46:49.231929 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:49.231899    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v\" (UID: \"005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v"
Jul 17 21:46:49.232525 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:49.232508    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v\" (UID: \"005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v"
Jul 17 21:46:49.391068 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:49.391047    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v"
Jul 17 21:46:49.514807 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:49.514779    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v"]
Jul 17 21:46:49.515999 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:46:49.515967    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005e1ac5_f99a_4d7d_bdcf_cd7e61a03aed.slice/crio-d3cdf24dd18f78f065220d57024b50c377dd366771207f06d0521026f0df65bf WatchSource:0}: Error finding container d3cdf24dd18f78f065220d57024b50c377dd366771207f06d0521026f0df65bf: Status 404 returned error can't find the container with id d3cdf24dd18f78f065220d57024b50c377dd366771207f06d0521026f0df65bf
Jul 17 21:46:50.462545 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:50.462439    2573 generic.go:356] "Generic (PLEG): container finished" podID="005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed" containerID="7b9e6aed1945a7b596f700961b7fcf66701ee332386e1cb073d2a648a45394e7" exitCode=6
Jul 17 21:46:50.462545 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:50.462481    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v" event={"ID":"005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed","Type":"ContainerDied","Data":"7b9e6aed1945a7b596f700961b7fcf66701ee332386e1cb073d2a648a45394e7"}
Jul 17 21:46:50.462719 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:50.462549    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v" event={"ID":"005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed","Type":"ContainerStarted","Data":"d3cdf24dd18f78f065220d57024b50c377dd366771207f06d0521026f0df65bf"}
Jul 17 21:46:50.462760 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:50.462750    2573 scope.go:117] "RemoveContainer" containerID="7b9e6aed1945a7b596f700961b7fcf66701ee332386e1cb073d2a648a45394e7"
Jul 17 21:46:51.467834 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:51.467805    2573 generic.go:356] "Generic (PLEG): container finished" podID="005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed" containerID="6d3f5642dff2ec5f11fda7fa6aeaeb37085148c1e97f1a6ba4129e631df2879b" exitCode=6
Jul 17 21:46:51.468315 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:51.467843    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v" event={"ID":"005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed","Type":"ContainerDied","Data":"6d3f5642dff2ec5f11fda7fa6aeaeb37085148c1e97f1a6ba4129e631df2879b"}
Jul 17 21:46:51.468315 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:51.467890    2573 scope.go:117] "RemoveContainer" containerID="7b9e6aed1945a7b596f700961b7fcf66701ee332386e1cb073d2a648a45394e7"
Jul 17 21:46:51.468315 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:51.468093    2573 scope.go:117] "RemoveContainer" containerID="6d3f5642dff2ec5f11fda7fa6aeaeb37085148c1e97f1a6ba4129e631df2879b"
Jul 17 21:46:51.468490 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:46:51.468347    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v_odh-ai-gateway-infra(005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v" podUID="005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed"
Jul 17 21:46:52.475470 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.475430    2573 generic.go:356] "Generic (PLEG): container finished" podID="86d4ad95-76d4-4bbf-805a-dce69fbf77b9" containerID="7bcc8fb641c46b6dd0445d9467dcc36257e8dd703451d7d45b4a937250636b53" exitCode=0
Jul 17 21:46:52.475820 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.475514    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv" event={"ID":"86d4ad95-76d4-4bbf-805a-dce69fbf77b9","Type":"ContainerDied","Data":"7bcc8fb641c46b6dd0445d9467dcc36257e8dd703451d7d45b4a937250636b53"}
Jul 17 21:46:52.475820 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.475654    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv" event={"ID":"86d4ad95-76d4-4bbf-805a-dce69fbf77b9","Type":"ContainerDied","Data":"3017428cd0395d835d8e73dcefc318382e4ba7915caf59cab72b1847826da3f6"}
Jul 17 21:46:52.475820 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.475683    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3017428cd0395d835d8e73dcefc318382e4ba7915caf59cab72b1847826da3f6"
Jul 17 21:46:52.475820 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.475721    2573 scope.go:117] "RemoveContainer" containerID="671f83d56d012b3caded4ab3c1dc107ff7a24c78c4648523b9e3d93b0cecbffe"
Jul 17 21:46:52.477948 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.477840    2573 scope.go:117] "RemoveContainer" containerID="6d3f5642dff2ec5f11fda7fa6aeaeb37085148c1e97f1a6ba4129e631df2879b"
Jul 17 21:46:52.478067 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:46:52.478033    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v_odh-ai-gateway-infra(005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v" podUID="005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed"
Jul 17 21:46:52.480958 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.480936    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:46:52.557617 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.557589    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-podinfo\") pod \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") "
Jul 17 21:46:52.557751 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.557636    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istiod-ca-cert\") pod \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") "
Jul 17 21:46:52.557751 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.557656    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-workload-certs\") pod \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") "
Jul 17 21:46:52.557869 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.557847    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-envoy\") pod \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") "
Jul 17 21:46:52.557935 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.557923    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-token\") pod \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") "
Jul 17 21:46:52.557935 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.557924    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "86d4ad95-76d4-4bbf-805a-dce69fbf77b9" (UID: "86d4ad95-76d4-4bbf-805a-dce69fbf77b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:46:52.558032 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.557947    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-data\") pod \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") "
Jul 17 21:46:52.558032 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.557970    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-credential-socket\") pod \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") "
Jul 17 21:46:52.558032 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.557983    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "86d4ad95-76d4-4bbf-805a-dce69fbf77b9" (UID: "86d4ad95-76d4-4bbf-805a-dce69fbf77b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:46:52.558032 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.558002    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgfnt\" (UniqueName: \"kubernetes.io/projected/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-kube-api-access-mgfnt\") pod \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") "
Jul 17 21:46:52.558229 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.558035    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-workload-socket\") pod \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\" (UID: \"86d4ad95-76d4-4bbf-805a-dce69fbf77b9\") "
Jul 17 21:46:52.558308 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.558275    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:46:52.558373 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.558311    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:46:52.558435 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.558410    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "86d4ad95-76d4-4bbf-805a-dce69fbf77b9" (UID: "86d4ad95-76d4-4bbf-805a-dce69fbf77b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:46:52.558739 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.558711    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "86d4ad95-76d4-4bbf-805a-dce69fbf77b9" (UID: "86d4ad95-76d4-4bbf-805a-dce69fbf77b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:46:52.558905 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.558881    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-data" (OuterVolumeSpecName: "istio-data") pod "86d4ad95-76d4-4bbf-805a-dce69fbf77b9" (UID: "86d4ad95-76d4-4bbf-805a-dce69fbf77b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:46:52.560022 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.559994    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "86d4ad95-76d4-4bbf-805a-dce69fbf77b9" (UID: "86d4ad95-76d4-4bbf-805a-dce69fbf77b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 21:46:52.560380 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.560349    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "86d4ad95-76d4-4bbf-805a-dce69fbf77b9" (UID: "86d4ad95-76d4-4bbf-805a-dce69fbf77b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:46:52.560658 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.560631    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-kube-api-access-mgfnt" (OuterVolumeSpecName: "kube-api-access-mgfnt") pod "86d4ad95-76d4-4bbf-805a-dce69fbf77b9" (UID: "86d4ad95-76d4-4bbf-805a-dce69fbf77b9"). InnerVolumeSpecName "kube-api-access-mgfnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:46:52.560729 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.560713    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-token" (OuterVolumeSpecName: "istio-token") pod "86d4ad95-76d4-4bbf-805a-dce69fbf77b9" (UID: "86d4ad95-76d4-4bbf-805a-dce69fbf77b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:46:52.658971 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.658944    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:46:52.658971 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.658971    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:46:52.659124 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.658981    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:46:52.659124 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.658991    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgfnt\" (UniqueName: \"kubernetes.io/projected/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-kube-api-access-mgfnt\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:46:52.659124 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.659000    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:46:52.659124 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.659008    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:46:52.659124 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:52.659016    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d4ad95-76d4-4bbf-805a-dce69fbf77b9-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:46:53.482869 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:53.482839    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"
Jul 17 21:46:53.515093 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:53.515064    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"]
Jul 17 21:46:53.521637 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:53.521611    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-db4eb08b-openshift-default-fbdb9df8f-qppwv"]
Jul 17 21:46:54.528818 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:54.528786    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86d4ad95-76d4-4bbf-805a-dce69fbf77b9" path="/var/lib/kubelet/pods/86d4ad95-76d4-4bbf-805a-dce69fbf77b9/volumes"
Jul 17 21:46:59.102159 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.102128    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"]
Jul 17 21:46:59.102614 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.102579    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86d4ad95-76d4-4bbf-805a-dce69fbf77b9" containerName="istio-proxy"
Jul 17 21:46:59.102614 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.102605    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="86d4ad95-76d4-4bbf-805a-dce69fbf77b9" containerName="istio-proxy"
Jul 17 21:46:59.102795 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.102620    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="849e6be6-9fac-445a-a58c-8dc6ba6cd5a3" containerName="revoke-keys"
Jul 17 21:46:59.102795 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.102629    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="849e6be6-9fac-445a-a58c-8dc6ba6cd5a3" containerName="revoke-keys"
Jul 17 21:46:59.102795 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.102652    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86d4ad95-76d4-4bbf-805a-dce69fbf77b9" containerName="istio-proxy"
Jul 17 21:46:59.102795 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.102661    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="86d4ad95-76d4-4bbf-805a-dce69fbf77b9" containerName="istio-proxy"
Jul 17 21:46:59.102795 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.102746    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="849e6be6-9fac-445a-a58c-8dc6ba6cd5a3" containerName="revoke-keys"
Jul 17 21:46:59.102795 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.102761    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="86d4ad95-76d4-4bbf-805a-dce69fbf77b9" containerName="istio-proxy"
Jul 17 21:46:59.102795 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.102772    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="86d4ad95-76d4-4bbf-805a-dce69fbf77b9" containerName="istio-proxy"
Jul 17 21:46:59.105780 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.105759    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.108381 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.108362    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-eecee1fd-openshift-default-dockercfg-582jk\""
Jul 17 21:46:59.119809 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.119787    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"]
Jul 17 21:46:59.203274 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.203242    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-workload-socket\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.203434 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.203321    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istiod-ca-cert\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.203434 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.203346    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-envoy\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.203434 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.203361    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-credential-socket\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.203532 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.203461    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-data\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.203567 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.203532    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmmpg\" (UniqueName: \"kubernetes.io/projected/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-kube-api-access-hmmpg\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.203567 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.203552    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-token\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.203626 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.203582    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-workload-certs\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.203626 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.203608    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-podinfo\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.304627 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.304595    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istiod-ca-cert\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.304627 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.304629    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-envoy\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.304795 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.304646    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-credential-socket\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.304795 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.304677    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-data\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.304795 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.304711    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmmpg\" (UniqueName: \"kubernetes.io/projected/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-kube-api-access-hmmpg\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.304795 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.304738    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-token\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.304795 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.304771    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-workload-certs\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.305020 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.304797    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-podinfo\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.305020 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.304822    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-workload-socket\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.305181 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.305155    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-data\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.305181 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.305168    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-credential-socket\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.305332 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.305205    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-workload-socket\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.305332 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.305234    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-workload-certs\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.305452 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.305432    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istiod-ca-cert\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.307020 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.306995    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-envoy\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.307430 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.307411    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-podinfo\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.313014 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.312994    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-token\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.313300 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.313268    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmmpg\" (UniqueName: \"kubernetes.io/projected/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-kube-api-access-hmmpg\") pod \"e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") " pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.416632 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.416576    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:46:59.548431 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.548408    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"]
Jul 17 21:46:59.550960 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:46:59.550934    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b24183d_ab11_40c6_aef5_be4d4adcd2b7.slice/crio-c083386f8a363474a585eb51e1d962118c9128392a3aa1428ca24cb505d7cce4 WatchSource:0}: Error finding container c083386f8a363474a585eb51e1d962118c9128392a3aa1428ca24cb505d7cce4: Status 404 returned error can't find the container with id c083386f8a363474a585eb51e1d962118c9128392a3aa1428ca24cb505d7cce4
Jul 17 21:46:59.553082 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.553047    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:46:59.553166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.553113    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:46:59.553166 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:46:59.553144    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:47:00.511960 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:00.511926    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j" event={"ID":"5b24183d-ab11-40c6-aef5-be4d4adcd2b7","Type":"ContainerStarted","Data":"582cc1ed9d88aa9db5931b96422f07e295ae1f129ec09a790867dce4319f5b91"}
Jul 17 21:47:00.512384 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:00.511973    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j" event={"ID":"5b24183d-ab11-40c6-aef5-be4d4adcd2b7","Type":"ContainerStarted","Data":"c083386f8a363474a585eb51e1d962118c9128392a3aa1428ca24cb505d7cce4"}
Jul 17 21:47:00.539485 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:00.539435    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j" podStartSLOduration=1.5394158660000001 podStartE2EDuration="1.539415866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:00.537805549 +0000 UTC m=+2320.501926160" watchObservedRunningTime="2026-07-17 21:47:00.539415866 +0000 UTC m=+2320.503536478"
Jul 17 21:47:01.416763 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:01.416730    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:47:01.421403 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:01.421380    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:47:01.516962 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:01.516933    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:47:01.517653 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:01.517637    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:47:04.651253 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:04.651224    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb"]
Jul 17 21:47:04.656316 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:04.656233    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb"
Jul 17 21:47:04.661144 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:04.661118    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb"]
Jul 17 21:47:04.745933 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:04.745905    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed883b47-6eb9-4cd8-832b-50148625b581-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb\" (UID: \"ed883b47-6eb9-4cd8-832b-50148625b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb"
Jul 17 21:47:04.846461 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:04.846430    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed883b47-6eb9-4cd8-832b-50148625b581-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb\" (UID: \"ed883b47-6eb9-4cd8-832b-50148625b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb"
Jul 17 21:47:04.847028 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:04.847008    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed883b47-6eb9-4cd8-832b-50148625b581-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb\" (UID: \"ed883b47-6eb9-4cd8-832b-50148625b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb"
Jul 17 21:47:04.968594 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:04.968535    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb"
Jul 17 21:47:05.090371 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:05.090338    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb"]
Jul 17 21:47:05.093419 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:47:05.093392    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded883b47_6eb9_4cd8_832b_50148625b581.slice/crio-d99f928214298577dd435c66630e06945790ad522d964efc18fcfd5dab7bfd4f WatchSource:0}: Error finding container d99f928214298577dd435c66630e06945790ad522d964efc18fcfd5dab7bfd4f: Status 404 returned error can't find the container with id d99f928214298577dd435c66630e06945790ad522d964efc18fcfd5dab7bfd4f
Jul 17 21:47:05.535724 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:05.535638    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb" event={"ID":"ed883b47-6eb9-4cd8-832b-50148625b581","Type":"ContainerStarted","Data":"1d13d9a05cd72032eae96d49ba09fe0a86eea899043bcd42dfd3134663e955be"}
Jul 17 21:47:05.535724 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:05.535692    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb" event={"ID":"ed883b47-6eb9-4cd8-832b-50148625b581","Type":"ContainerStarted","Data":"d99f928214298577dd435c66630e06945790ad522d964efc18fcfd5dab7bfd4f"}
Jul 17 21:47:05.550025 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:05.549979    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb" podStartSLOduration=1.5499632669999999 podStartE2EDuration="1.549963267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:05.548417286 +0000 UTC m=+2325.512537897" watchObservedRunningTime="2026-07-17 21:47:05.549963267 +0000 UTC m=+2325.514083898"
Jul 17 21:47:06.540649 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:06.540614    2573 generic.go:356] "Generic (PLEG): container finished" podID="ed883b47-6eb9-4cd8-832b-50148625b581" containerID="1d13d9a05cd72032eae96d49ba09fe0a86eea899043bcd42dfd3134663e955be" exitCode=7
Jul 17 21:47:06.540649 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:06.540655    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb" event={"ID":"ed883b47-6eb9-4cd8-832b-50148625b581","Type":"ContainerDied","Data":"1d13d9a05cd72032eae96d49ba09fe0a86eea899043bcd42dfd3134663e955be"}
Jul 17 21:47:06.541167 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:06.540909    2573 scope.go:117] "RemoveContainer" containerID="1d13d9a05cd72032eae96d49ba09fe0a86eea899043bcd42dfd3134663e955be"
Jul 17 21:47:07.525052 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:07.524986    2573 scope.go:117] "RemoveContainer" containerID="6d3f5642dff2ec5f11fda7fa6aeaeb37085148c1e97f1a6ba4129e631df2879b"
Jul 17 21:47:07.548210 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:07.548180    2573 generic.go:356] "Generic (PLEG): container finished" podID="ed883b47-6eb9-4cd8-832b-50148625b581" containerID="49c5347cbe41746ae5a330825951e107cc1c0a83d071b8087645ca3cb8c4e53f" exitCode=7
Jul 17 21:47:07.548610 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:07.548251    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb" event={"ID":"ed883b47-6eb9-4cd8-832b-50148625b581","Type":"ContainerDied","Data":"49c5347cbe41746ae5a330825951e107cc1c0a83d071b8087645ca3cb8c4e53f"}
Jul 17 21:47:07.548610 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:07.548335    2573 scope.go:117] "RemoveContainer" containerID="1d13d9a05cd72032eae96d49ba09fe0a86eea899043bcd42dfd3134663e955be"
Jul 17 21:47:07.548610 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:07.548517    2573 scope.go:117] "RemoveContainer" containerID="49c5347cbe41746ae5a330825951e107cc1c0a83d071b8087645ca3cb8c4e53f"
Jul 17 21:47:07.548791 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:47:07.548738    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb_odh-ai-gateway-infra(ed883b47-6eb9-4cd8-832b-50148625b581)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb" podUID="ed883b47-6eb9-4cd8-832b-50148625b581"
Jul 17 21:47:08.556308 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:08.556192    2573 scope.go:117] "RemoveContainer" containerID="49c5347cbe41746ae5a330825951e107cc1c0a83d071b8087645ca3cb8c4e53f"
Jul 17 21:47:08.556741 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:47:08.556466    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb_odh-ai-gateway-infra(ed883b47-6eb9-4cd8-832b-50148625b581)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb" podUID="ed883b47-6eb9-4cd8-832b-50148625b581"
Jul 17 21:47:08.557923 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:08.557898    2573 generic.go:356] "Generic (PLEG): container finished" podID="005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed" containerID="35da2627099f4a21819386a86648689a7849e4c5fd7ace2a5b1452b8b9525fbc" exitCode=6
Jul 17 21:47:08.558055 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:08.557933    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v" event={"ID":"005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed","Type":"ContainerDied","Data":"35da2627099f4a21819386a86648689a7849e4c5fd7ace2a5b1452b8b9525fbc"}
Jul 17 21:47:08.558055 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:08.557987    2573 scope.go:117] "RemoveContainer" containerID="6d3f5642dff2ec5f11fda7fa6aeaeb37085148c1e97f1a6ba4129e631df2879b"
Jul 17 21:47:08.558314 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:08.558297    2573 scope.go:117] "RemoveContainer" containerID="35da2627099f4a21819386a86648689a7849e4c5fd7ace2a5b1452b8b9525fbc"
Jul 17 21:47:08.558492 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:47:08.558467    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v_odh-ai-gateway-infra(005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v" podUID="005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed"
Jul 17 21:47:09.596956 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:09.596924    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v"]
Jul 17 21:47:09.772939 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:09.772917    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v"
Jul 17 21:47:09.885872 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:09.885790    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed-maas-api-service-ca\") pod \"005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed\" (UID: \"005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed\") "
Jul 17 21:47:09.886145 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:09.886120    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed" (UID: "005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:47:09.986473 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:09.986438    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:47:10.569753 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:10.569730    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v"
Jul 17 21:47:10.569893 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:10.569732    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v" event={"ID":"005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed","Type":"ContainerDied","Data":"d3cdf24dd18f78f065220d57024b50c377dd366771207f06d0521026f0df65bf"}
Jul 17 21:47:10.569893 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:10.569854    2573 scope.go:117] "RemoveContainer" containerID="35da2627099f4a21819386a86648689a7849e4c5fd7ace2a5b1452b8b9525fbc"
Jul 17 21:47:10.585696 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:10.585663    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v"]
Jul 17 21:47:10.588925 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:10.588902    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5fj2v"]
Jul 17 21:47:12.532420 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:12.532379    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed" path="/var/lib/kubelet/pods/005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed/volumes"
Jul 17 21:47:15.593455 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:15.593424    2573 generic.go:356] "Generic (PLEG): container finished" podID="5b24183d-ab11-40c6-aef5-be4d4adcd2b7" containerID="582cc1ed9d88aa9db5931b96422f07e295ae1f129ec09a790867dce4319f5b91" exitCode=0
Jul 17 21:47:15.593852 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:15.593469    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j" event={"ID":"5b24183d-ab11-40c6-aef5-be4d4adcd2b7","Type":"ContainerDied","Data":"582cc1ed9d88aa9db5931b96422f07e295ae1f129ec09a790867dce4319f5b91"}
Jul 17 21:47:15.593852 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:15.593803    2573 scope.go:117] "RemoveContainer" containerID="582cc1ed9d88aa9db5931b96422f07e295ae1f129ec09a790867dce4319f5b91"
Jul 17 21:47:15.594783 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:15.594754    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:47:15.594836 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:15.594822    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:47:15.594886 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:15.594874    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:47:16.517218 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:16.517182    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:47:16.599376 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:16.599336    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j" event={"ID":"5b24183d-ab11-40c6-aef5-be4d4adcd2b7","Type":"ContainerStarted","Data":"baf867a59ade0462a5d8961f87e7ce68cd6bfe165e7ddaa4e4cf3029f5200dc5"}
Jul 17 21:47:17.227269 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.227235    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq"]
Jul 17 21:47:17.227696 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.227672    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed" containerName="revoke-keys"
Jul 17 21:47:17.227785 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.227699    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed" containerName="revoke-keys"
Jul 17 21:47:17.227785 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.227711    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed" containerName="revoke-keys"
Jul 17 21:47:17.227785 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.227719    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed" containerName="revoke-keys"
Jul 17 21:47:17.227785 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.227740    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed" containerName="revoke-keys"
Jul 17 21:47:17.227785 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.227751    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed" containerName="revoke-keys"
Jul 17 21:47:17.228047 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.227835    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed" containerName="revoke-keys"
Jul 17 21:47:17.228047 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.227849    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed" containerName="revoke-keys"
Jul 17 21:47:17.231047 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.231020    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq"
Jul 17 21:47:17.238956 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.238922    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq"]
Jul 17 21:47:17.356374 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.356343    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf17eed-ac43-40df-8767-6b793e8d86b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq\" (UID: \"acf17eed-ac43-40df-8767-6b793e8d86b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq"
Jul 17 21:47:17.417518 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.417492    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:47:17.418575 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.418553    2573 patch_prober.go:28] interesting pod/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Jul 17 21:47:17.418675 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.418598    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j" podUID="5b24183d-ab11-40c6-aef5-be4d4adcd2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Jul 17 21:47:17.457014 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.456988    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf17eed-ac43-40df-8767-6b793e8d86b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq\" (UID: \"acf17eed-ac43-40df-8767-6b793e8d86b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq"
Jul 17 21:47:17.457700 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.457684    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf17eed-ac43-40df-8767-6b793e8d86b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq\" (UID: \"acf17eed-ac43-40df-8767-6b793e8d86b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq"
Jul 17 21:47:17.548699 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.548628    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq"
Jul 17 21:47:17.700208 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:17.700184    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq"]
Jul 17 21:47:17.702139 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:47:17.702107    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf17eed_ac43_40df_8767_6b793e8d86b4.slice/crio-8e96df4f9a9d48ecf0e4ecd7c98c0536f45c4f2dce5d15094fbb2ff24e8856ff WatchSource:0}: Error finding container 8e96df4f9a9d48ecf0e4ecd7c98c0536f45c4f2dce5d15094fbb2ff24e8856ff: Status 404 returned error can't find the container with id 8e96df4f9a9d48ecf0e4ecd7c98c0536f45c4f2dce5d15094fbb2ff24e8856ff
Jul 17 21:47:18.417610 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:18.417575    2573 patch_prober.go:28] interesting pod/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Jul 17 21:47:18.417778 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:18.417631    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j" podUID="5b24183d-ab11-40c6-aef5-be4d4adcd2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Jul 17 21:47:18.610709 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:18.610679    2573 generic.go:356] "Generic (PLEG): container finished" podID="acf17eed-ac43-40df-8767-6b793e8d86b4" containerID="a9f78d34eb4096c2d5d55aca56b4dba39e7027a6d8fb0d6dcc626f7b8ad7f90c" exitCode=6
Jul 17 21:47:18.610812 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:18.610764    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq" event={"ID":"acf17eed-ac43-40df-8767-6b793e8d86b4","Type":"ContainerDied","Data":"a9f78d34eb4096c2d5d55aca56b4dba39e7027a6d8fb0d6dcc626f7b8ad7f90c"}
Jul 17 21:47:18.610812 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:18.610809    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq" event={"ID":"acf17eed-ac43-40df-8767-6b793e8d86b4","Type":"ContainerStarted","Data":"8e96df4f9a9d48ecf0e4ecd7c98c0536f45c4f2dce5d15094fbb2ff24e8856ff"}
Jul 17 21:47:18.610935 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:18.610909    2573 scope.go:117] "RemoveContainer" containerID="a9f78d34eb4096c2d5d55aca56b4dba39e7027a6d8fb0d6dcc626f7b8ad7f90c"
Jul 17 21:47:19.417501 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:19.417467    2573 patch_prober.go:28] interesting pod/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Jul 17 21:47:19.417893 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:19.417527    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j" podUID="5b24183d-ab11-40c6-aef5-be4d4adcd2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Jul 17 21:47:19.525497 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:19.525469    2573 scope.go:117] "RemoveContainer" containerID="49c5347cbe41746ae5a330825951e107cc1c0a83d071b8087645ca3cb8c4e53f"
Jul 17 21:47:19.616124 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:19.616098    2573 generic.go:356] "Generic (PLEG): container finished" podID="acf17eed-ac43-40df-8767-6b793e8d86b4" containerID="b276ea6f02ace65c8f6c316883aea3db63bb1c2261a7dda6eeae6949c3827b35" exitCode=6
Jul 17 21:47:19.616236 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:19.616180    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq" event={"ID":"acf17eed-ac43-40df-8767-6b793e8d86b4","Type":"ContainerDied","Data":"b276ea6f02ace65c8f6c316883aea3db63bb1c2261a7dda6eeae6949c3827b35"}
Jul 17 21:47:19.616333 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:19.616248    2573 scope.go:117] "RemoveContainer" containerID="a9f78d34eb4096c2d5d55aca56b4dba39e7027a6d8fb0d6dcc626f7b8ad7f90c"
Jul 17 21:47:19.616435 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:19.616420    2573 scope.go:117] "RemoveContainer" containerID="b276ea6f02ace65c8f6c316883aea3db63bb1c2261a7dda6eeae6949c3827b35"
Jul 17 21:47:19.616673 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:47:19.616655    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq_odh-ai-gateway-infra(acf17eed-ac43-40df-8767-6b793e8d86b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq" podUID="acf17eed-ac43-40df-8767-6b793e8d86b4"
Jul 17 21:47:20.282007 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:20.281978    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb"]
Jul 17 21:47:20.282353 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:20.282338    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="005e1ac5-f99a-4d7d-bdcf-cd7e61a03aed" containerName="revoke-keys"
Jul 17 21:47:20.285199 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:20.285184    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb"
Jul 17 21:47:20.291830 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:20.291809    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb"]
Jul 17 21:47:20.383405 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:20.383367    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377b140-6c59-4f9b-beea-45d2f2ae7e87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-8wtfb\" (UID: \"4377b140-6c59-4f9b-beea-45d2f2ae7e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb"
Jul 17 21:47:20.421798 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:20.421773    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:47:20.422163 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:20.422007    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:47:20.422665 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:20.422648    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:47:20.484773 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:20.484739    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377b140-6c59-4f9b-beea-45d2f2ae7e87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-8wtfb\" (UID: \"4377b140-6c59-4f9b-beea-45d2f2ae7e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb"
Jul 17 21:47:20.485428 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:20.485398    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377b140-6c59-4f9b-beea-45d2f2ae7e87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-8wtfb\" (UID: \"4377b140-6c59-4f9b-beea-45d2f2ae7e87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb"
Jul 17 21:47:20.597218 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:20.597191    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb"
Jul 17 21:47:20.620701 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:20.620677    2573 scope.go:117] "RemoveContainer" containerID="7bcc8fb641c46b6dd0445d9467dcc36257e8dd703451d7d45b4a937250636b53"
Jul 17 21:47:20.623244 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:20.623219    2573 generic.go:356] "Generic (PLEG): container finished" podID="ed883b47-6eb9-4cd8-832b-50148625b581" containerID="a8384f7961291724bb7f56dadce092ca3a818cc347b86c3f5901402e4cc3c1fd" exitCode=7
Jul 17 21:47:20.623362 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:20.623314    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb" event={"ID":"ed883b47-6eb9-4cd8-832b-50148625b581","Type":"ContainerDied","Data":"a8384f7961291724bb7f56dadce092ca3a818cc347b86c3f5901402e4cc3c1fd"}
Jul 17 21:47:20.623434 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:20.623389    2573 scope.go:117] "RemoveContainer" containerID="49c5347cbe41746ae5a330825951e107cc1c0a83d071b8087645ca3cb8c4e53f"
Jul 17 21:47:20.623719 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:20.623699    2573 scope.go:117] "RemoveContainer" containerID="a8384f7961291724bb7f56dadce092ca3a818cc347b86c3f5901402e4cc3c1fd"
Jul 17 21:47:20.623930 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:47:20.623907    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb_odh-ai-gateway-infra(ed883b47-6eb9-4cd8-832b-50148625b581)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb" podUID="ed883b47-6eb9-4cd8-832b-50148625b581"
Jul 17 21:47:20.625672 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:20.625654    2573 scope.go:117] "RemoveContainer" containerID="b276ea6f02ace65c8f6c316883aea3db63bb1c2261a7dda6eeae6949c3827b35"
Jul 17 21:47:20.625834 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:47:20.625816    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq_odh-ai-gateway-infra(acf17eed-ac43-40df-8767-6b793e8d86b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq" podUID="acf17eed-ac43-40df-8767-6b793e8d86b4"
Jul 17 21:47:20.726394 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:20.726372    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb"]
Jul 17 21:47:20.729028 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:47:20.729002    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4377b140_6c59_4f9b_beea_45d2f2ae7e87.slice/crio-38720c8dd74b0ec59d2c256f7617a4e7e6ea9fbf4bb5cb084b174d70e7e7f441 WatchSource:0}: Error finding container 38720c8dd74b0ec59d2c256f7617a4e7e6ea9fbf4bb5cb084b174d70e7e7f441: Status 404 returned error can't find the container with id 38720c8dd74b0ec59d2c256f7617a4e7e6ea9fbf4bb5cb084b174d70e7e7f441
Jul 17 21:47:21.634809 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:21.634779    2573 generic.go:356] "Generic (PLEG): container finished" podID="4377b140-6c59-4f9b-beea-45d2f2ae7e87" containerID="2d2d1d78858022c6bff0700a690b97641e9e38c66e152e4106e24de85a4ee2cc" exitCode=6
Jul 17 21:47:21.635256 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:21.634813    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb" event={"ID":"4377b140-6c59-4f9b-beea-45d2f2ae7e87","Type":"ContainerDied","Data":"2d2d1d78858022c6bff0700a690b97641e9e38c66e152e4106e24de85a4ee2cc"}
Jul 17 21:47:21.635256 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:21.634918    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb" event={"ID":"4377b140-6c59-4f9b-beea-45d2f2ae7e87","Type":"ContainerStarted","Data":"38720c8dd74b0ec59d2c256f7617a4e7e6ea9fbf4bb5cb084b174d70e7e7f441"}
Jul 17 21:47:21.635256 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:21.635135    2573 scope.go:117] "RemoveContainer" containerID="2d2d1d78858022c6bff0700a690b97641e9e38c66e152e4106e24de85a4ee2cc"
Jul 17 21:47:21.645951 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:21.645934    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb"]
Jul 17 21:47:21.773196 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:21.773171    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb"
Jul 17 21:47:21.894202 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:21.894152    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed883b47-6eb9-4cd8-832b-50148625b581-maas-api-service-ca\") pod \"ed883b47-6eb9-4cd8-832b-50148625b581\" (UID: \"ed883b47-6eb9-4cd8-832b-50148625b581\") "
Jul 17 21:47:21.894501 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:21.894479    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed883b47-6eb9-4cd8-832b-50148625b581-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ed883b47-6eb9-4cd8-832b-50148625b581" (UID: "ed883b47-6eb9-4cd8-832b-50148625b581"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:47:21.995692 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:21.995666    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed883b47-6eb9-4cd8-832b-50148625b581-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:47:22.639575 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:22.639550    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb"
Jul 17 21:47:22.639575 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:22.639558    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb" event={"ID":"ed883b47-6eb9-4cd8-832b-50148625b581","Type":"ContainerDied","Data":"d99f928214298577dd435c66630e06945790ad522d964efc18fcfd5dab7bfd4f"}
Jul 17 21:47:22.640040 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:22.639610    2573 scope.go:117] "RemoveContainer" containerID="a8384f7961291724bb7f56dadce092ca3a818cc347b86c3f5901402e4cc3c1fd"
Jul 17 21:47:22.641847 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:22.641825    2573 generic.go:356] "Generic (PLEG): container finished" podID="4377b140-6c59-4f9b-beea-45d2f2ae7e87" containerID="b2a4a4c3e6c0899e4ef10354b7663987b864906da087ff220da73829abf99f43" exitCode=6
Jul 17 21:47:22.641945 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:22.641869    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb" event={"ID":"4377b140-6c59-4f9b-beea-45d2f2ae7e87","Type":"ContainerDied","Data":"b2a4a4c3e6c0899e4ef10354b7663987b864906da087ff220da73829abf99f43"}
Jul 17 21:47:22.642087 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:22.642073    2573 scope.go:117] "RemoveContainer" containerID="b2a4a4c3e6c0899e4ef10354b7663987b864906da087ff220da73829abf99f43"
Jul 17 21:47:22.642246 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:47:22.642230    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-8wtfb_odh-ai-gateway-infra(4377b140-6c59-4f9b-beea-45d2f2ae7e87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb" podUID="4377b140-6c59-4f9b-beea-45d2f2ae7e87"
Jul 17 21:47:22.647513 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:22.647497    2573 scope.go:117] "RemoveContainer" containerID="2d2d1d78858022c6bff0700a690b97641e9e38c66e152e4106e24de85a4ee2cc"
Jul 17 21:47:22.654311 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:22.654270    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb"]
Jul 17 21:47:22.659736 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:22.659714    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-2vhhb"]
Jul 17 21:47:23.647533 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:23.647500    2573 scope.go:117] "RemoveContainer" containerID="b2a4a4c3e6c0899e4ef10354b7663987b864906da087ff220da73829abf99f43"
Jul 17 21:47:23.647936 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:47:23.647682    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-8wtfb_odh-ai-gateway-infra(4377b140-6c59-4f9b-beea-45d2f2ae7e87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb" podUID="4377b140-6c59-4f9b-beea-45d2f2ae7e87"
Jul 17 21:47:24.530110 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:24.530079    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed883b47-6eb9-4cd8-832b-50148625b581" path="/var/lib/kubelet/pods/ed883b47-6eb9-4cd8-832b-50148625b581/volumes"
Jul 17 21:47:34.524456 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:34.524419    2573 scope.go:117] "RemoveContainer" containerID="b276ea6f02ace65c8f6c316883aea3db63bb1c2261a7dda6eeae6949c3827b35"
Jul 17 21:47:34.525000 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:34.524651    2573 scope.go:117] "RemoveContainer" containerID="b2a4a4c3e6c0899e4ef10354b7663987b864906da087ff220da73829abf99f43"
Jul 17 21:47:35.697754 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:35.697721    2573 generic.go:356] "Generic (PLEG): container finished" podID="4377b140-6c59-4f9b-beea-45d2f2ae7e87" containerID="aaba716789e8e41a7cdd076f89160edd12d60169700939702b81e34f2ea64350" exitCode=6
Jul 17 21:47:35.698204 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:35.697795    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb" event={"ID":"4377b140-6c59-4f9b-beea-45d2f2ae7e87","Type":"ContainerDied","Data":"aaba716789e8e41a7cdd076f89160edd12d60169700939702b81e34f2ea64350"}
Jul 17 21:47:35.698204 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:35.697862    2573 scope.go:117] "RemoveContainer" containerID="b2a4a4c3e6c0899e4ef10354b7663987b864906da087ff220da73829abf99f43"
Jul 17 21:47:35.698204 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:35.698171    2573 scope.go:117] "RemoveContainer" containerID="aaba716789e8e41a7cdd076f89160edd12d60169700939702b81e34f2ea64350"
Jul 17 21:47:35.698444 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:47:35.698415    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-8wtfb_odh-ai-gateway-infra(4377b140-6c59-4f9b-beea-45d2f2ae7e87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb" podUID="4377b140-6c59-4f9b-beea-45d2f2ae7e87"
Jul 17 21:47:35.699867 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:35.699847    2573 generic.go:356] "Generic (PLEG): container finished" podID="acf17eed-ac43-40df-8767-6b793e8d86b4" containerID="0bea8f91e9a9eaa446c158c796245b6991936a2e49b4fd7497b04cd034b771e1" exitCode=6
Jul 17 21:47:35.699963 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:35.699875    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq" event={"ID":"acf17eed-ac43-40df-8767-6b793e8d86b4","Type":"ContainerDied","Data":"0bea8f91e9a9eaa446c158c796245b6991936a2e49b4fd7497b04cd034b771e1"}
Jul 17 21:47:35.700108 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:35.700096    2573 scope.go:117] "RemoveContainer" containerID="0bea8f91e9a9eaa446c158c796245b6991936a2e49b4fd7497b04cd034b771e1"
Jul 17 21:47:35.700267 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:47:35.700252    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq_odh-ai-gateway-infra(acf17eed-ac43-40df-8767-6b793e8d86b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq" podUID="acf17eed-ac43-40df-8767-6b793e8d86b4"
Jul 17 21:47:35.708743 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:35.708726    2573 scope.go:117] "RemoveContainer" containerID="b276ea6f02ace65c8f6c316883aea3db63bb1c2261a7dda6eeae6949c3827b35"
Jul 17 21:47:36.722811 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:36.722775    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb"]
Jul 17 21:47:36.735655 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:36.735628    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq"]
Jul 17 21:47:36.864125 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:36.864100    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb"
Jul 17 21:47:36.875709 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:36.875688    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq"
Jul 17 21:47:36.912113 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:36.912091    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377b140-6c59-4f9b-beea-45d2f2ae7e87-maas-api-service-ca\") pod \"4377b140-6c59-4f9b-beea-45d2f2ae7e87\" (UID: \"4377b140-6c59-4f9b-beea-45d2f2ae7e87\") "
Jul 17 21:47:36.912261 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:36.912154    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf17eed-ac43-40df-8767-6b793e8d86b4-maas-api-service-ca\") pod \"acf17eed-ac43-40df-8767-6b793e8d86b4\" (UID: \"acf17eed-ac43-40df-8767-6b793e8d86b4\") "
Jul 17 21:47:36.912446 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:36.912423    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4377b140-6c59-4f9b-beea-45d2f2ae7e87-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4377b140-6c59-4f9b-beea-45d2f2ae7e87" (UID: "4377b140-6c59-4f9b-beea-45d2f2ae7e87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:47:36.912513 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:36.912492    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf17eed-ac43-40df-8767-6b793e8d86b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "acf17eed-ac43-40df-8767-6b793e8d86b4" (UID: "acf17eed-ac43-40df-8767-6b793e8d86b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:47:37.012768 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:37.012695    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377b140-6c59-4f9b-beea-45d2f2ae7e87-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:47:37.012768 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:37.012718    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf17eed-ac43-40df-8767-6b793e8d86b4-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:47:37.712359 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:37.712321    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb" event={"ID":"4377b140-6c59-4f9b-beea-45d2f2ae7e87","Type":"ContainerDied","Data":"38720c8dd74b0ec59d2c256f7617a4e7e6ea9fbf4bb5cb084b174d70e7e7f441"}
Jul 17 21:47:37.712359 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:37.712342    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb"
Jul 17 21:47:37.712600 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:37.712386    2573 scope.go:117] "RemoveContainer" containerID="aaba716789e8e41a7cdd076f89160edd12d60169700939702b81e34f2ea64350"
Jul 17 21:47:37.713835 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:37.713811    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq" event={"ID":"acf17eed-ac43-40df-8767-6b793e8d86b4","Type":"ContainerDied","Data":"8e96df4f9a9d48ecf0e4ecd7c98c0536f45c4f2dce5d15094fbb2ff24e8856ff"}
Jul 17 21:47:37.713963 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:37.713843    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq"
Jul 17 21:47:37.722891 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:37.722861    2573 scope.go:117] "RemoveContainer" containerID="0bea8f91e9a9eaa446c158c796245b6991936a2e49b4fd7497b04cd034b771e1"
Jul 17 21:47:37.740560 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:37.740540    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb"]
Jul 17 21:47:37.747228 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:37.747203    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-8wtfb"]
Jul 17 21:47:37.760036 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:37.760018    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq"]
Jul 17 21:47:37.763969 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:37.763947    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5pzxq"]
Jul 17 21:47:38.529344 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:38.529312    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4377b140-6c59-4f9b-beea-45d2f2ae7e87" path="/var/lib/kubelet/pods/4377b140-6c59-4f9b-beea-45d2f2ae7e87/volumes"
Jul 17 21:47:38.529642 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:38.529629    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acf17eed-ac43-40df-8767-6b793e8d86b4" path="/var/lib/kubelet/pods/acf17eed-ac43-40df-8767-6b793e8d86b4/volumes"
Jul 17 21:47:54.695061 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695028    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf"]
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695308    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf17eed-ac43-40df-8767-6b793e8d86b4" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695320    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf17eed-ac43-40df-8767-6b793e8d86b4" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695330    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf17eed-ac43-40df-8767-6b793e8d86b4" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695336    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf17eed-ac43-40df-8767-6b793e8d86b4" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695343    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed883b47-6eb9-4cd8-832b-50148625b581" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695349    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed883b47-6eb9-4cd8-832b-50148625b581" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695355    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed883b47-6eb9-4cd8-832b-50148625b581" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695360    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed883b47-6eb9-4cd8-832b-50148625b581" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695366    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4377b140-6c59-4f9b-beea-45d2f2ae7e87" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695371    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4377b140-6c59-4f9b-beea-45d2f2ae7e87" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695377    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4377b140-6c59-4f9b-beea-45d2f2ae7e87" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695381    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4377b140-6c59-4f9b-beea-45d2f2ae7e87" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695440    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4377b140-6c59-4f9b-beea-45d2f2ae7e87" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695448    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4377b140-6c59-4f9b-beea-45d2f2ae7e87" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695454    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="acf17eed-ac43-40df-8767-6b793e8d86b4" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695460    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed883b47-6eb9-4cd8-832b-50148625b581" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695467    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed883b47-6eb9-4cd8-832b-50148625b581" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695473    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed883b47-6eb9-4cd8-832b-50148625b581" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695479    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="acf17eed-ac43-40df-8767-6b793e8d86b4" containerName="revoke-keys"
Jul 17 21:47:54.695569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.695485    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="acf17eed-ac43-40df-8767-6b793e8d86b4" containerName="revoke-keys"
Jul 17 21:47:54.699793 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.699776    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf"
Jul 17 21:47:54.702124 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.702099    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:47:54.702238 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.702183    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:47:54.705179 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.705158    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf"]
Jul 17 21:47:54.739853 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.739822    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14623272-5824-4ac9-b5b7-acbccbe991f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf\" (UID: \"14623272-5824-4ac9-b5b7-acbccbe991f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf"
Jul 17 21:47:54.841092 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.841062    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14623272-5824-4ac9-b5b7-acbccbe991f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf\" (UID: \"14623272-5824-4ac9-b5b7-acbccbe991f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf"
Jul 17 21:47:54.841650 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:54.841631    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14623272-5824-4ac9-b5b7-acbccbe991f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf\" (UID: \"14623272-5824-4ac9-b5b7-acbccbe991f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf"
Jul 17 21:47:55.011304 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:55.011201    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf"
Jul 17 21:47:55.133775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:55.133683    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf"]
Jul 17 21:47:55.136390 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:47:55.136363    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14623272_5824_4ac9_b5b7_acbccbe991f4.slice/crio-f6ba11ed5789c1916f48152f069f2f1380795048ae33616ba14b377f9410c911 WatchSource:0}: Error finding container f6ba11ed5789c1916f48152f069f2f1380795048ae33616ba14b377f9410c911: Status 404 returned error can't find the container with id f6ba11ed5789c1916f48152f069f2f1380795048ae33616ba14b377f9410c911
Jul 17 21:47:55.788833 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:55.788800    2573 generic.go:356] "Generic (PLEG): container finished" podID="14623272-5824-4ac9-b5b7-acbccbe991f4" containerID="54de2a3e21842137058d686e9a17790ca074317d5608df8cea2aeaa8b7b0c54d" exitCode=7
Jul 17 21:47:55.789147 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:55.788851    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf" event={"ID":"14623272-5824-4ac9-b5b7-acbccbe991f4","Type":"ContainerDied","Data":"54de2a3e21842137058d686e9a17790ca074317d5608df8cea2aeaa8b7b0c54d"}
Jul 17 21:47:55.789147 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:55.788932    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf" event={"ID":"14623272-5824-4ac9-b5b7-acbccbe991f4","Type":"ContainerStarted","Data":"f6ba11ed5789c1916f48152f069f2f1380795048ae33616ba14b377f9410c911"}
Jul 17 21:47:55.789147 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:55.789141    2573 scope.go:117] "RemoveContainer" containerID="54de2a3e21842137058d686e9a17790ca074317d5608df8cea2aeaa8b7b0c54d"
Jul 17 21:47:56.794269 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:56.794237    2573 generic.go:356] "Generic (PLEG): container finished" podID="14623272-5824-4ac9-b5b7-acbccbe991f4" containerID="7904b643a59ddbc04a420f4bcff0de113d538b7eadc09345714aa5b296282f21" exitCode=7
Jul 17 21:47:56.794720 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:56.794327    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf" event={"ID":"14623272-5824-4ac9-b5b7-acbccbe991f4","Type":"ContainerDied","Data":"7904b643a59ddbc04a420f4bcff0de113d538b7eadc09345714aa5b296282f21"}
Jul 17 21:47:56.794720 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:56.794390    2573 scope.go:117] "RemoveContainer" containerID="54de2a3e21842137058d686e9a17790ca074317d5608df8cea2aeaa8b7b0c54d"
Jul 17 21:47:56.794720 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:56.794602    2573 scope.go:117] "RemoveContainer" containerID="7904b643a59ddbc04a420f4bcff0de113d538b7eadc09345714aa5b296282f21"
Jul 17 21:47:56.794871 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:47:56.794819    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf_odh-ai-gateway-infra(14623272-5824-4ac9-b5b7-acbccbe991f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf" podUID="14623272-5824-4ac9-b5b7-acbccbe991f4"
Jul 17 21:47:57.800176 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:47:57.800148    2573 scope.go:117] "RemoveContainer" containerID="7904b643a59ddbc04a420f4bcff0de113d538b7eadc09345714aa5b296282f21"
Jul 17 21:47:57.800546 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:47:57.800340    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf_odh-ai-gateway-infra(14623272-5824-4ac9-b5b7-acbccbe991f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf" podUID="14623272-5824-4ac9-b5b7-acbccbe991f4"
Jul 17 21:48:08.525462 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:08.525426    2573 scope.go:117] "RemoveContainer" containerID="7904b643a59ddbc04a420f4bcff0de113d538b7eadc09345714aa5b296282f21"
Jul 17 21:48:08.845701 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:08.845672    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf" event={"ID":"14623272-5824-4ac9-b5b7-acbccbe991f4","Type":"ContainerStarted","Data":"5449193cdc252c95d40eb399d191924e97b04a84c0b7c4455fd2c1a422374248"}
Jul 17 21:48:08.871151 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:08.871105    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf" podStartSLOduration=14.871091451 podStartE2EDuration="14.871091451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:48:08.871016494 +0000 UTC m=+2388.835137127" watchObservedRunningTime="2026-07-17 21:48:08.871091451 +0000 UTC m=+2388.835212062"
Jul 17 21:48:09.852189 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:09.852155    2573 generic.go:356] "Generic (PLEG): container finished" podID="14623272-5824-4ac9-b5b7-acbccbe991f4" containerID="5449193cdc252c95d40eb399d191924e97b04a84c0b7c4455fd2c1a422374248" exitCode=7
Jul 17 21:48:09.852605 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:09.852199    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf" event={"ID":"14623272-5824-4ac9-b5b7-acbccbe991f4","Type":"ContainerDied","Data":"5449193cdc252c95d40eb399d191924e97b04a84c0b7c4455fd2c1a422374248"}
Jul 17 21:48:09.852605 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:09.852253    2573 scope.go:117] "RemoveContainer" containerID="7904b643a59ddbc04a420f4bcff0de113d538b7eadc09345714aa5b296282f21"
Jul 17 21:48:09.852605 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:09.852529    2573 scope.go:117] "RemoveContainer" containerID="5449193cdc252c95d40eb399d191924e97b04a84c0b7c4455fd2c1a422374248"
Jul 17 21:48:09.852794 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:48:09.852746    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf_odh-ai-gateway-infra(14623272-5824-4ac9-b5b7-acbccbe991f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf" podUID="14623272-5824-4ac9-b5b7-acbccbe991f4"
Jul 17 21:48:09.882341 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:09.882314    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf"]
Jul 17 21:48:10.985118 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:10.985081    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf"
Jul 17 21:48:11.065455 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:11.065421    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14623272-5824-4ac9-b5b7-acbccbe991f4-maas-api-service-ca\") pod \"14623272-5824-4ac9-b5b7-acbccbe991f4\" (UID: \"14623272-5824-4ac9-b5b7-acbccbe991f4\") "
Jul 17 21:48:11.065796 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:11.065774    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14623272-5824-4ac9-b5b7-acbccbe991f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "14623272-5824-4ac9-b5b7-acbccbe991f4" (UID: "14623272-5824-4ac9-b5b7-acbccbe991f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:48:11.166331 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:11.166239    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14623272-5824-4ac9-b5b7-acbccbe991f4-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:48:11.863817 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:11.863783    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf" event={"ID":"14623272-5824-4ac9-b5b7-acbccbe991f4","Type":"ContainerDied","Data":"f6ba11ed5789c1916f48152f069f2f1380795048ae33616ba14b377f9410c911"}
Jul 17 21:48:11.863964 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:11.863838    2573 scope.go:117] "RemoveContainer" containerID="5449193cdc252c95d40eb399d191924e97b04a84c0b7c4455fd2c1a422374248"
Jul 17 21:48:11.863964 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:11.863796    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf"
Jul 17 21:48:11.885557 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:11.885534    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf"]
Jul 17 21:48:11.889982 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:11.889963    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-jdwjf"]
Jul 17 21:48:12.529273 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:12.529240    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14623272-5824-4ac9-b5b7-acbccbe991f4" path="/var/lib/kubelet/pods/14623272-5824-4ac9-b5b7-acbccbe991f4/volumes"
Jul 17 21:48:17.294404 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.294368    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb"]
Jul 17 21:48:17.294790 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.294678    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14623272-5824-4ac9-b5b7-acbccbe991f4" containerName="revoke-keys"
Jul 17 21:48:17.294790 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.294691    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="14623272-5824-4ac9-b5b7-acbccbe991f4" containerName="revoke-keys"
Jul 17 21:48:17.294790 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.294704    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14623272-5824-4ac9-b5b7-acbccbe991f4" containerName="revoke-keys"
Jul 17 21:48:17.294790 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.294711    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="14623272-5824-4ac9-b5b7-acbccbe991f4" containerName="revoke-keys"
Jul 17 21:48:17.294790 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.294725    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14623272-5824-4ac9-b5b7-acbccbe991f4" containerName="revoke-keys"
Jul 17 21:48:17.294790 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.294731    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="14623272-5824-4ac9-b5b7-acbccbe991f4" containerName="revoke-keys"
Jul 17 21:48:17.294790 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.294742    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf17eed-ac43-40df-8767-6b793e8d86b4" containerName="revoke-keys"
Jul 17 21:48:17.294790 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.294748    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf17eed-ac43-40df-8767-6b793e8d86b4" containerName="revoke-keys"
Jul 17 21:48:17.294790 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.294755    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed883b47-6eb9-4cd8-832b-50148625b581" containerName="revoke-keys"
Jul 17 21:48:17.294790 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.294761    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed883b47-6eb9-4cd8-832b-50148625b581" containerName="revoke-keys"
Jul 17 21:48:17.294790 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.294767    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4377b140-6c59-4f9b-beea-45d2f2ae7e87" containerName="revoke-keys"
Jul 17 21:48:17.294790 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.294772    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4377b140-6c59-4f9b-beea-45d2f2ae7e87" containerName="revoke-keys"
Jul 17 21:48:17.295136 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.294818    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="14623272-5824-4ac9-b5b7-acbccbe991f4" containerName="revoke-keys"
Jul 17 21:48:17.295136 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.294828    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="14623272-5824-4ac9-b5b7-acbccbe991f4" containerName="revoke-keys"
Jul 17 21:48:17.295136 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.294836    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4377b140-6c59-4f9b-beea-45d2f2ae7e87" containerName="revoke-keys"
Jul 17 21:48:17.298867 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.298851    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb"
Jul 17 21:48:17.301280 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.301261    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:48:17.301280 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.301273    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:48:17.305317 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.305247    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb"]
Jul 17 21:48:17.415514 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.415474    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a4ffcd-cb66-450b-b1db-ba334187b5f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb\" (UID: \"b6a4ffcd-cb66-450b-b1db-ba334187b5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb"
Jul 17 21:48:17.516540 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.516509    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a4ffcd-cb66-450b-b1db-ba334187b5f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb\" (UID: \"b6a4ffcd-cb66-450b-b1db-ba334187b5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb"
Jul 17 21:48:17.517072 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.517053    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a4ffcd-cb66-450b-b1db-ba334187b5f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb\" (UID: \"b6a4ffcd-cb66-450b-b1db-ba334187b5f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb"
Jul 17 21:48:17.610081 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.610059    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb"
Jul 17 21:48:17.735782 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.735751    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb"]
Jul 17 21:48:17.738088 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:48:17.738048    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a4ffcd_cb66_450b_b1db_ba334187b5f0.slice/crio-0613e817a46e0f5cfa9578bd7331e79246e1cad4baa0786aaec00fdc5251f4a1 WatchSource:0}: Error finding container 0613e817a46e0f5cfa9578bd7331e79246e1cad4baa0786aaec00fdc5251f4a1: Status 404 returned error can't find the container with id 0613e817a46e0f5cfa9578bd7331e79246e1cad4baa0786aaec00fdc5251f4a1
Jul 17 21:48:17.894356 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:17.894299    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb" event={"ID":"b6a4ffcd-cb66-450b-b1db-ba334187b5f0","Type":"ContainerStarted","Data":"0613e817a46e0f5cfa9578bd7331e79246e1cad4baa0786aaec00fdc5251f4a1"}
Jul 17 21:48:18.900381 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:18.900341    2573 generic.go:356] "Generic (PLEG): container finished" podID="b6a4ffcd-cb66-450b-b1db-ba334187b5f0" containerID="1e665cb75554419c1d6abcef8f5d7bab3a98b24cec7ea7e5921d035091136f88" exitCode=6
Jul 17 21:48:18.900760 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:18.900429    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb" event={"ID":"b6a4ffcd-cb66-450b-b1db-ba334187b5f0","Type":"ContainerDied","Data":"1e665cb75554419c1d6abcef8f5d7bab3a98b24cec7ea7e5921d035091136f88"}
Jul 17 21:48:18.900760 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:18.900566    2573 scope.go:117] "RemoveContainer" containerID="1e665cb75554419c1d6abcef8f5d7bab3a98b24cec7ea7e5921d035091136f88"
Jul 17 21:48:19.906234 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:19.906203    2573 generic.go:356] "Generic (PLEG): container finished" podID="b6a4ffcd-cb66-450b-b1db-ba334187b5f0" containerID="d13e17886ab4347765b88d6563c5cb64190056c8006363e32d774199d6345859" exitCode=6
Jul 17 21:48:19.906634 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:19.906250    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb" event={"ID":"b6a4ffcd-cb66-450b-b1db-ba334187b5f0","Type":"ContainerDied","Data":"d13e17886ab4347765b88d6563c5cb64190056c8006363e32d774199d6345859"}
Jul 17 21:48:19.906634 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:19.906331    2573 scope.go:117] "RemoveContainer" containerID="1e665cb75554419c1d6abcef8f5d7bab3a98b24cec7ea7e5921d035091136f88"
Jul 17 21:48:19.906634 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:19.906504    2573 scope.go:117] "RemoveContainer" containerID="d13e17886ab4347765b88d6563c5cb64190056c8006363e32d774199d6345859"
Jul 17 21:48:19.906756 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:48:19.906697    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb_odh-ai-gateway-infra(b6a4ffcd-cb66-450b-b1db-ba334187b5f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb" podUID="b6a4ffcd-cb66-450b-b1db-ba334187b5f0"
Jul 17 21:48:20.912732 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:20.912663    2573 scope.go:117] "RemoveContainer" containerID="d13e17886ab4347765b88d6563c5cb64190056c8006363e32d774199d6345859"
Jul 17 21:48:20.913111 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:48:20.912906    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb_odh-ai-gateway-infra(b6a4ffcd-cb66-450b-b1db-ba334187b5f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb" podUID="b6a4ffcd-cb66-450b-b1db-ba334187b5f0"
Jul 17 21:48:33.525103 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:33.525076    2573 scope.go:117] "RemoveContainer" containerID="d13e17886ab4347765b88d6563c5cb64190056c8006363e32d774199d6345859"
Jul 17 21:48:33.964341 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:33.964307    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb" event={"ID":"b6a4ffcd-cb66-450b-b1db-ba334187b5f0","Type":"ContainerStarted","Data":"2c71d5ef37145b05d1ace07f63a2dd4283f3cabbdb9e8e04901f940b4bcf1ff3"}
Jul 17 21:48:33.981175 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:33.981127    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb" podStartSLOduration=16.981112587 podStartE2EDuration="16.981112587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:48:33.979001507 +0000 UTC m=+2413.943122116" watchObservedRunningTime="2026-07-17 21:48:33.981112587 +0000 UTC m=+2413.945233197"
Jul 17 21:48:34.970342 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:34.970307    2573 generic.go:356] "Generic (PLEG): container finished" podID="b6a4ffcd-cb66-450b-b1db-ba334187b5f0" containerID="2c71d5ef37145b05d1ace07f63a2dd4283f3cabbdb9e8e04901f940b4bcf1ff3" exitCode=6
Jul 17 21:48:34.970342 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:34.970318    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb" event={"ID":"b6a4ffcd-cb66-450b-b1db-ba334187b5f0","Type":"ContainerDied","Data":"2c71d5ef37145b05d1ace07f63a2dd4283f3cabbdb9e8e04901f940b4bcf1ff3"}
Jul 17 21:48:34.970785 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:34.970378    2573 scope.go:117] "RemoveContainer" containerID="d13e17886ab4347765b88d6563c5cb64190056c8006363e32d774199d6345859"
Jul 17 21:48:34.970785 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:34.970662    2573 scope.go:117] "RemoveContainer" containerID="2c71d5ef37145b05d1ace07f63a2dd4283f3cabbdb9e8e04901f940b4bcf1ff3"
Jul 17 21:48:34.970850 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:48:34.970831    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb_odh-ai-gateway-infra(b6a4ffcd-cb66-450b-b1db-ba334187b5f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb" podUID="b6a4ffcd-cb66-450b-b1db-ba334187b5f0"
Jul 17 21:48:34.993679 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:34.993659    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb"]
Jul 17 21:48:36.103163 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:36.103135    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb"
Jul 17 21:48:36.261928 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:36.261842    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a4ffcd-cb66-450b-b1db-ba334187b5f0-maas-api-service-ca\") pod \"b6a4ffcd-cb66-450b-b1db-ba334187b5f0\" (UID: \"b6a4ffcd-cb66-450b-b1db-ba334187b5f0\") "
Jul 17 21:48:36.262205 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:36.262184    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a4ffcd-cb66-450b-b1db-ba334187b5f0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b6a4ffcd-cb66-450b-b1db-ba334187b5f0" (UID: "b6a4ffcd-cb66-450b-b1db-ba334187b5f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:48:36.363071 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:36.363040    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a4ffcd-cb66-450b-b1db-ba334187b5f0-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:48:36.984957 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:36.984929    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb"
Jul 17 21:48:36.984957 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:36.984951    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb" event={"ID":"b6a4ffcd-cb66-450b-b1db-ba334187b5f0","Type":"ContainerDied","Data":"0613e817a46e0f5cfa9578bd7331e79246e1cad4baa0786aaec00fdc5251f4a1"}
Jul 17 21:48:36.985228 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:36.985026    2573 scope.go:117] "RemoveContainer" containerID="2c71d5ef37145b05d1ace07f63a2dd4283f3cabbdb9e8e04901f940b4bcf1ff3"
Jul 17 21:48:37.010810 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:37.010779    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb"]
Jul 17 21:48:37.016748 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:37.016724    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nxxxb"]
Jul 17 21:48:38.529074 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:48:38.529039    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6a4ffcd-cb66-450b-b1db-ba334187b5f0" path="/var/lib/kubelet/pods/b6a4ffcd-cb66-450b-b1db-ba334187b5f0/volumes"
Jul 17 21:49:07.332391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.332360    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6"]
Jul 17 21:49:07.332861 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.332641    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a4ffcd-cb66-450b-b1db-ba334187b5f0" containerName="revoke-keys"
Jul 17 21:49:07.332861 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.332654    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a4ffcd-cb66-450b-b1db-ba334187b5f0" containerName="revoke-keys"
Jul 17 21:49:07.332861 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.332661    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a4ffcd-cb66-450b-b1db-ba334187b5f0" containerName="revoke-keys"
Jul 17 21:49:07.332861 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.332666    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a4ffcd-cb66-450b-b1db-ba334187b5f0" containerName="revoke-keys"
Jul 17 21:49:07.332861 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.332687    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a4ffcd-cb66-450b-b1db-ba334187b5f0" containerName="revoke-keys"
Jul 17 21:49:07.332861 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.332694    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a4ffcd-cb66-450b-b1db-ba334187b5f0" containerName="revoke-keys"
Jul 17 21:49:07.332861 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.332757    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a4ffcd-cb66-450b-b1db-ba334187b5f0" containerName="revoke-keys"
Jul 17 21:49:07.332861 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.332767    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a4ffcd-cb66-450b-b1db-ba334187b5f0" containerName="revoke-keys"
Jul 17 21:49:07.332861 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.332776    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="14623272-5824-4ac9-b5b7-acbccbe991f4" containerName="revoke-keys"
Jul 17 21:49:07.335597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.335581    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6"
Jul 17 21:49:07.338312 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.338160    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:49:07.341676 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.338681    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:49:07.344802 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.344774    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6"]
Jul 17 21:49:07.392752 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.392719    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c86753-6e53-401e-b977-122a4561499c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6\" (UID: \"a2c86753-6e53-401e-b977-122a4561499c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6"
Jul 17 21:49:07.493631 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.493593    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c86753-6e53-401e-b977-122a4561499c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6\" (UID: \"a2c86753-6e53-401e-b977-122a4561499c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6"
Jul 17 21:49:07.494161 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.494143    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c86753-6e53-401e-b977-122a4561499c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6\" (UID: \"a2c86753-6e53-401e-b977-122a4561499c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6"
Jul 17 21:49:07.650472 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.650444    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6"
Jul 17 21:49:07.778022 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:07.777989    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6"]
Jul 17 21:49:07.781110 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:49:07.781089    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c86753_6e53_401e_b977_122a4561499c.slice/crio-0ff0b90a10e757fb98c0bcb71af321ddf8524c598917e5079292663e286b70be WatchSource:0}: Error finding container 0ff0b90a10e757fb98c0bcb71af321ddf8524c598917e5079292663e286b70be: Status 404 returned error can't find the container with id 0ff0b90a10e757fb98c0bcb71af321ddf8524c598917e5079292663e286b70be
Jul 17 21:49:08.107993 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:08.107954    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6" event={"ID":"a2c86753-6e53-401e-b977-122a4561499c","Type":"ContainerStarted","Data":"2f714e08f9fb82194b95575949b7ed90aa080acd745fa16422decb2aaf5ab04e"}
Jul 17 21:49:08.108150 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:08.108008    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6" event={"ID":"a2c86753-6e53-401e-b977-122a4561499c","Type":"ContainerStarted","Data":"0ff0b90a10e757fb98c0bcb71af321ddf8524c598917e5079292663e286b70be"}
Jul 17 21:49:08.125037 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:08.124990    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6" podStartSLOduration=1.124976228 podStartE2EDuration="1.124976228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:49:08.123692869 +0000 UTC m=+2448.087813478" watchObservedRunningTime="2026-07-17 21:49:08.124976228 +0000 UTC m=+2448.089096838"
Jul 17 21:49:09.113155 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:09.113108    2573 generic.go:356] "Generic (PLEG): container finished" podID="a2c86753-6e53-401e-b977-122a4561499c" containerID="2f714e08f9fb82194b95575949b7ed90aa080acd745fa16422decb2aaf5ab04e" exitCode=6
Jul 17 21:49:09.113655 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:09.113162    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6" event={"ID":"a2c86753-6e53-401e-b977-122a4561499c","Type":"ContainerDied","Data":"2f714e08f9fb82194b95575949b7ed90aa080acd745fa16422decb2aaf5ab04e"}
Jul 17 21:49:09.113655 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:09.113428    2573 scope.go:117] "RemoveContainer" containerID="2f714e08f9fb82194b95575949b7ed90aa080acd745fa16422decb2aaf5ab04e"
Jul 17 21:49:10.119298 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:10.119260    2573 generic.go:356] "Generic (PLEG): container finished" podID="a2c86753-6e53-401e-b977-122a4561499c" containerID="22e713b61cae4dfe7c63b941a0acd5264d6749d05d159c8efdc9c05269220e3a" exitCode=6
Jul 17 21:49:10.119761 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:10.119324    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6" event={"ID":"a2c86753-6e53-401e-b977-122a4561499c","Type":"ContainerDied","Data":"22e713b61cae4dfe7c63b941a0acd5264d6749d05d159c8efdc9c05269220e3a"}
Jul 17 21:49:10.119761 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:10.119404    2573 scope.go:117] "RemoveContainer" containerID="2f714e08f9fb82194b95575949b7ed90aa080acd745fa16422decb2aaf5ab04e"
Jul 17 21:49:10.119761 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:10.119570    2573 scope.go:117] "RemoveContainer" containerID="22e713b61cae4dfe7c63b941a0acd5264d6749d05d159c8efdc9c05269220e3a"
Jul 17 21:49:10.119921 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:49:10.119816    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6_odh-ai-gateway-infra(a2c86753-6e53-401e-b977-122a4561499c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6" podUID="a2c86753-6e53-401e-b977-122a4561499c"
Jul 17 21:49:11.125681 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:11.125653    2573 scope.go:117] "RemoveContainer" containerID="22e713b61cae4dfe7c63b941a0acd5264d6749d05d159c8efdc9c05269220e3a"
Jul 17 21:49:11.126045 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:49:11.125851    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6_odh-ai-gateway-infra(a2c86753-6e53-401e-b977-122a4561499c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6" podUID="a2c86753-6e53-401e-b977-122a4561499c"
Jul 17 21:49:25.525128 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:25.525098    2573 scope.go:117] "RemoveContainer" containerID="22e713b61cae4dfe7c63b941a0acd5264d6749d05d159c8efdc9c05269220e3a"
Jul 17 21:49:26.186831 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:26.186804    2573 generic.go:356] "Generic (PLEG): container finished" podID="a2c86753-6e53-401e-b977-122a4561499c" containerID="6d9a50788ee7ca3da948de95030404890b01ca782c6e6e668a47fd5b3e9935b2" exitCode=6
Jul 17 21:49:26.186923 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:26.186881    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6" event={"ID":"a2c86753-6e53-401e-b977-122a4561499c","Type":"ContainerDied","Data":"6d9a50788ee7ca3da948de95030404890b01ca782c6e6e668a47fd5b3e9935b2"}
Jul 17 21:49:26.186965 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:26.186944    2573 scope.go:117] "RemoveContainer" containerID="22e713b61cae4dfe7c63b941a0acd5264d6749d05d159c8efdc9c05269220e3a"
Jul 17 21:49:26.187214 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:26.187194    2573 scope.go:117] "RemoveContainer" containerID="6d9a50788ee7ca3da948de95030404890b01ca782c6e6e668a47fd5b3e9935b2"
Jul 17 21:49:26.187428 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:49:26.187409    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6_odh-ai-gateway-infra(a2c86753-6e53-401e-b977-122a4561499c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6" podUID="a2c86753-6e53-401e-b977-122a4561499c"
Jul 17 21:49:27.210595 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:27.210558    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6"]
Jul 17 21:49:27.337547 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:27.337524    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6"
Jul 17 21:49:27.439196 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:27.439165    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c86753-6e53-401e-b977-122a4561499c-maas-api-service-ca\") pod \"a2c86753-6e53-401e-b977-122a4561499c\" (UID: \"a2c86753-6e53-401e-b977-122a4561499c\") "
Jul 17 21:49:27.439511 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:27.439490    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c86753-6e53-401e-b977-122a4561499c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a2c86753-6e53-401e-b977-122a4561499c" (UID: "a2c86753-6e53-401e-b977-122a4561499c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:49:27.540676 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:27.540603    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c86753-6e53-401e-b977-122a4561499c-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:49:28.198588 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:28.198560    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6"
Jul 17 21:49:28.198742 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:28.198553    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6" event={"ID":"a2c86753-6e53-401e-b977-122a4561499c","Type":"ContainerDied","Data":"0ff0b90a10e757fb98c0bcb71af321ddf8524c598917e5079292663e286b70be"}
Jul 17 21:49:28.198742 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:28.198685    2573 scope.go:117] "RemoveContainer" containerID="6d9a50788ee7ca3da948de95030404890b01ca782c6e6e668a47fd5b3e9935b2"
Jul 17 21:49:28.230729 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:28.230706    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6"]
Jul 17 21:49:28.232769 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:28.232747    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dblc6"]
Jul 17 21:49:28.530112 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:28.530038    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2c86753-6e53-401e-b977-122a4561499c" path="/var/lib/kubelet/pods/a2c86753-6e53-401e-b977-122a4561499c/volumes"
Jul 17 21:49:34.789597 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:34.789560    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k"]
Jul 17 21:49:34.790048 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:34.789954    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c86753-6e53-401e-b977-122a4561499c" containerName="revoke-keys"
Jul 17 21:49:34.790048 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:34.789972    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c86753-6e53-401e-b977-122a4561499c" containerName="revoke-keys"
Jul 17 21:49:34.790048 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:34.789985    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c86753-6e53-401e-b977-122a4561499c" containerName="revoke-keys"
Jul 17 21:49:34.790048 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:34.789995    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c86753-6e53-401e-b977-122a4561499c" containerName="revoke-keys"
Jul 17 21:49:34.790244 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:34.790081    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c86753-6e53-401e-b977-122a4561499c" containerName="revoke-keys"
Jul 17 21:49:34.790244 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:34.790096    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a4ffcd-cb66-450b-b1db-ba334187b5f0" containerName="revoke-keys"
Jul 17 21:49:34.790244 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:34.790104    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c86753-6e53-401e-b977-122a4561499c" containerName="revoke-keys"
Jul 17 21:49:34.790244 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:34.790112    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c86753-6e53-401e-b977-122a4561499c" containerName="revoke-keys"
Jul 17 21:49:34.794495 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:34.794473    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k"
Jul 17 21:49:34.796964 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:34.796946    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:49:34.797057 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:34.796993    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:49:34.800658 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:34.800636    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k"]
Jul 17 21:49:34.895824 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:34.895801    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c1a62d-47b5-4b5a-9da5-766f6c658875-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k\" (UID: \"03c1a62d-47b5-4b5a-9da5-766f6c658875\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k"
Jul 17 21:49:34.996696 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:34.996666    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c1a62d-47b5-4b5a-9da5-766f6c658875-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k\" (UID: \"03c1a62d-47b5-4b5a-9da5-766f6c658875\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k"
Jul 17 21:49:34.997279 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:34.997256    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c1a62d-47b5-4b5a-9da5-766f6c658875-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k\" (UID: \"03c1a62d-47b5-4b5a-9da5-766f6c658875\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k"
Jul 17 21:49:35.105962 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:35.105932    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k"
Jul 17 21:49:35.228627 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:35.228542    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k"]
Jul 17 21:49:35.230926 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:49:35.230899    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c1a62d_47b5_4b5a_9da5_766f6c658875.slice/crio-f118273648211d1a77b56985890cb5f5b6283eed544a70688e2140d79fc6a637 WatchSource:0}: Error finding container f118273648211d1a77b56985890cb5f5b6283eed544a70688e2140d79fc6a637: Status 404 returned error can't find the container with id f118273648211d1a77b56985890cb5f5b6283eed544a70688e2140d79fc6a637
Jul 17 21:49:36.231466 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:36.231432    2573 generic.go:356] "Generic (PLEG): container finished" podID="03c1a62d-47b5-4b5a-9da5-766f6c658875" containerID="1afe0750068a45b04a47f2bec2d3826e04788c6e2d28e8437fc5e1f060449ddc" exitCode=7
Jul 17 21:49:36.231877 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:36.231513    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k" event={"ID":"03c1a62d-47b5-4b5a-9da5-766f6c658875","Type":"ContainerDied","Data":"1afe0750068a45b04a47f2bec2d3826e04788c6e2d28e8437fc5e1f060449ddc"}
Jul 17 21:49:36.231877 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:36.231564    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k" event={"ID":"03c1a62d-47b5-4b5a-9da5-766f6c658875","Type":"ContainerStarted","Data":"f118273648211d1a77b56985890cb5f5b6283eed544a70688e2140d79fc6a637"}
Jul 17 21:49:36.231877 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:36.231625    2573 scope.go:117] "RemoveContainer" containerID="1afe0750068a45b04a47f2bec2d3826e04788c6e2d28e8437fc5e1f060449ddc"
Jul 17 21:49:37.237466 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:37.237380    2573 generic.go:356] "Generic (PLEG): container finished" podID="03c1a62d-47b5-4b5a-9da5-766f6c658875" containerID="806cdc490c59493a5a1f4a8fe45900e90308eee0f58b6fe05a5cc6d388358c58" exitCode=7
Jul 17 21:49:37.237466 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:37.237458    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k" event={"ID":"03c1a62d-47b5-4b5a-9da5-766f6c658875","Type":"ContainerDied","Data":"806cdc490c59493a5a1f4a8fe45900e90308eee0f58b6fe05a5cc6d388358c58"}
Jul 17 21:49:37.237899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:37.237514    2573 scope.go:117] "RemoveContainer" containerID="1afe0750068a45b04a47f2bec2d3826e04788c6e2d28e8437fc5e1f060449ddc"
Jul 17 21:49:37.237899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:37.237705    2573 scope.go:117] "RemoveContainer" containerID="806cdc490c59493a5a1f4a8fe45900e90308eee0f58b6fe05a5cc6d388358c58"
Jul 17 21:49:37.237899 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:49:37.237892    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k_odh-ai-gateway-infra(03c1a62d-47b5-4b5a-9da5-766f6c658875)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k" podUID="03c1a62d-47b5-4b5a-9da5-766f6c658875"
Jul 17 21:49:38.243763 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:38.243734    2573 scope.go:117] "RemoveContainer" containerID="806cdc490c59493a5a1f4a8fe45900e90308eee0f58b6fe05a5cc6d388358c58"
Jul 17 21:49:38.244152 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:49:38.243906    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k_odh-ai-gateway-infra(03c1a62d-47b5-4b5a-9da5-766f6c658875)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k" podUID="03c1a62d-47b5-4b5a-9da5-766f6c658875"
Jul 17 21:49:53.524945 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:53.524912    2573 scope.go:117] "RemoveContainer" containerID="806cdc490c59493a5a1f4a8fe45900e90308eee0f58b6fe05a5cc6d388358c58"
Jul 17 21:49:54.308238 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:54.308201    2573 generic.go:356] "Generic (PLEG): container finished" podID="03c1a62d-47b5-4b5a-9da5-766f6c658875" containerID="a0cec6c45e146015417aef364f1442db11e48d7604f6b89ed571a437e2e4415b" exitCode=7
Jul 17 21:49:54.308421 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:54.308269    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k" event={"ID":"03c1a62d-47b5-4b5a-9da5-766f6c658875","Type":"ContainerDied","Data":"a0cec6c45e146015417aef364f1442db11e48d7604f6b89ed571a437e2e4415b"}
Jul 17 21:49:54.308421 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:54.308351    2573 scope.go:117] "RemoveContainer" containerID="806cdc490c59493a5a1f4a8fe45900e90308eee0f58b6fe05a5cc6d388358c58"
Jul 17 21:49:54.308658 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:54.308642    2573 scope.go:117] "RemoveContainer" containerID="a0cec6c45e146015417aef364f1442db11e48d7604f6b89ed571a437e2e4415b"
Jul 17 21:49:54.308867 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:49:54.308849    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k_odh-ai-gateway-infra(03c1a62d-47b5-4b5a-9da5-766f6c658875)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k" podUID="03c1a62d-47b5-4b5a-9da5-766f6c658875"
Jul 17 21:49:55.334426 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:55.334394    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k"]
Jul 17 21:49:55.457877 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:55.457852    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k"
Jul 17 21:49:55.566744 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:55.566708    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c1a62d-47b5-4b5a-9da5-766f6c658875-maas-api-service-ca\") pod \"03c1a62d-47b5-4b5a-9da5-766f6c658875\" (UID: \"03c1a62d-47b5-4b5a-9da5-766f6c658875\") "
Jul 17 21:49:55.567061 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:55.567037    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c1a62d-47b5-4b5a-9da5-766f6c658875-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "03c1a62d-47b5-4b5a-9da5-766f6c658875" (UID: "03c1a62d-47b5-4b5a-9da5-766f6c658875"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:49:55.668443 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:55.668407    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c1a62d-47b5-4b5a-9da5-766f6c658875-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:49:56.318658 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:56.318628    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k"
Jul 17 21:49:56.318658 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:56.318661    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k" event={"ID":"03c1a62d-47b5-4b5a-9da5-766f6c658875","Type":"ContainerDied","Data":"f118273648211d1a77b56985890cb5f5b6283eed544a70688e2140d79fc6a637"}
Jul 17 21:49:56.318907 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:56.318720    2573 scope.go:117] "RemoveContainer" containerID="a0cec6c45e146015417aef364f1442db11e48d7604f6b89ed571a437e2e4415b"
Jul 17 21:49:56.338587 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:56.338565    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k"]
Jul 17 21:49:56.344430 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:56.344410    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t7j4k"]
Jul 17 21:49:56.529548 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:49:56.529519    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03c1a62d-47b5-4b5a-9da5-766f6c658875" path="/var/lib/kubelet/pods/03c1a62d-47b5-4b5a-9da5-766f6c658875/volumes"
Jul 17 21:50:00.417706 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.417675    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b"]
Jul 17 21:50:00.418075 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.417944    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c86753-6e53-401e-b977-122a4561499c" containerName="revoke-keys"
Jul 17 21:50:00.418075 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.417956    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c86753-6e53-401e-b977-122a4561499c" containerName="revoke-keys"
Jul 17 21:50:00.418075 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.417964    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03c1a62d-47b5-4b5a-9da5-766f6c658875" containerName="revoke-keys"
Jul 17 21:50:00.418075 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.417970    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="03c1a62d-47b5-4b5a-9da5-766f6c658875" containerName="revoke-keys"
Jul 17 21:50:00.418075 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.418010    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03c1a62d-47b5-4b5a-9da5-766f6c658875" containerName="revoke-keys"
Jul 17 21:50:00.418075 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.418017    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="03c1a62d-47b5-4b5a-9da5-766f6c658875" containerName="revoke-keys"
Jul 17 21:50:00.418075 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.418026    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03c1a62d-47b5-4b5a-9da5-766f6c658875" containerName="revoke-keys"
Jul 17 21:50:00.418075 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.418031    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="03c1a62d-47b5-4b5a-9da5-766f6c658875" containerName="revoke-keys"
Jul 17 21:50:00.418075 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.418076    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="03c1a62d-47b5-4b5a-9da5-766f6c658875" containerName="revoke-keys"
Jul 17 21:50:00.418361 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.418085    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="03c1a62d-47b5-4b5a-9da5-766f6c658875" containerName="revoke-keys"
Jul 17 21:50:00.418361 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.418093    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="03c1a62d-47b5-4b5a-9da5-766f6c658875" containerName="revoke-keys"
Jul 17 21:50:00.420826 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.420809    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b"
Jul 17 21:50:00.423206 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.423184    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:50:00.423352 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.423266    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:50:00.428413 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.428255    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b"]
Jul 17 21:50:00.507181 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.507157    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d546cc49-303a-4bbe-b0df-09158f57d427-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-7h86b\" (UID: \"d546cc49-303a-4bbe-b0df-09158f57d427\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b"
Jul 17 21:50:00.607889 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.607859    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d546cc49-303a-4bbe-b0df-09158f57d427-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-7h86b\" (UID: \"d546cc49-303a-4bbe-b0df-09158f57d427\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b"
Jul 17 21:50:00.608462 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.608445    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d546cc49-303a-4bbe-b0df-09158f57d427-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-7h86b\" (UID: \"d546cc49-303a-4bbe-b0df-09158f57d427\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b"
Jul 17 21:50:00.731721 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.731662    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b"
Jul 17 21:50:00.856275 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:00.856250    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b"]
Jul 17 21:50:00.858651 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:50:00.858627    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd546cc49_303a_4bbe_b0df_09158f57d427.slice/crio-95454f7ac1aa499dd012a7ef708078ece283e82a7a2ba2b4c2a7d941c5039472 WatchSource:0}: Error finding container 95454f7ac1aa499dd012a7ef708078ece283e82a7a2ba2b4c2a7d941c5039472: Status 404 returned error can't find the container with id 95454f7ac1aa499dd012a7ef708078ece283e82a7a2ba2b4c2a7d941c5039472
Jul 17 21:50:01.340402 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:01.340368    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b" event={"ID":"d546cc49-303a-4bbe-b0df-09158f57d427","Type":"ContainerStarted","Data":"fadb643a8ebd80451b7e4894684878505036c8809ef4df713bc7938e2e24821a"}
Jul 17 21:50:01.340552 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:01.340456    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b" event={"ID":"d546cc49-303a-4bbe-b0df-09158f57d427","Type":"ContainerStarted","Data":"95454f7ac1aa499dd012a7ef708078ece283e82a7a2ba2b4c2a7d941c5039472"}
Jul 17 21:50:01.357170 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:01.357123    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b" podStartSLOduration=1.3571086 podStartE2EDuration="1.3571086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:50:01.355923196 +0000 UTC m=+2501.320043807" watchObservedRunningTime="2026-07-17 21:50:01.3571086 +0000 UTC m=+2501.321229257"
Jul 17 21:50:02.345619 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:02.345576    2573 generic.go:356] "Generic (PLEG): container finished" podID="d546cc49-303a-4bbe-b0df-09158f57d427" containerID="fadb643a8ebd80451b7e4894684878505036c8809ef4df713bc7938e2e24821a" exitCode=6
Jul 17 21:50:02.345619 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:02.345613    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b" event={"ID":"d546cc49-303a-4bbe-b0df-09158f57d427","Type":"ContainerDied","Data":"fadb643a8ebd80451b7e4894684878505036c8809ef4df713bc7938e2e24821a"}
Jul 17 21:50:02.346134 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:02.345829    2573 scope.go:117] "RemoveContainer" containerID="fadb643a8ebd80451b7e4894684878505036c8809ef4df713bc7938e2e24821a"
Jul 17 21:50:03.351738 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:03.351707    2573 generic.go:356] "Generic (PLEG): container finished" podID="d546cc49-303a-4bbe-b0df-09158f57d427" containerID="bc3d6c5cd58c1b6f4c515307b999279218d6b270197db748e867856c9e6a6f4c" exitCode=6
Jul 17 21:50:03.352162 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:03.351747    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b" event={"ID":"d546cc49-303a-4bbe-b0df-09158f57d427","Type":"ContainerDied","Data":"bc3d6c5cd58c1b6f4c515307b999279218d6b270197db748e867856c9e6a6f4c"}
Jul 17 21:50:03.352162 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:03.351804    2573 scope.go:117] "RemoveContainer" containerID="fadb643a8ebd80451b7e4894684878505036c8809ef4df713bc7938e2e24821a"
Jul 17 21:50:03.352162 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:03.351963    2573 scope.go:117] "RemoveContainer" containerID="bc3d6c5cd58c1b6f4c515307b999279218d6b270197db748e867856c9e6a6f4c"
Jul 17 21:50:03.352162 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:50:03.352155    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-7h86b_odh-ai-gateway-infra(d546cc49-303a-4bbe-b0df-09158f57d427)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b" podUID="d546cc49-303a-4bbe-b0df-09158f57d427"
Jul 17 21:50:04.357302 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:04.357271    2573 scope.go:117] "RemoveContainer" containerID="bc3d6c5cd58c1b6f4c515307b999279218d6b270197db748e867856c9e6a6f4c"
Jul 17 21:50:04.357644 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:50:04.357435    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-7h86b_odh-ai-gateway-infra(d546cc49-303a-4bbe-b0df-09158f57d427)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b" podUID="d546cc49-303a-4bbe-b0df-09158f57d427"
Jul 17 21:50:04.768323 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:04.768215    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"]
Jul 17 21:50:04.768480 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:04.768428    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j" podUID="5b24183d-ab11-40c6-aef5-be4d4adcd2b7" containerName="istio-proxy" containerID="cri-o://baf867a59ade0462a5d8961f87e7ce68cd6bfe165e7ddaa4e4cf3029f5200dc5" gracePeriod=30
Jul 17 21:50:05.424249 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:05.424212    2573 patch_prober.go:28] interesting pod/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 21:50:05.424621 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:05.424264    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j" podUID="5b24183d-ab11-40c6-aef5-be4d4adcd2b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 21:50:07.384739 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:07.384708    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b"]
Jul 17 21:50:07.388878 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:07.388863    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b"
Jul 17 21:50:07.396097 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:07.396061    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b"]
Jul 17 21:50:07.561355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:07.561325    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b\" (UID: \"3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b"
Jul 17 21:50:07.662479 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:07.662391    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b\" (UID: \"3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b"
Jul 17 21:50:07.663023 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:07.662997    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b\" (UID: \"3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b"
Jul 17 21:50:07.700024 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:07.699996    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b"
Jul 17 21:50:07.828419 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:07.828331    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b"]
Jul 17 21:50:07.830461 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:50:07.830435    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3893eaf3_4d5c_4aa8_a1ed_5d6a7800419c.slice/crio-a47ed640e5e06881a9b4c3de67b44285f6bd619a0fe23bf0f9ce765e4bd56716 WatchSource:0}: Error finding container a47ed640e5e06881a9b4c3de67b44285f6bd619a0fe23bf0f9ce765e4bd56716: Status 404 returned error can't find the container with id a47ed640e5e06881a9b4c3de67b44285f6bd619a0fe23bf0f9ce765e4bd56716
Jul 17 21:50:08.372848 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:08.372818    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b" event={"ID":"3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c","Type":"ContainerStarted","Data":"f766f8632397724938fe5e8a982d6aa7de786c1c04ebddbbdfb5ebe2e4f467e9"}
Jul 17 21:50:08.373060 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:08.372868    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b" event={"ID":"3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c","Type":"ContainerStarted","Data":"a47ed640e5e06881a9b4c3de67b44285f6bd619a0fe23bf0f9ce765e4bd56716"}
Jul 17 21:50:08.387752 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:08.387708    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b" podStartSLOduration=1.3876926059999999 podStartE2EDuration="1.387692606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:50:08.386671478 +0000 UTC m=+2508.350792087" watchObservedRunningTime="2026-07-17 21:50:08.387692606 +0000 UTC m=+2508.351813215"
Jul 17 21:50:09.378229 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:09.378197    2573 generic.go:356] "Generic (PLEG): container finished" podID="3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c" containerID="f766f8632397724938fe5e8a982d6aa7de786c1c04ebddbbdfb5ebe2e4f467e9" exitCode=6
Jul 17 21:50:09.378229 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:09.378232    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b" event={"ID":"3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c","Type":"ContainerDied","Data":"f766f8632397724938fe5e8a982d6aa7de786c1c04ebddbbdfb5ebe2e4f467e9"}
Jul 17 21:50:09.378455 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:09.378441    2573 scope.go:117] "RemoveContainer" containerID="f766f8632397724938fe5e8a982d6aa7de786c1c04ebddbbdfb5ebe2e4f467e9"
Jul 17 21:50:10.030852 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.030787    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:50:10.181234 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.181199    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-workload-socket\") pod \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") "
Jul 17 21:50:10.181234 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.181232    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-envoy\") pod \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") "
Jul 17 21:50:10.181494 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.181249    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-workload-certs\") pod \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") "
Jul 17 21:50:10.181494 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.181269    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istiod-ca-cert\") pod \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") "
Jul 17 21:50:10.181494 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.181337    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-credential-socket\") pod \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") "
Jul 17 21:50:10.181494 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.181381    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-data\") pod \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") "
Jul 17 21:50:10.181494 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.181413    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmmpg\" (UniqueName: \"kubernetes.io/projected/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-kube-api-access-hmmpg\") pod \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") "
Jul 17 21:50:10.181494 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.181446    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-podinfo\") pod \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") "
Jul 17 21:50:10.181494 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.181474    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-token\") pod \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\" (UID: \"5b24183d-ab11-40c6-aef5-be4d4adcd2b7\") "
Jul 17 21:50:10.181830 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.181530    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5b24183d-ab11-40c6-aef5-be4d4adcd2b7" (UID: "5b24183d-ab11-40c6-aef5-be4d4adcd2b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:50:10.181830 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.181593    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5b24183d-ab11-40c6-aef5-be4d4adcd2b7" (UID: "5b24183d-ab11-40c6-aef5-be4d4adcd2b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:50:10.181830 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.181740    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5b24183d-ab11-40c6-aef5-be4d4adcd2b7" (UID: "5b24183d-ab11-40c6-aef5-be4d4adcd2b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:50:10.181830 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.181770    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:50:10.181830 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.181773    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5b24183d-ab11-40c6-aef5-be4d4adcd2b7" (UID: "5b24183d-ab11-40c6-aef5-be4d4adcd2b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:50:10.181830 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.181789    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:50:10.182660 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.182629    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-data" (OuterVolumeSpecName: "istio-data") pod "5b24183d-ab11-40c6-aef5-be4d4adcd2b7" (UID: "5b24183d-ab11-40c6-aef5-be4d4adcd2b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:50:10.183635 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.183608    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-kube-api-access-hmmpg" (OuterVolumeSpecName: "kube-api-access-hmmpg") pod "5b24183d-ab11-40c6-aef5-be4d4adcd2b7" (UID: "5b24183d-ab11-40c6-aef5-be4d4adcd2b7"). InnerVolumeSpecName "kube-api-access-hmmpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:50:10.183743 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.183606    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5b24183d-ab11-40c6-aef5-be4d4adcd2b7" (UID: "5b24183d-ab11-40c6-aef5-be4d4adcd2b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:50:10.183904 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.183881    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5b24183d-ab11-40c6-aef5-be4d4adcd2b7" (UID: "5b24183d-ab11-40c6-aef5-be4d4adcd2b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 21:50:10.184003 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.183947    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-token" (OuterVolumeSpecName: "istio-token") pod "5b24183d-ab11-40c6-aef5-be4d4adcd2b7" (UID: "5b24183d-ab11-40c6-aef5-be4d4adcd2b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:50:10.283086 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.283017    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:50:10.283086 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.283041    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmmpg\" (UniqueName: \"kubernetes.io/projected/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-kube-api-access-hmmpg\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:50:10.283086 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.283050    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:50:10.283086 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.283068    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:50:10.283086 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.283077    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:50:10.283086 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.283084    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:50:10.283086 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.283092    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b24183d-ab11-40c6-aef5-be4d4adcd2b7-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:50:10.383803 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.383771    2573 generic.go:356] "Generic (PLEG): container finished" podID="3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c" containerID="b275d27c3528bf47faeedf1070c041bff687a90c23d5274de444d1074b84cc76" exitCode=6
Jul 17 21:50:10.383953 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.383849    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b" event={"ID":"3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c","Type":"ContainerDied","Data":"b275d27c3528bf47faeedf1070c041bff687a90c23d5274de444d1074b84cc76"}
Jul 17 21:50:10.383953 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.383932    2573 scope.go:117] "RemoveContainer" containerID="f766f8632397724938fe5e8a982d6aa7de786c1c04ebddbbdfb5ebe2e4f467e9"
Jul 17 21:50:10.384080 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.384013    2573 scope.go:117] "RemoveContainer" containerID="b275d27c3528bf47faeedf1070c041bff687a90c23d5274de444d1074b84cc76"
Jul 17 21:50:10.384241 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:50:10.384219    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b_odh-ai-gateway-infra(3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b" podUID="3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c"
Jul 17 21:50:10.385360 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.385337    2573 generic.go:356] "Generic (PLEG): container finished" podID="5b24183d-ab11-40c6-aef5-be4d4adcd2b7" containerID="baf867a59ade0462a5d8961f87e7ce68cd6bfe165e7ddaa4e4cf3029f5200dc5" exitCode=0
Jul 17 21:50:10.385428 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.385380    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"
Jul 17 21:50:10.385428 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.385400    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j" event={"ID":"5b24183d-ab11-40c6-aef5-be4d4adcd2b7","Type":"ContainerDied","Data":"baf867a59ade0462a5d8961f87e7ce68cd6bfe165e7ddaa4e4cf3029f5200dc5"}
Jul 17 21:50:10.385608 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.385431    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j" event={"ID":"5b24183d-ab11-40c6-aef5-be4d4adcd2b7","Type":"ContainerDied","Data":"c083386f8a363474a585eb51e1d962118c9128392a3aa1428ca24cb505d7cce4"}
Jul 17 21:50:10.398186 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.398166    2573 scope.go:117] "RemoveContainer" containerID="baf867a59ade0462a5d8961f87e7ce68cd6bfe165e7ddaa4e4cf3029f5200dc5"
Jul 17 21:50:10.405483 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.405456    2573 scope.go:117] "RemoveContainer" containerID="582cc1ed9d88aa9db5931b96422f07e295ae1f129ec09a790867dce4319f5b91"
Jul 17 21:50:10.411063 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.411041    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"]
Jul 17 21:50:10.412752 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.412740    2573 scope.go:117] "RemoveContainer" containerID="baf867a59ade0462a5d8961f87e7ce68cd6bfe165e7ddaa4e4cf3029f5200dc5"
Jul 17 21:50:10.412984 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:50:10.412966    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf867a59ade0462a5d8961f87e7ce68cd6bfe165e7ddaa4e4cf3029f5200dc5\": container with ID starting with baf867a59ade0462a5d8961f87e7ce68cd6bfe165e7ddaa4e4cf3029f5200dc5 not found: ID does not exist" containerID="baf867a59ade0462a5d8961f87e7ce68cd6bfe165e7ddaa4e4cf3029f5200dc5"
Jul 17 21:50:10.413039 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.412993    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf867a59ade0462a5d8961f87e7ce68cd6bfe165e7ddaa4e4cf3029f5200dc5"} err="failed to get container status \"baf867a59ade0462a5d8961f87e7ce68cd6bfe165e7ddaa4e4cf3029f5200dc5\": rpc error: code = NotFound desc = could not find container \"baf867a59ade0462a5d8961f87e7ce68cd6bfe165e7ddaa4e4cf3029f5200dc5\": container with ID starting with baf867a59ade0462a5d8961f87e7ce68cd6bfe165e7ddaa4e4cf3029f5200dc5 not found: ID does not exist"
Jul 17 21:50:10.413039 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.413015    2573 scope.go:117] "RemoveContainer" containerID="582cc1ed9d88aa9db5931b96422f07e295ae1f129ec09a790867dce4319f5b91"
Jul 17 21:50:10.413215 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:50:10.413200    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582cc1ed9d88aa9db5931b96422f07e295ae1f129ec09a790867dce4319f5b91\": container with ID starting with 582cc1ed9d88aa9db5931b96422f07e295ae1f129ec09a790867dce4319f5b91 not found: ID does not exist" containerID="582cc1ed9d88aa9db5931b96422f07e295ae1f129ec09a790867dce4319f5b91"
Jul 17 21:50:10.413253 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.413219    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582cc1ed9d88aa9db5931b96422f07e295ae1f129ec09a790867dce4319f5b91"} err="failed to get container status \"582cc1ed9d88aa9db5931b96422f07e295ae1f129ec09a790867dce4319f5b91\": rpc error: code = NotFound desc = could not find container \"582cc1ed9d88aa9db5931b96422f07e295ae1f129ec09a790867dce4319f5b91\": container with ID starting with 582cc1ed9d88aa9db5931b96422f07e295ae1f129ec09a790867dce4319f5b91 not found: ID does not exist"
Jul 17 21:50:10.415838 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.415819    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-eecee1fd-openshift-default-7cfcbd5dcf-j6w7j"]
Jul 17 21:50:10.529524 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:10.529494    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b24183d-ab11-40c6-aef5-be4d4adcd2b7" path="/var/lib/kubelet/pods/5b24183d-ab11-40c6-aef5-be4d4adcd2b7/volumes"
Jul 17 21:50:11.391605 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:11.391578    2573 scope.go:117] "RemoveContainer" containerID="b275d27c3528bf47faeedf1070c041bff687a90c23d5274de444d1074b84cc76"
Jul 17 21:50:11.391983 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:50:11.391757    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b_odh-ai-gateway-infra(3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b" podUID="3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c"
Jul 17 21:50:15.524458 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:15.524422    2573 scope.go:117] "RemoveContainer" containerID="bc3d6c5cd58c1b6f4c515307b999279218d6b270197db748e867856c9e6a6f4c"
Jul 17 21:50:16.413317 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:16.413269    2573 generic.go:356] "Generic (PLEG): container finished" podID="d546cc49-303a-4bbe-b0df-09158f57d427" containerID="37d2b1bc87553ba78251a6e69dcc3d1e5c761e00906e4ddd1913ade3c77fb45a" exitCode=6
Jul 17 21:50:16.413483 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:16.413319    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b" event={"ID":"d546cc49-303a-4bbe-b0df-09158f57d427","Type":"ContainerDied","Data":"37d2b1bc87553ba78251a6e69dcc3d1e5c761e00906e4ddd1913ade3c77fb45a"}
Jul 17 21:50:16.413483 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:16.413378    2573 scope.go:117] "RemoveContainer" containerID="bc3d6c5cd58c1b6f4c515307b999279218d6b270197db748e867856c9e6a6f4c"
Jul 17 21:50:16.413634 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:16.413620    2573 scope.go:117] "RemoveContainer" containerID="37d2b1bc87553ba78251a6e69dcc3d1e5c761e00906e4ddd1913ade3c77fb45a"
Jul 17 21:50:16.413842 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:50:16.413816    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-7h86b_odh-ai-gateway-infra(d546cc49-303a-4bbe-b0df-09158f57d427)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b" podUID="d546cc49-303a-4bbe-b0df-09158f57d427"
Jul 17 21:50:17.446788 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:17.446758    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b"]
Jul 17 21:50:17.582519 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:17.582495    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b"
Jul 17 21:50:17.636734 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:17.636711    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d546cc49-303a-4bbe-b0df-09158f57d427-maas-api-service-ca\") pod \"d546cc49-303a-4bbe-b0df-09158f57d427\" (UID: \"d546cc49-303a-4bbe-b0df-09158f57d427\") "
Jul 17 21:50:17.637028 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:17.637010    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d546cc49-303a-4bbe-b0df-09158f57d427-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d546cc49-303a-4bbe-b0df-09158f57d427" (UID: "d546cc49-303a-4bbe-b0df-09158f57d427"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:50:17.737779 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:17.737722    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d546cc49-303a-4bbe-b0df-09158f57d427-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:50:18.425267 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:18.425230    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b" event={"ID":"d546cc49-303a-4bbe-b0df-09158f57d427","Type":"ContainerDied","Data":"95454f7ac1aa499dd012a7ef708078ece283e82a7a2ba2b4c2a7d941c5039472"}
Jul 17 21:50:18.425447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:18.425314    2573 scope.go:117] "RemoveContainer" containerID="37d2b1bc87553ba78251a6e69dcc3d1e5c761e00906e4ddd1913ade3c77fb45a"
Jul 17 21:50:18.425447 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:18.425239    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b"
Jul 17 21:50:18.445616 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:18.445592    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b"]
Jul 17 21:50:18.451711 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:18.451691    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-7h86b"]
Jul 17 21:50:18.529615 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:18.529583    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d546cc49-303a-4bbe-b0df-09158f57d427" path="/var/lib/kubelet/pods/d546cc49-303a-4bbe-b0df-09158f57d427/volumes"
Jul 17 21:50:24.524949 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:24.524916    2573 scope.go:117] "RemoveContainer" containerID="b275d27c3528bf47faeedf1070c041bff687a90c23d5274de444d1074b84cc76"
Jul 17 21:50:25.456113 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:25.456021    2573 generic.go:356] "Generic (PLEG): container finished" podID="3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c" containerID="409a72775c4ca127e52371100ec5ad012ee07ca06e8fc43d236cf590734c6b3d" exitCode=6
Jul 17 21:50:25.456113 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:25.456090    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b" event={"ID":"3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c","Type":"ContainerDied","Data":"409a72775c4ca127e52371100ec5ad012ee07ca06e8fc43d236cf590734c6b3d"}
Jul 17 21:50:25.456375 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:25.456147    2573 scope.go:117] "RemoveContainer" containerID="b275d27c3528bf47faeedf1070c041bff687a90c23d5274de444d1074b84cc76"
Jul 17 21:50:25.456489 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:25.456466    2573 scope.go:117] "RemoveContainer" containerID="409a72775c4ca127e52371100ec5ad012ee07ca06e8fc43d236cf590734c6b3d"
Jul 17 21:50:25.456669 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:50:25.456650    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b_odh-ai-gateway-infra(3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b" podUID="3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c"
Jul 17 21:50:26.481218 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:26.481183    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b"]
Jul 17 21:50:26.611479 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:26.611457    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b"
Jul 17 21:50:26.702631 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:26.702598    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c-maas-api-service-ca\") pod \"3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c\" (UID: \"3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c\") "
Jul 17 21:50:26.702934 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:26.702910    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c" (UID: "3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:50:26.803207 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:26.803137    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:50:27.466974 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:27.466943    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b"
Jul 17 21:50:27.466974 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:27.466968    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b" event={"ID":"3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c","Type":"ContainerDied","Data":"a47ed640e5e06881a9b4c3de67b44285f6bd619a0fe23bf0f9ce765e4bd56716"}
Jul 17 21:50:27.467221 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:27.467022    2573 scope.go:117] "RemoveContainer" containerID="409a72775c4ca127e52371100ec5ad012ee07ca06e8fc43d236cf590734c6b3d"
Jul 17 21:50:27.487677 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:27.487653    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b"]
Jul 17 21:50:27.496259 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:27.496238    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-m8c2b"]
Jul 17 21:50:28.529260 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:28.529229    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c" path="/var/lib/kubelet/pods/3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c/volumes"
Jul 17 21:50:34.831220 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.831184    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7"]
Jul 17 21:50:34.831600 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.831493    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c" containerName="revoke-keys"
Jul 17 21:50:34.831600 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.831507    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c" containerName="revoke-keys"
Jul 17 21:50:34.831600 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.831521    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d546cc49-303a-4bbe-b0df-09158f57d427" containerName="revoke-keys"
Jul 17 21:50:34.831600 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.831529    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d546cc49-303a-4bbe-b0df-09158f57d427" containerName="revoke-keys"
Jul 17 21:50:34.831600 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.831537    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b24183d-ab11-40c6-aef5-be4d4adcd2b7" containerName="istio-proxy"
Jul 17 21:50:34.831600 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.831542    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b24183d-ab11-40c6-aef5-be4d4adcd2b7" containerName="istio-proxy"
Jul 17 21:50:34.831600 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.831548    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d546cc49-303a-4bbe-b0df-09158f57d427" containerName="revoke-keys"
Jul 17 21:50:34.831600 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.831554    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d546cc49-303a-4bbe-b0df-09158f57d427" containerName="revoke-keys"
Jul 17 21:50:34.831600 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.831575    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c" containerName="revoke-keys"
Jul 17 21:50:34.831600 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.831581    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c" containerName="revoke-keys"
Jul 17 21:50:34.831916 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.831628    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c" containerName="revoke-keys"
Jul 17 21:50:34.831916 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.831635    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d546cc49-303a-4bbe-b0df-09158f57d427" containerName="revoke-keys"
Jul 17 21:50:34.831916 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.831642    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b24183d-ab11-40c6-aef5-be4d4adcd2b7" containerName="istio-proxy"
Jul 17 21:50:34.831916 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.831650    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c" containerName="revoke-keys"
Jul 17 21:50:34.831916 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.831660    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b24183d-ab11-40c6-aef5-be4d4adcd2b7" containerName="istio-proxy"
Jul 17 21:50:34.831916 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.831665    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d546cc49-303a-4bbe-b0df-09158f57d427" containerName="revoke-keys"
Jul 17 21:50:34.831916 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.831672    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c" containerName="revoke-keys"
Jul 17 21:50:34.835804 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.835788    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7"
Jul 17 21:50:34.837994 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.837969    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:50:34.838110 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.838025    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:50:34.841419 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.841396    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7"]
Jul 17 21:50:34.858822 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.858796    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9305133c-22a3-47d0-a8d3-c2144350d95c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7\" (UID: \"9305133c-22a3-47d0-a8d3-c2144350d95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7"
Jul 17 21:50:34.959421 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.959399    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9305133c-22a3-47d0-a8d3-c2144350d95c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7\" (UID: \"9305133c-22a3-47d0-a8d3-c2144350d95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7"
Jul 17 21:50:34.959969 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:34.959951    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9305133c-22a3-47d0-a8d3-c2144350d95c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7\" (UID: \"9305133c-22a3-47d0-a8d3-c2144350d95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7"
Jul 17 21:50:35.147998 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:35.147968    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7"
Jul 17 21:50:35.478982 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:35.478958    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7"]
Jul 17 21:50:35.480279 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:50:35.480250    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9305133c_22a3_47d0_a8d3_c2144350d95c.slice/crio-bea30ae544c28d658b3ce7185122dcb999e803b11358568259d1d54d704c52ad WatchSource:0}: Error finding container bea30ae544c28d658b3ce7185122dcb999e803b11358568259d1d54d704c52ad: Status 404 returned error can't find the container with id bea30ae544c28d658b3ce7185122dcb999e803b11358568259d1d54d704c52ad
Jul 17 21:50:35.500591 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:35.500569    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7" event={"ID":"9305133c-22a3-47d0-a8d3-c2144350d95c","Type":"ContainerStarted","Data":"bea30ae544c28d658b3ce7185122dcb999e803b11358568259d1d54d704c52ad"}
Jul 17 21:50:36.505399 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:36.505362    2573 generic.go:356] "Generic (PLEG): container finished" podID="9305133c-22a3-47d0-a8d3-c2144350d95c" containerID="0908ee66601095c16802d5417e1ea1e4b60355a5669102917c48cb7b7d3cbcbf" exitCode=6
Jul 17 21:50:36.505847 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:36.505447    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7" event={"ID":"9305133c-22a3-47d0-a8d3-c2144350d95c","Type":"ContainerDied","Data":"0908ee66601095c16802d5417e1ea1e4b60355a5669102917c48cb7b7d3cbcbf"}
Jul 17 21:50:36.505847 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:36.505610    2573 scope.go:117] "RemoveContainer" containerID="0908ee66601095c16802d5417e1ea1e4b60355a5669102917c48cb7b7d3cbcbf"
Jul 17 21:50:37.511988 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:37.511893    2573 generic.go:356] "Generic (PLEG): container finished" podID="9305133c-22a3-47d0-a8d3-c2144350d95c" containerID="846b40a4b64ef9d686a28452fdc5dc5ba5e8b11362ab38877634b21ffb799968" exitCode=6
Jul 17 21:50:37.511988 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:37.511946    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7" event={"ID":"9305133c-22a3-47d0-a8d3-c2144350d95c","Type":"ContainerDied","Data":"846b40a4b64ef9d686a28452fdc5dc5ba5e8b11362ab38877634b21ffb799968"}
Jul 17 21:50:37.512515 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:37.512016    2573 scope.go:117] "RemoveContainer" containerID="0908ee66601095c16802d5417e1ea1e4b60355a5669102917c48cb7b7d3cbcbf"
Jul 17 21:50:37.512515 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:37.512173    2573 scope.go:117] "RemoveContainer" containerID="846b40a4b64ef9d686a28452fdc5dc5ba5e8b11362ab38877634b21ffb799968"
Jul 17 21:50:37.512515 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:50:37.512443    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7_odh-ai-gateway-infra(9305133c-22a3-47d0-a8d3-c2144350d95c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7" podUID="9305133c-22a3-47d0-a8d3-c2144350d95c"
Jul 17 21:50:38.517916 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:38.517888    2573 scope.go:117] "RemoveContainer" containerID="846b40a4b64ef9d686a28452fdc5dc5ba5e8b11362ab38877634b21ffb799968"
Jul 17 21:50:38.518319 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:50:38.518050    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7_odh-ai-gateway-infra(9305133c-22a3-47d0-a8d3-c2144350d95c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7" podUID="9305133c-22a3-47d0-a8d3-c2144350d95c"
Jul 17 21:50:49.525203 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:49.525173    2573 scope.go:117] "RemoveContainer" containerID="846b40a4b64ef9d686a28452fdc5dc5ba5e8b11362ab38877634b21ffb799968"
Jul 17 21:50:50.464430 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.464350    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx"]
Jul 17 21:50:50.464654 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.464641    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c" containerName="revoke-keys"
Jul 17 21:50:50.464700 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.464656    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3893eaf3-4d5c-4aa8-a1ed-5d6a7800419c" containerName="revoke-keys"
Jul 17 21:50:50.464700 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.464674    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b24183d-ab11-40c6-aef5-be4d4adcd2b7" containerName="istio-proxy"
Jul 17 21:50:50.464700 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.464680    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b24183d-ab11-40c6-aef5-be4d4adcd2b7" containerName="istio-proxy"
Jul 17 21:50:50.464700 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.464686    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d546cc49-303a-4bbe-b0df-09158f57d427" containerName="revoke-keys"
Jul 17 21:50:50.464700 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.464692    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d546cc49-303a-4bbe-b0df-09158f57d427" containerName="revoke-keys"
Jul 17 21:50:50.464918 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.464740    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d546cc49-303a-4bbe-b0df-09158f57d427" containerName="revoke-keys"
Jul 17 21:50:50.467476 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.467461    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx"
Jul 17 21:50:50.475613 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.475590    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx"]
Jul 17 21:50:50.564364 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.564329    2573 generic.go:356] "Generic (PLEG): container finished" podID="9305133c-22a3-47d0-a8d3-c2144350d95c" containerID="57b2ad9ccdee973ea2a6fde3a27705bb73403dfd41f712b86b0d0489998daca5" exitCode=6
Jul 17 21:50:50.564768 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.564400    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7" event={"ID":"9305133c-22a3-47d0-a8d3-c2144350d95c","Type":"ContainerDied","Data":"57b2ad9ccdee973ea2a6fde3a27705bb73403dfd41f712b86b0d0489998daca5"}
Jul 17 21:50:50.564768 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.564456    2573 scope.go:117] "RemoveContainer" containerID="846b40a4b64ef9d686a28452fdc5dc5ba5e8b11362ab38877634b21ffb799968"
Jul 17 21:50:50.564768 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.564759    2573 scope.go:117] "RemoveContainer" containerID="57b2ad9ccdee973ea2a6fde3a27705bb73403dfd41f712b86b0d0489998daca5"
Jul 17 21:50:50.565007 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:50:50.564978    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7_odh-ai-gateway-infra(9305133c-22a3-47d0-a8d3-c2144350d95c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7" podUID="9305133c-22a3-47d0-a8d3-c2144350d95c"
Jul 17 21:50:50.579911 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.579892    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7477adbc-2bc0-40e1-b29b-f2e3711626aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-9zlcx\" (UID: \"7477adbc-2bc0-40e1-b29b-f2e3711626aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx"
Jul 17 21:50:50.680799 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.680769    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7477adbc-2bc0-40e1-b29b-f2e3711626aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-9zlcx\" (UID: \"7477adbc-2bc0-40e1-b29b-f2e3711626aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx"
Jul 17 21:50:50.681368 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.681350    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7477adbc-2bc0-40e1-b29b-f2e3711626aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-9zlcx\" (UID: \"7477adbc-2bc0-40e1-b29b-f2e3711626aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx"
Jul 17 21:50:50.778742 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.778677    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx"
Jul 17 21:50:50.901743 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:50.901704    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx"]
Jul 17 21:50:50.904988 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:50:50.904965    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7477adbc_2bc0_40e1_b29b_f2e3711626aa.slice/crio-78242743ff0b75a14a549666976d716d11e1fd091fc585f9e355ffb2dc7286a7 WatchSource:0}: Error finding container 78242743ff0b75a14a549666976d716d11e1fd091fc585f9e355ffb2dc7286a7: Status 404 returned error can't find the container with id 78242743ff0b75a14a549666976d716d11e1fd091fc585f9e355ffb2dc7286a7
Jul 17 21:50:51.569598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:51.569570    2573 generic.go:356] "Generic (PLEG): container finished" podID="7477adbc-2bc0-40e1-b29b-f2e3711626aa" containerID="00b586229392eb966efd95bba4655c42bd150984ae503c5d96e25fbc8d5c6308" exitCode=6
Jul 17 21:50:51.569943 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:51.569652    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx" event={"ID":"7477adbc-2bc0-40e1-b29b-f2e3711626aa","Type":"ContainerDied","Data":"00b586229392eb966efd95bba4655c42bd150984ae503c5d96e25fbc8d5c6308"}
Jul 17 21:50:51.569943 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:51.569703    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx" event={"ID":"7477adbc-2bc0-40e1-b29b-f2e3711626aa","Type":"ContainerStarted","Data":"78242743ff0b75a14a549666976d716d11e1fd091fc585f9e355ffb2dc7286a7"}
Jul 17 21:50:51.569943 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:51.569847    2573 scope.go:117] "RemoveContainer" containerID="00b586229392eb966efd95bba4655c42bd150984ae503c5d96e25fbc8d5c6308"
Jul 17 21:50:51.592555 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:51.592534    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7"]
Jul 17 21:50:51.709147 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:51.709125    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7"
Jul 17 21:50:51.790219 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:51.790163    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9305133c-22a3-47d0-a8d3-c2144350d95c-maas-api-service-ca\") pod \"9305133c-22a3-47d0-a8d3-c2144350d95c\" (UID: \"9305133c-22a3-47d0-a8d3-c2144350d95c\") "
Jul 17 21:50:51.790541 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:51.790516    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9305133c-22a3-47d0-a8d3-c2144350d95c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9305133c-22a3-47d0-a8d3-c2144350d95c" (UID: "9305133c-22a3-47d0-a8d3-c2144350d95c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:50:51.890938 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:51.890913    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9305133c-22a3-47d0-a8d3-c2144350d95c-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:50:52.576864 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:52.576829    2573 generic.go:356] "Generic (PLEG): container finished" podID="7477adbc-2bc0-40e1-b29b-f2e3711626aa" containerID="e339c645a85c2cf2c0b7d53340423fa167e227ebce4b3bae8973350658fddfad" exitCode=6
Jul 17 21:50:52.577314 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:52.576889    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx" event={"ID":"7477adbc-2bc0-40e1-b29b-f2e3711626aa","Type":"ContainerDied","Data":"e339c645a85c2cf2c0b7d53340423fa167e227ebce4b3bae8973350658fddfad"}
Jul 17 21:50:52.577314 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:52.576958    2573 scope.go:117] "RemoveContainer" containerID="00b586229392eb966efd95bba4655c42bd150984ae503c5d96e25fbc8d5c6308"
Jul 17 21:50:52.577314 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:52.577110    2573 scope.go:117] "RemoveContainer" containerID="e339c645a85c2cf2c0b7d53340423fa167e227ebce4b3bae8973350658fddfad"
Jul 17 21:50:52.577314 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:50:52.577280    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-9zlcx_odh-ai-gateway-infra(7477adbc-2bc0-40e1-b29b-f2e3711626aa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx" podUID="7477adbc-2bc0-40e1-b29b-f2e3711626aa"
Jul 17 21:50:52.578363 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:52.578342    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7" event={"ID":"9305133c-22a3-47d0-a8d3-c2144350d95c","Type":"ContainerDied","Data":"bea30ae544c28d658b3ce7185122dcb999e803b11358568259d1d54d704c52ad"}
Jul 17 21:50:52.578363 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:52.578358    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7"
Jul 17 21:50:52.586836 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:52.586820    2573 scope.go:117] "RemoveContainer" containerID="57b2ad9ccdee973ea2a6fde3a27705bb73403dfd41f712b86b0d0489998daca5"
Jul 17 21:50:52.604662 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:52.604637    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7"]
Jul 17 21:50:52.611687 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:52.611668    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-sznn7"]
Jul 17 21:50:53.585413 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:53.585385    2573 scope.go:117] "RemoveContainer" containerID="e339c645a85c2cf2c0b7d53340423fa167e227ebce4b3bae8973350658fddfad"
Jul 17 21:50:53.585762 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:50:53.585542    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-9zlcx_odh-ai-gateway-infra(7477adbc-2bc0-40e1-b29b-f2e3711626aa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx" podUID="7477adbc-2bc0-40e1-b29b-f2e3711626aa"
Jul 17 21:50:54.530106 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:50:54.530073    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9305133c-22a3-47d0-a8d3-c2144350d95c" path="/var/lib/kubelet/pods/9305133c-22a3-47d0-a8d3-c2144350d95c/volumes"
Jul 17 21:51:08.524801 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:08.524768    2573 scope.go:117] "RemoveContainer" containerID="e339c645a85c2cf2c0b7d53340423fa167e227ebce4b3bae8973350658fddfad"
Jul 17 21:51:09.649899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:09.649856    2573 generic.go:356] "Generic (PLEG): container finished" podID="7477adbc-2bc0-40e1-b29b-f2e3711626aa" containerID="b2bdad99a8dd8f46d5dc488f6e01aa2b62abb142f85ec8dc54015100c8bd23ab" exitCode=6
Jul 17 21:51:09.650333 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:09.649931    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx" event={"ID":"7477adbc-2bc0-40e1-b29b-f2e3711626aa","Type":"ContainerDied","Data":"b2bdad99a8dd8f46d5dc488f6e01aa2b62abb142f85ec8dc54015100c8bd23ab"}
Jul 17 21:51:09.650333 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:09.650014    2573 scope.go:117] "RemoveContainer" containerID="e339c645a85c2cf2c0b7d53340423fa167e227ebce4b3bae8973350658fddfad"
Jul 17 21:51:09.650333 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:09.650301    2573 scope.go:117] "RemoveContainer" containerID="b2bdad99a8dd8f46d5dc488f6e01aa2b62abb142f85ec8dc54015100c8bd23ab"
Jul 17 21:51:09.650556 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:51:09.650524    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-9zlcx_odh-ai-gateway-infra(7477adbc-2bc0-40e1-b29b-f2e3711626aa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx" podUID="7477adbc-2bc0-40e1-b29b-f2e3711626aa"
Jul 17 21:51:10.673155 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:10.673124    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx"]
Jul 17 21:51:10.795819 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:10.795789    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx"
Jul 17 21:51:10.932613 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:10.932544    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7477adbc-2bc0-40e1-b29b-f2e3711626aa-maas-api-service-ca\") pod \"7477adbc-2bc0-40e1-b29b-f2e3711626aa\" (UID: \"7477adbc-2bc0-40e1-b29b-f2e3711626aa\") "
Jul 17 21:51:10.932872 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:10.932850    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7477adbc-2bc0-40e1-b29b-f2e3711626aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7477adbc-2bc0-40e1-b29b-f2e3711626aa" (UID: "7477adbc-2bc0-40e1-b29b-f2e3711626aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:51:11.033006 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:11.032976    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7477adbc-2bc0-40e1-b29b-f2e3711626aa-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:51:11.660795 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:11.660767    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx"
Jul 17 21:51:11.661000 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:11.660771    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx" event={"ID":"7477adbc-2bc0-40e1-b29b-f2e3711626aa","Type":"ContainerDied","Data":"78242743ff0b75a14a549666976d716d11e1fd091fc585f9e355ffb2dc7286a7"}
Jul 17 21:51:11.661000 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:11.660904    2573 scope.go:117] "RemoveContainer" containerID="b2bdad99a8dd8f46d5dc488f6e01aa2b62abb142f85ec8dc54015100c8bd23ab"
Jul 17 21:51:11.679394 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:11.679366    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx"]
Jul 17 21:51:11.698801 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:11.698775    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-9zlcx"]
Jul 17 21:51:12.529838 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:12.529800    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7477adbc-2bc0-40e1-b29b-f2e3711626aa" path="/var/lib/kubelet/pods/7477adbc-2bc0-40e1-b29b-f2e3711626aa/volumes"
Jul 17 21:51:24.874458 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:24.874427    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4"]
Jul 17 21:51:24.874831 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:24.874691    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7477adbc-2bc0-40e1-b29b-f2e3711626aa" containerName="revoke-keys"
Jul 17 21:51:24.874831 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:24.874701    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7477adbc-2bc0-40e1-b29b-f2e3711626aa" containerName="revoke-keys"
Jul 17 21:51:24.874831 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:24.874713    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9305133c-22a3-47d0-a8d3-c2144350d95c" containerName="revoke-keys"
Jul 17 21:51:24.874831 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:24.874719    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9305133c-22a3-47d0-a8d3-c2144350d95c" containerName="revoke-keys"
Jul 17 21:51:24.874831 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:24.874731    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9305133c-22a3-47d0-a8d3-c2144350d95c" containerName="revoke-keys"
Jul 17 21:51:24.874831 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:24.874737    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9305133c-22a3-47d0-a8d3-c2144350d95c" containerName="revoke-keys"
Jul 17 21:51:24.874831 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:24.874743    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7477adbc-2bc0-40e1-b29b-f2e3711626aa" containerName="revoke-keys"
Jul 17 21:51:24.874831 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:24.874748    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7477adbc-2bc0-40e1-b29b-f2e3711626aa" containerName="revoke-keys"
Jul 17 21:51:24.874831 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:24.874792    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9305133c-22a3-47d0-a8d3-c2144350d95c" containerName="revoke-keys"
Jul 17 21:51:24.874831 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:24.874800    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9305133c-22a3-47d0-a8d3-c2144350d95c" containerName="revoke-keys"
Jul 17 21:51:24.874831 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:24.874807    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7477adbc-2bc0-40e1-b29b-f2e3711626aa" containerName="revoke-keys"
Jul 17 21:51:24.874831 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:24.874815    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7477adbc-2bc0-40e1-b29b-f2e3711626aa" containerName="revoke-keys"
Jul 17 21:51:24.874831 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:24.874819    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7477adbc-2bc0-40e1-b29b-f2e3711626aa" containerName="revoke-keys"
Jul 17 21:51:24.878941 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:24.878922    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4"
Jul 17 21:51:24.881202 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:24.881173    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:51:24.881338 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:24.881217    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:51:24.884568 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:24.884549    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4"]
Jul 17 21:51:25.033549 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:25.033517    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c199464e-0434-4313-8fa8-8cfd23b70f81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4\" (UID: \"c199464e-0434-4313-8fa8-8cfd23b70f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4"
Jul 17 21:51:25.134948 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:25.134866    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c199464e-0434-4313-8fa8-8cfd23b70f81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4\" (UID: \"c199464e-0434-4313-8fa8-8cfd23b70f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4"
Jul 17 21:51:25.135485 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:25.135468    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c199464e-0434-4313-8fa8-8cfd23b70f81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4\" (UID: \"c199464e-0434-4313-8fa8-8cfd23b70f81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4"
Jul 17 21:51:25.190519 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:25.190493    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4"
Jul 17 21:51:25.318921 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:25.318894    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4"]
Jul 17 21:51:25.321211 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:51:25.321181    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc199464e_0434_4313_8fa8_8cfd23b70f81.slice/crio-ee1e60fc2b1d983dd5b07c0a9e6b3c428e58032f0a532ca6fa8168ed5a339414 WatchSource:0}: Error finding container ee1e60fc2b1d983dd5b07c0a9e6b3c428e58032f0a532ca6fa8168ed5a339414: Status 404 returned error can't find the container with id ee1e60fc2b1d983dd5b07c0a9e6b3c428e58032f0a532ca6fa8168ed5a339414
Jul 17 21:51:25.721875 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:25.721835    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4" event={"ID":"c199464e-0434-4313-8fa8-8cfd23b70f81","Type":"ContainerStarted","Data":"8c4406c606b1cfa204c8719128b81cd218ec08109e3f6815bda20558256976a6"}
Jul 17 21:51:25.722040 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:25.721905    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4" event={"ID":"c199464e-0434-4313-8fa8-8cfd23b70f81","Type":"ContainerStarted","Data":"ee1e60fc2b1d983dd5b07c0a9e6b3c428e58032f0a532ca6fa8168ed5a339414"}
Jul 17 21:51:25.738079 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:25.738024    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4" podStartSLOduration=1.738009453 podStartE2EDuration="1.738009453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:51:25.736638092 +0000 UTC m=+2585.700758702" watchObservedRunningTime="2026-07-17 21:51:25.738009453 +0000 UTC m=+2585.702130101"
Jul 17 21:51:26.727130 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:26.727096    2573 generic.go:356] "Generic (PLEG): container finished" podID="c199464e-0434-4313-8fa8-8cfd23b70f81" containerID="8c4406c606b1cfa204c8719128b81cd218ec08109e3f6815bda20558256976a6" exitCode=6
Jul 17 21:51:26.727535 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:26.727152    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4" event={"ID":"c199464e-0434-4313-8fa8-8cfd23b70f81","Type":"ContainerDied","Data":"8c4406c606b1cfa204c8719128b81cd218ec08109e3f6815bda20558256976a6"}
Jul 17 21:51:26.727535 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:26.727372    2573 scope.go:117] "RemoveContainer" containerID="8c4406c606b1cfa204c8719128b81cd218ec08109e3f6815bda20558256976a6"
Jul 17 21:51:27.733058 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:27.733024    2573 generic.go:356] "Generic (PLEG): container finished" podID="c199464e-0434-4313-8fa8-8cfd23b70f81" containerID="cadf754574b524acbcd83e93907187fdb83008e78cb36df033396946dd2cad12" exitCode=6
Jul 17 21:51:27.733532 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:27.733073    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4" event={"ID":"c199464e-0434-4313-8fa8-8cfd23b70f81","Type":"ContainerDied","Data":"cadf754574b524acbcd83e93907187fdb83008e78cb36df033396946dd2cad12"}
Jul 17 21:51:27.733532 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:27.733122    2573 scope.go:117] "RemoveContainer" containerID="8c4406c606b1cfa204c8719128b81cd218ec08109e3f6815bda20558256976a6"
Jul 17 21:51:27.733532 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:27.733328    2573 scope.go:117] "RemoveContainer" containerID="cadf754574b524acbcd83e93907187fdb83008e78cb36df033396946dd2cad12"
Jul 17 21:51:27.733677 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:51:27.733577    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4_odh-ai-gateway-infra(c199464e-0434-4313-8fa8-8cfd23b70f81)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4" podUID="c199464e-0434-4313-8fa8-8cfd23b70f81"
Jul 17 21:51:28.738859 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:28.738830    2573 scope.go:117] "RemoveContainer" containerID="cadf754574b524acbcd83e93907187fdb83008e78cb36df033396946dd2cad12"
Jul 17 21:51:28.739256 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:51:28.739001    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4_odh-ai-gateway-infra(c199464e-0434-4313-8fa8-8cfd23b70f81)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4" podUID="c199464e-0434-4313-8fa8-8cfd23b70f81"
Jul 17 21:51:29.307686 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:29.307652    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz"]
Jul 17 21:51:29.307991 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:29.307973    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9305133c-22a3-47d0-a8d3-c2144350d95c" containerName="revoke-keys"
Jul 17 21:51:29.308065 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:29.307993    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9305133c-22a3-47d0-a8d3-c2144350d95c" containerName="revoke-keys"
Jul 17 21:51:29.308065 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:29.308021    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7477adbc-2bc0-40e1-b29b-f2e3711626aa" containerName="revoke-keys"
Jul 17 21:51:29.308065 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:29.308029    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7477adbc-2bc0-40e1-b29b-f2e3711626aa" containerName="revoke-keys"
Jul 17 21:51:29.308212 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:29.308119    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9305133c-22a3-47d0-a8d3-c2144350d95c" containerName="revoke-keys"
Jul 17 21:51:29.312112 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:29.312090    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz"
Jul 17 21:51:29.317690 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:29.317628    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz"]
Jul 17 21:51:29.467216 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:29.467178    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977f9c25-60d9-4d3e-a3ee-ebfeaaca7260-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz\" (UID: \"977f9c25-60d9-4d3e-a3ee-ebfeaaca7260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz"
Jul 17 21:51:29.568755 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:29.568661    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977f9c25-60d9-4d3e-a3ee-ebfeaaca7260-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz\" (UID: \"977f9c25-60d9-4d3e-a3ee-ebfeaaca7260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz"
Jul 17 21:51:29.569274 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:29.569256    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977f9c25-60d9-4d3e-a3ee-ebfeaaca7260-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz\" (UID: \"977f9c25-60d9-4d3e-a3ee-ebfeaaca7260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz"
Jul 17 21:51:29.623523 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:29.623499    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz"
Jul 17 21:51:29.746655 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:29.746471    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz"]
Jul 17 21:51:29.749116 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:51:29.749091    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977f9c25_60d9_4d3e_a3ee_ebfeaaca7260.slice/crio-9f867db292a8c43f8f10c806c0f3976e7abbc9aba691f56ac97728cd88fd83e2 WatchSource:0}: Error finding container 9f867db292a8c43f8f10c806c0f3976e7abbc9aba691f56ac97728cd88fd83e2: Status 404 returned error can't find the container with id 9f867db292a8c43f8f10c806c0f3976e7abbc9aba691f56ac97728cd88fd83e2
Jul 17 21:51:30.749163 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:30.749080    2573 generic.go:356] "Generic (PLEG): container finished" podID="977f9c25-60d9-4d3e-a3ee-ebfeaaca7260" containerID="ca9cc53f24cfb1a95f89a2386271dd5f145485b14e1a9cb712808a35c02bf392" exitCode=6
Jul 17 21:51:30.749616 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:30.749157    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz" event={"ID":"977f9c25-60d9-4d3e-a3ee-ebfeaaca7260","Type":"ContainerDied","Data":"ca9cc53f24cfb1a95f89a2386271dd5f145485b14e1a9cb712808a35c02bf392"}
Jul 17 21:51:30.749616 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:30.749205    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz" event={"ID":"977f9c25-60d9-4d3e-a3ee-ebfeaaca7260","Type":"ContainerStarted","Data":"9f867db292a8c43f8f10c806c0f3976e7abbc9aba691f56ac97728cd88fd83e2"}
Jul 17 21:51:30.749616 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:30.749344    2573 scope.go:117] "RemoveContainer" containerID="ca9cc53f24cfb1a95f89a2386271dd5f145485b14e1a9cb712808a35c02bf392"
Jul 17 21:51:31.755674 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:31.755631    2573 generic.go:356] "Generic (PLEG): container finished" podID="977f9c25-60d9-4d3e-a3ee-ebfeaaca7260" containerID="7ca692580b6ca75f9b06413e0abc99564d986665dffd991d23e334e4629fca16" exitCode=6
Jul 17 21:51:31.756092 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:31.755680    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz" event={"ID":"977f9c25-60d9-4d3e-a3ee-ebfeaaca7260","Type":"ContainerDied","Data":"7ca692580b6ca75f9b06413e0abc99564d986665dffd991d23e334e4629fca16"}
Jul 17 21:51:31.756092 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:31.755787    2573 scope.go:117] "RemoveContainer" containerID="ca9cc53f24cfb1a95f89a2386271dd5f145485b14e1a9cb712808a35c02bf392"
Jul 17 21:51:31.756092 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:31.756044    2573 scope.go:117] "RemoveContainer" containerID="7ca692580b6ca75f9b06413e0abc99564d986665dffd991d23e334e4629fca16"
Jul 17 21:51:31.756258 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:51:31.756240    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz_odh-ai-gateway-infra(977f9c25-60d9-4d3e-a3ee-ebfeaaca7260)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz" podUID="977f9c25-60d9-4d3e-a3ee-ebfeaaca7260"
Jul 17 21:51:32.761605 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:32.761579    2573 scope.go:117] "RemoveContainer" containerID="7ca692580b6ca75f9b06413e0abc99564d986665dffd991d23e334e4629fca16"
Jul 17 21:51:32.761970 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:51:32.761734    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz_odh-ai-gateway-infra(977f9c25-60d9-4d3e-a3ee-ebfeaaca7260)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz" podUID="977f9c25-60d9-4d3e-a3ee-ebfeaaca7260"
Jul 17 21:51:42.525086 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:42.525055    2573 scope.go:117] "RemoveContainer" containerID="cadf754574b524acbcd83e93907187fdb83008e78cb36df033396946dd2cad12"
Jul 17 21:51:43.806349 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:43.806316    2573 generic.go:356] "Generic (PLEG): container finished" podID="c199464e-0434-4313-8fa8-8cfd23b70f81" containerID="f461434adfbcb554a10099381239b8cf726284f1f67258f303209cbe53a2d3ee" exitCode=6
Jul 17 21:51:43.806773 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:43.806371    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4" event={"ID":"c199464e-0434-4313-8fa8-8cfd23b70f81","Type":"ContainerDied","Data":"f461434adfbcb554a10099381239b8cf726284f1f67258f303209cbe53a2d3ee"}
Jul 17 21:51:43.806773 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:43.806423    2573 scope.go:117] "RemoveContainer" containerID="cadf754574b524acbcd83e93907187fdb83008e78cb36df033396946dd2cad12"
Jul 17 21:51:43.806773 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:43.806770    2573 scope.go:117] "RemoveContainer" containerID="f461434adfbcb554a10099381239b8cf726284f1f67258f303209cbe53a2d3ee"
Jul 17 21:51:43.806964 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:51:43.806941    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4_odh-ai-gateway-infra(c199464e-0434-4313-8fa8-8cfd23b70f81)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4" podUID="c199464e-0434-4313-8fa8-8cfd23b70f81"
Jul 17 21:51:44.524579 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:44.524546    2573 scope.go:117] "RemoveContainer" containerID="7ca692580b6ca75f9b06413e0abc99564d986665dffd991d23e334e4629fca16"
Jul 17 21:51:44.816498 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:44.816406    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz" event={"ID":"977f9c25-60d9-4d3e-a3ee-ebfeaaca7260","Type":"ContainerStarted","Data":"c3e7dbb496b817b4765796f82e94d65ebc531b81a8a8dd8efb8bcb3b6aa70b3e"}
Jul 17 21:51:44.833913 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:44.833871    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz" podStartSLOduration=15.833856687 podStartE2EDuration="15.833856687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:51:44.833058671 +0000 UTC m=+2604.797179282" watchObservedRunningTime="2026-07-17 21:51:44.833856687 +0000 UTC m=+2604.797977297"
Jul 17 21:51:44.837687 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:44.837664    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4"]
Jul 17 21:51:44.967204 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:44.967178    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4"
Jul 17 21:51:45.087116 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:45.087080    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c199464e-0434-4313-8fa8-8cfd23b70f81-maas-api-service-ca\") pod \"c199464e-0434-4313-8fa8-8cfd23b70f81\" (UID: \"c199464e-0434-4313-8fa8-8cfd23b70f81\") "
Jul 17 21:51:45.087495 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:45.087475    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c199464e-0434-4313-8fa8-8cfd23b70f81-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c199464e-0434-4313-8fa8-8cfd23b70f81" (UID: "c199464e-0434-4313-8fa8-8cfd23b70f81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:51:45.187696 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:45.187667    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c199464e-0434-4313-8fa8-8cfd23b70f81-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:51:45.821377 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:45.821346    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4"
Jul 17 21:51:45.821856 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:45.821378    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4" event={"ID":"c199464e-0434-4313-8fa8-8cfd23b70f81","Type":"ContainerDied","Data":"ee1e60fc2b1d983dd5b07c0a9e6b3c428e58032f0a532ca6fa8168ed5a339414"}
Jul 17 21:51:45.821856 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:45.821458    2573 scope.go:117] "RemoveContainer" containerID="f461434adfbcb554a10099381239b8cf726284f1f67258f303209cbe53a2d3ee"
Jul 17 21:51:45.823326 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:45.823277    2573 generic.go:356] "Generic (PLEG): container finished" podID="977f9c25-60d9-4d3e-a3ee-ebfeaaca7260" containerID="c3e7dbb496b817b4765796f82e94d65ebc531b81a8a8dd8efb8bcb3b6aa70b3e" exitCode=6
Jul 17 21:51:45.823471 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:45.823322    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz" event={"ID":"977f9c25-60d9-4d3e-a3ee-ebfeaaca7260","Type":"ContainerDied","Data":"c3e7dbb496b817b4765796f82e94d65ebc531b81a8a8dd8efb8bcb3b6aa70b3e"}
Jul 17 21:51:45.823681 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:45.823663    2573 scope.go:117] "RemoveContainer" containerID="c3e7dbb496b817b4765796f82e94d65ebc531b81a8a8dd8efb8bcb3b6aa70b3e"
Jul 17 21:51:45.823911 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:51:45.823889    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz_odh-ai-gateway-infra(977f9c25-60d9-4d3e-a3ee-ebfeaaca7260)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz" podUID="977f9c25-60d9-4d3e-a3ee-ebfeaaca7260"
Jul 17 21:51:45.829915 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:45.829898    2573 scope.go:117] "RemoveContainer" containerID="7ca692580b6ca75f9b06413e0abc99564d986665dffd991d23e334e4629fca16"
Jul 17 21:51:45.850493 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:45.850444    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz"]
Jul 17 21:51:45.861187 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:45.861160    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4"]
Jul 17 21:51:45.863540 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:45.863511    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-vvpr4"]
Jul 17 21:51:46.529603 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:46.529568    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c199464e-0434-4313-8fa8-8cfd23b70f81" path="/var/lib/kubelet/pods/c199464e-0434-4313-8fa8-8cfd23b70f81/volumes"
Jul 17 21:51:46.956539 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:46.956514    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz"
Jul 17 21:51:47.001604 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:47.001576    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977f9c25-60d9-4d3e-a3ee-ebfeaaca7260-maas-api-service-ca\") pod \"977f9c25-60d9-4d3e-a3ee-ebfeaaca7260\" (UID: \"977f9c25-60d9-4d3e-a3ee-ebfeaaca7260\") "
Jul 17 21:51:47.001902 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:47.001881    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/977f9c25-60d9-4d3e-a3ee-ebfeaaca7260-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "977f9c25-60d9-4d3e-a3ee-ebfeaaca7260" (UID: "977f9c25-60d9-4d3e-a3ee-ebfeaaca7260"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:51:47.102105 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:47.102076    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977f9c25-60d9-4d3e-a3ee-ebfeaaca7260-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:51:47.835132 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:47.835107    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz"
Jul 17 21:51:47.835132 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:47.835120    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz" event={"ID":"977f9c25-60d9-4d3e-a3ee-ebfeaaca7260","Type":"ContainerDied","Data":"9f867db292a8c43f8f10c806c0f3976e7abbc9aba691f56ac97728cd88fd83e2"}
Jul 17 21:51:47.835408 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:47.835179    2573 scope.go:117] "RemoveContainer" containerID="c3e7dbb496b817b4765796f82e94d65ebc531b81a8a8dd8efb8bcb3b6aa70b3e"
Jul 17 21:51:47.856004 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:47.855981    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz"]
Jul 17 21:51:47.861836 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:47.861817    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-wp4hz"]
Jul 17 21:51:48.529961 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:48.529927    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="977f9c25-60d9-4d3e-a3ee-ebfeaaca7260" path="/var/lib/kubelet/pods/977f9c25-60d9-4d3e-a3ee-ebfeaaca7260/volumes"
Jul 17 21:51:51.976089 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:51.976059    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"]
Jul 17 21:51:51.976486 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:51.976366    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977f9c25-60d9-4d3e-a3ee-ebfeaaca7260" containerName="revoke-keys"
Jul 17 21:51:51.976486 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:51.976379    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="977f9c25-60d9-4d3e-a3ee-ebfeaaca7260" containerName="revoke-keys"
Jul 17 21:51:51.976486 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:51.976389    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977f9c25-60d9-4d3e-a3ee-ebfeaaca7260" containerName="revoke-keys"
Jul 17 21:51:51.976486 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:51.976395    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="977f9c25-60d9-4d3e-a3ee-ebfeaaca7260" containerName="revoke-keys"
Jul 17 21:51:51.976486 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:51.976406    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c199464e-0434-4313-8fa8-8cfd23b70f81" containerName="revoke-keys"
Jul 17 21:51:51.976486 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:51.976412    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c199464e-0434-4313-8fa8-8cfd23b70f81" containerName="revoke-keys"
Jul 17 21:51:51.976486 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:51.976420    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c199464e-0434-4313-8fa8-8cfd23b70f81" containerName="revoke-keys"
Jul 17 21:51:51.976486 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:51.976425    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c199464e-0434-4313-8fa8-8cfd23b70f81" containerName="revoke-keys"
Jul 17 21:51:51.976486 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:51.976470    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c199464e-0434-4313-8fa8-8cfd23b70f81" containerName="revoke-keys"
Jul 17 21:51:51.976486 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:51.976477    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="977f9c25-60d9-4d3e-a3ee-ebfeaaca7260" containerName="revoke-keys"
Jul 17 21:51:51.976486 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:51.976484    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="977f9c25-60d9-4d3e-a3ee-ebfeaaca7260" containerName="revoke-keys"
Jul 17 21:51:51.976486 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:51.976490    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c199464e-0434-4313-8fa8-8cfd23b70f81" containerName="revoke-keys"
Jul 17 21:51:51.976871 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:51.976494    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="977f9c25-60d9-4d3e-a3ee-ebfeaaca7260" containerName="revoke-keys"
Jul 17 21:51:51.976871 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:51.976503    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c199464e-0434-4313-8fa8-8cfd23b70f81" containerName="revoke-keys"
Jul 17 21:51:51.981192 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:51.981170    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:51.983579 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:51.983563    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-62f6c7d6-openshift-default-dockercfg-6bgcd\""
Jul 17 21:51:51.990895 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:51.990870    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"]
Jul 17 21:51:52.035523 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.035491    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-envoy\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.035634 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.035525    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-credential-socket\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.035634 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.035546    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-workload-socket\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.035634 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.035593    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg67x\" (UniqueName: \"kubernetes.io/projected/b6cafea6-3c01-4096-8747-e32639fdcb4f-kube-api-access-wg67x\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.035824 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.035671    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cafea6-3c01-4096-8747-e32639fdcb4f-istiod-ca-cert\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.035824 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.035704    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-data\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.035824 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.035739    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-podinfo\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.035824 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.035792    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-workload-certs\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.035824 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.035819    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-token\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.136540 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.136510    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-workload-certs\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.136540 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.136540    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-token\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.136759 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.136579    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-envoy\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.136759 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.136596    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-credential-socket\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.136759 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.136622    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-workload-socket\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.136759 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.136646    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg67x\" (UniqueName: \"kubernetes.io/projected/b6cafea6-3c01-4096-8747-e32639fdcb4f-kube-api-access-wg67x\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.136759 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.136691    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cafea6-3c01-4096-8747-e32639fdcb4f-istiod-ca-cert\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.136759 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.136720    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-data\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.136759 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.136749    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-podinfo\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.137106 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.137052    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-credential-socket\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.137167 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.137114    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-data\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.137167 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.137144    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-workload-socket\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.137272 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.137249    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-workload-certs\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.137413 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.137396    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cafea6-3c01-4096-8747-e32639fdcb4f-istiod-ca-cert\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.138766 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.138743    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-envoy\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.139252 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.139231    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-podinfo\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.144218 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.144196    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-token\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.144857 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.144835    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg67x\" (UniqueName: \"kubernetes.io/projected/b6cafea6-3c01-4096-8747-e32639fdcb4f-kube-api-access-wg67x\") pod \"e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") " pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.292094 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.292039    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:52.419791 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.419740    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"]
Jul 17 21:51:52.422115 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:51:52.422094    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cafea6_3c01_4096_8747_e32639fdcb4f.slice/crio-7b8578453db05a2af788061196bc4f30e67847a55d531f971296fc184d758aa0 WatchSource:0}: Error finding container 7b8578453db05a2af788061196bc4f30e67847a55d531f971296fc184d758aa0: Status 404 returned error can't find the container with id 7b8578453db05a2af788061196bc4f30e67847a55d531f971296fc184d758aa0
Jul 17 21:51:52.424628 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.424592    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:51:52.424693 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.424665    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:51:52.424734 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.424695    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:51:52.859240 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.859201    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w" event={"ID":"b6cafea6-3c01-4096-8747-e32639fdcb4f","Type":"ContainerStarted","Data":"6814ce10ad3ee65df5255b0b0e8fddff288c96b38a5a3ae23b6a522ac18731f8"}
Jul 17 21:51:52.859440 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.859259    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w" event={"ID":"b6cafea6-3c01-4096-8747-e32639fdcb4f","Type":"ContainerStarted","Data":"7b8578453db05a2af788061196bc4f30e67847a55d531f971296fc184d758aa0"}
Jul 17 21:51:52.877694 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:52.877648    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w" podStartSLOduration=1.8776345669999999 podStartE2EDuration="1.877634567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:51:52.876112375 +0000 UTC m=+2612.840232985" watchObservedRunningTime="2026-07-17 21:51:52.877634567 +0000 UTC m=+2612.841755176"
Jul 17 21:51:53.292908 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:53.292839    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:53.297180 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:53.297159    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:53.862837 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:53.862805    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:51:53.863711 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:51:53.863691    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:52:07.484805 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.484775    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46"]
Jul 17 21:52:07.485170 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.485148    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977f9c25-60d9-4d3e-a3ee-ebfeaaca7260" containerName="revoke-keys"
Jul 17 21:52:07.485170 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.485162    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="977f9c25-60d9-4d3e-a3ee-ebfeaaca7260" containerName="revoke-keys"
Jul 17 21:52:07.485320 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.485180    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c199464e-0434-4313-8fa8-8cfd23b70f81" containerName="revoke-keys"
Jul 17 21:52:07.485320 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.485186    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c199464e-0434-4313-8fa8-8cfd23b70f81" containerName="revoke-keys"
Jul 17 21:52:07.487820 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.487798    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46"
Jul 17 21:52:07.490208 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.490188    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:52:07.490317 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.490277    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:52:07.495356 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.495333    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46"]
Jul 17 21:52:07.559697 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.559668    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1eca942-91b0-458b-ba7e-dd8d9ab1f107-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46\" (UID: \"c1eca942-91b0-458b-ba7e-dd8d9ab1f107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46"
Jul 17 21:52:07.660257 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.660188    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1eca942-91b0-458b-ba7e-dd8d9ab1f107-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46\" (UID: \"c1eca942-91b0-458b-ba7e-dd8d9ab1f107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46"
Jul 17 21:52:07.660792 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.660773    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1eca942-91b0-458b-ba7e-dd8d9ab1f107-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46\" (UID: \"c1eca942-91b0-458b-ba7e-dd8d9ab1f107\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46"
Jul 17 21:52:07.840318 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.840271    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46"
Jul 17 21:52:07.918709 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.918675    2573 generic.go:356] "Generic (PLEG): container finished" podID="b6cafea6-3c01-4096-8747-e32639fdcb4f" containerID="6814ce10ad3ee65df5255b0b0e8fddff288c96b38a5a3ae23b6a522ac18731f8" exitCode=0
Jul 17 21:52:07.918819 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.918761    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w" event={"ID":"b6cafea6-3c01-4096-8747-e32639fdcb4f","Type":"ContainerDied","Data":"6814ce10ad3ee65df5255b0b0e8fddff288c96b38a5a3ae23b6a522ac18731f8"}
Jul 17 21:52:07.919308 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.919264    2573 scope.go:117] "RemoveContainer" containerID="6814ce10ad3ee65df5255b0b0e8fddff288c96b38a5a3ae23b6a522ac18731f8"
Jul 17 21:52:07.920537 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.920499    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:52:07.920648 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.920598    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:52:07.920716 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:07.920653    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:52:08.175757 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:08.175722    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46"]
Jul 17 21:52:08.177945 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:52:08.177917    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1eca942_91b0_458b_ba7e_dd8d9ab1f107.slice/crio-650089e293c10ec5805a52d32f28c138fd121029c033aa4e8774729f3a66718f WatchSource:0}: Error finding container 650089e293c10ec5805a52d32f28c138fd121029c033aa4e8774729f3a66718f: Status 404 returned error can't find the container with id 650089e293c10ec5805a52d32f28c138fd121029c033aa4e8774729f3a66718f
Jul 17 21:52:08.863587 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:08.863547    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:52:08.923878 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:08.923843    2573 generic.go:356] "Generic (PLEG): container finished" podID="c1eca942-91b0-458b-ba7e-dd8d9ab1f107" containerID="0a8b39fa842a93f51af5c95d68751a4bf3c2ea896c065804f6c1ac58316bcc4d" exitCode=6
Jul 17 21:52:08.924047 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:08.923932    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46" event={"ID":"c1eca942-91b0-458b-ba7e-dd8d9ab1f107","Type":"ContainerDied","Data":"0a8b39fa842a93f51af5c95d68751a4bf3c2ea896c065804f6c1ac58316bcc4d"}
Jul 17 21:52:08.924047 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:08.923992    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46" event={"ID":"c1eca942-91b0-458b-ba7e-dd8d9ab1f107","Type":"ContainerStarted","Data":"650089e293c10ec5805a52d32f28c138fd121029c033aa4e8774729f3a66718f"}
Jul 17 21:52:08.924167 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:08.924150    2573 scope.go:117] "RemoveContainer" containerID="0a8b39fa842a93f51af5c95d68751a4bf3c2ea896c065804f6c1ac58316bcc4d"
Jul 17 21:52:08.925952 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:08.925930    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w" event={"ID":"b6cafea6-3c01-4096-8747-e32639fdcb4f","Type":"ContainerStarted","Data":"650d86edadcdf879c29babbc873a88b9e2c67d5f0d7cbd597f8686f770c78f55"}
Jul 17 21:52:09.292918 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:09.292890    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:52:09.294148 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:09.294123    2573 patch_prober.go:28] interesting pod/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 17 21:52:09.294260 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:09.294172    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w" podUID="b6cafea6-3c01-4096-8747-e32639fdcb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 17 21:52:09.957609 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:09.957574    2573 generic.go:356] "Generic (PLEG): container finished" podID="c1eca942-91b0-458b-ba7e-dd8d9ab1f107" containerID="9045195dde0b289f69955579f6782606bffc5446729ebee7b52c3b0311abdb4f" exitCode=6
Jul 17 21:52:09.958099 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:09.957667    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46" event={"ID":"c1eca942-91b0-458b-ba7e-dd8d9ab1f107","Type":"ContainerDied","Data":"9045195dde0b289f69955579f6782606bffc5446729ebee7b52c3b0311abdb4f"}
Jul 17 21:52:09.958099 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:09.957765    2573 scope.go:117] "RemoveContainer" containerID="0a8b39fa842a93f51af5c95d68751a4bf3c2ea896c065804f6c1ac58316bcc4d"
Jul 17 21:52:09.958099 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:09.957857    2573 scope.go:117] "RemoveContainer" containerID="9045195dde0b289f69955579f6782606bffc5446729ebee7b52c3b0311abdb4f"
Jul 17 21:52:09.958267 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:52:09.958096    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46_odh-ai-gateway-infra(c1eca942-91b0-458b-ba7e-dd8d9ab1f107)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46" podUID="c1eca942-91b0-458b-ba7e-dd8d9ab1f107"
Jul 17 21:52:10.292936 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:10.292897    2573 patch_prober.go:28] interesting pod/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 17 21:52:10.293108 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:10.292960    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w" podUID="b6cafea6-3c01-4096-8747-e32639fdcb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 17 21:52:10.964272 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:10.964243    2573 scope.go:117] "RemoveContainer" containerID="9045195dde0b289f69955579f6782606bffc5446729ebee7b52c3b0311abdb4f"
Jul 17 21:52:10.964654 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:52:10.964486    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46_odh-ai-gateway-infra(c1eca942-91b0-458b-ba7e-dd8d9ab1f107)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46" podUID="c1eca942-91b0-458b-ba7e-dd8d9ab1f107"
Jul 17 21:52:11.293217 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:11.293122    2573 patch_prober.go:28] interesting pod/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 17 21:52:11.293217 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:11.293185    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w" podUID="b6cafea6-3c01-4096-8747-e32639fdcb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 17 21:52:12.293027 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:12.292988    2573 patch_prober.go:28] interesting pod/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 17 21:52:12.293453 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:12.293045    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w" podUID="b6cafea6-3c01-4096-8747-e32639fdcb4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 17 21:52:13.296935 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:13.296901    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:52:13.297356 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:13.297165    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:52:13.297925 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:13.297908    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:52:19.353029 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:19.352996    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8"]
Jul 17 21:52:19.357561 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:19.357545    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8"
Jul 17 21:52:19.363333 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:19.363312    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8"]
Jul 17 21:52:19.458656 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:19.458628    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef719948-beff-42e9-acb1-7d1b78834816-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8\" (UID: \"ef719948-beff-42e9-acb1-7d1b78834816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8"
Jul 17 21:52:19.559618 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:19.559591    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef719948-beff-42e9-acb1-7d1b78834816-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8\" (UID: \"ef719948-beff-42e9-acb1-7d1b78834816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8"
Jul 17 21:52:19.560203 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:19.560184    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef719948-beff-42e9-acb1-7d1b78834816-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8\" (UID: \"ef719948-beff-42e9-acb1-7d1b78834816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8"
Jul 17 21:52:19.669331 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:19.669268    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8"
Jul 17 21:52:19.998760 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:19.998738    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8"]
Jul 17 21:52:20.001033 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:52:20.001000    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef719948_beff_42e9_acb1_7d1b78834816.slice/crio-765e110f3686b7f45aebd37c7b9fb52f6d95e77b7d08aa6047463e66c2dec010 WatchSource:0}: Error finding container 765e110f3686b7f45aebd37c7b9fb52f6d95e77b7d08aa6047463e66c2dec010: Status 404 returned error can't find the container with id 765e110f3686b7f45aebd37c7b9fb52f6d95e77b7d08aa6047463e66c2dec010
Jul 17 21:52:21.006448 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:21.006355    2573 generic.go:356] "Generic (PLEG): container finished" podID="ef719948-beff-42e9-acb1-7d1b78834816" containerID="5a76b79bea01fe5f4cb049d7b0c3d56c5e1ebbc079895b7022f4922608442605" exitCode=6
Jul 17 21:52:21.006789 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:21.006449    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8" event={"ID":"ef719948-beff-42e9-acb1-7d1b78834816","Type":"ContainerDied","Data":"5a76b79bea01fe5f4cb049d7b0c3d56c5e1ebbc079895b7022f4922608442605"}
Jul 17 21:52:21.006789 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:21.006531    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8" event={"ID":"ef719948-beff-42e9-acb1-7d1b78834816","Type":"ContainerStarted","Data":"765e110f3686b7f45aebd37c7b9fb52f6d95e77b7d08aa6047463e66c2dec010"}
Jul 17 21:52:21.006789 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:21.006563    2573 scope.go:117] "RemoveContainer" containerID="5a76b79bea01fe5f4cb049d7b0c3d56c5e1ebbc079895b7022f4922608442605"
Jul 17 21:52:22.012936 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:22.012845    2573 generic.go:356] "Generic (PLEG): container finished" podID="ef719948-beff-42e9-acb1-7d1b78834816" containerID="c98e702537685e13766c5d84da8f65028d53ffb86a142794dd7d98fe5003a5f3" exitCode=6
Jul 17 21:52:22.012936 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:22.012896    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8" event={"ID":"ef719948-beff-42e9-acb1-7d1b78834816","Type":"ContainerDied","Data":"c98e702537685e13766c5d84da8f65028d53ffb86a142794dd7d98fe5003a5f3"}
Jul 17 21:52:22.013458 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:22.012944    2573 scope.go:117] "RemoveContainer" containerID="5a76b79bea01fe5f4cb049d7b0c3d56c5e1ebbc079895b7022f4922608442605"
Jul 17 21:52:22.013458 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:22.013137    2573 scope.go:117] "RemoveContainer" containerID="c98e702537685e13766c5d84da8f65028d53ffb86a142794dd7d98fe5003a5f3"
Jul 17 21:52:22.013458 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:52:22.013356    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8_odh-ai-gateway-infra(ef719948-beff-42e9-acb1-7d1b78834816)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8" podUID="ef719948-beff-42e9-acb1-7d1b78834816"
Jul 17 21:52:23.019426 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:23.019400    2573 scope.go:117] "RemoveContainer" containerID="c98e702537685e13766c5d84da8f65028d53ffb86a142794dd7d98fe5003a5f3"
Jul 17 21:52:23.019788 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:52:23.019576    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8_odh-ai-gateway-infra(ef719948-beff-42e9-acb1-7d1b78834816)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8" podUID="ef719948-beff-42e9-acb1-7d1b78834816"
Jul 17 21:52:23.524544 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:23.524517    2573 scope.go:117] "RemoveContainer" containerID="9045195dde0b289f69955579f6782606bffc5446729ebee7b52c3b0311abdb4f"
Jul 17 21:52:24.025424 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:24.025392    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46" event={"ID":"c1eca942-91b0-458b-ba7e-dd8d9ab1f107","Type":"ContainerStarted","Data":"febad8773ea3d3dd3c38647ccaa390989bf49f3a886f578338f8a6a07d2b7ccd"}
Jul 17 21:52:24.042344 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:24.042297    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46" podStartSLOduration=17.042266889 podStartE2EDuration="17.042266889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:52:24.040708509 +0000 UTC m=+2644.004829177" watchObservedRunningTime="2026-07-17 21:52:24.042266889 +0000 UTC m=+2644.006387499"
Jul 17 21:52:25.031020 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:25.030984    2573 generic.go:356] "Generic (PLEG): container finished" podID="c1eca942-91b0-458b-ba7e-dd8d9ab1f107" containerID="febad8773ea3d3dd3c38647ccaa390989bf49f3a886f578338f8a6a07d2b7ccd" exitCode=6
Jul 17 21:52:25.031485 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:25.031060    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46" event={"ID":"c1eca942-91b0-458b-ba7e-dd8d9ab1f107","Type":"ContainerDied","Data":"febad8773ea3d3dd3c38647ccaa390989bf49f3a886f578338f8a6a07d2b7ccd"}
Jul 17 21:52:25.031485 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:25.031135    2573 scope.go:117] "RemoveContainer" containerID="9045195dde0b289f69955579f6782606bffc5446729ebee7b52c3b0311abdb4f"
Jul 17 21:52:25.031485 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:25.031465    2573 scope.go:117] "RemoveContainer" containerID="febad8773ea3d3dd3c38647ccaa390989bf49f3a886f578338f8a6a07d2b7ccd"
Jul 17 21:52:25.031691 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:52:25.031669    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46_odh-ai-gateway-infra(c1eca942-91b0-458b-ba7e-dd8d9ab1f107)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46" podUID="c1eca942-91b0-458b-ba7e-dd8d9ab1f107"
Jul 17 21:52:25.058152 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:25.058125    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46"]
Jul 17 21:52:25.119256 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:25.119220    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls"]
Jul 17 21:52:25.123465 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:25.123444    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls"
Jul 17 21:52:25.130039 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:25.130011    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls"]
Jul 17 21:52:25.203948 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:25.203919    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8b6132-c3de-4a7c-873d-b55e9cdc83e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls\" (UID: \"8b8b6132-c3de-4a7c-873d-b55e9cdc83e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls"
Jul 17 21:52:25.305329 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:25.305231    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8b6132-c3de-4a7c-873d-b55e9cdc83e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls\" (UID: \"8b8b6132-c3de-4a7c-873d-b55e9cdc83e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls"
Jul 17 21:52:25.305808 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:25.305788    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8b6132-c3de-4a7c-873d-b55e9cdc83e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls\" (UID: \"8b8b6132-c3de-4a7c-873d-b55e9cdc83e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls"
Jul 17 21:52:25.434640 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:25.434607    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls"
Jul 17 21:52:25.554803 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:25.554777    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls"]
Jul 17 21:52:25.557307 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:52:25.557232    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8b6132_c3de_4a7c_873d_b55e9cdc83e1.slice/crio-7a86e287fad266369da1becd79a315fe55e5fd4b2d07e92ecd90289ad90aeb6c WatchSource:0}: Error finding container 7a86e287fad266369da1becd79a315fe55e5fd4b2d07e92ecd90289ad90aeb6c: Status 404 returned error can't find the container with id 7a86e287fad266369da1becd79a315fe55e5fd4b2d07e92ecd90289ad90aeb6c
Jul 17 21:52:26.036896 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:26.036862    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls" event={"ID":"8b8b6132-c3de-4a7c-873d-b55e9cdc83e1","Type":"ContainerStarted","Data":"1a8866421a8e739941d26e021259ab41867cf6fc872a60b82ca9a1fb10aa0a6a"}
Jul 17 21:52:26.037333 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:26.036915    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls" event={"ID":"8b8b6132-c3de-4a7c-873d-b55e9cdc83e1","Type":"ContainerStarted","Data":"7a86e287fad266369da1becd79a315fe55e5fd4b2d07e92ecd90289ad90aeb6c"}
Jul 17 21:52:26.052816 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:26.052746    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls" podStartSLOduration=1.052731169 podStartE2EDuration="1.052731169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:52:26.050979373 +0000 UTC m=+2646.015099982" watchObservedRunningTime="2026-07-17 21:52:26.052731169 +0000 UTC m=+2646.016851779"
Jul 17 21:52:26.165497 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:26.165473    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46"
Jul 17 21:52:26.314819 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:26.314797    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1eca942-91b0-458b-ba7e-dd8d9ab1f107-maas-api-service-ca\") pod \"c1eca942-91b0-458b-ba7e-dd8d9ab1f107\" (UID: \"c1eca942-91b0-458b-ba7e-dd8d9ab1f107\") "
Jul 17 21:52:26.315102 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:26.315079    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1eca942-91b0-458b-ba7e-dd8d9ab1f107-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c1eca942-91b0-458b-ba7e-dd8d9ab1f107" (UID: "c1eca942-91b0-458b-ba7e-dd8d9ab1f107"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:52:26.415744 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:26.415691    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1eca942-91b0-458b-ba7e-dd8d9ab1f107-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:52:27.043635 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:27.043603    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46"
Jul 17 21:52:27.044063 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:27.043641    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46" event={"ID":"c1eca942-91b0-458b-ba7e-dd8d9ab1f107","Type":"ContainerDied","Data":"650089e293c10ec5805a52d32f28c138fd121029c033aa4e8774729f3a66718f"}
Jul 17 21:52:27.044063 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:27.043706    2573 scope.go:117] "RemoveContainer" containerID="febad8773ea3d3dd3c38647ccaa390989bf49f3a886f578338f8a6a07d2b7ccd"
Jul 17 21:52:27.045214 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:27.045187    2573 generic.go:356] "Generic (PLEG): container finished" podID="8b8b6132-c3de-4a7c-873d-b55e9cdc83e1" containerID="1a8866421a8e739941d26e021259ab41867cf6fc872a60b82ca9a1fb10aa0a6a" exitCode=6
Jul 17 21:52:27.045328 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:27.045233    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls" event={"ID":"8b8b6132-c3de-4a7c-873d-b55e9cdc83e1","Type":"ContainerDied","Data":"1a8866421a8e739941d26e021259ab41867cf6fc872a60b82ca9a1fb10aa0a6a"}
Jul 17 21:52:27.045527 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:27.045504    2573 scope.go:117] "RemoveContainer" containerID="1a8866421a8e739941d26e021259ab41867cf6fc872a60b82ca9a1fb10aa0a6a"
Jul 17 21:52:27.078764 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:27.078742    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46"]
Jul 17 21:52:27.082314 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:27.082279    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-8rr46"]
Jul 17 21:52:28.051648 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:28.051557    2573 generic.go:356] "Generic (PLEG): container finished" podID="8b8b6132-c3de-4a7c-873d-b55e9cdc83e1" containerID="dc466319f7bedc1152e2b2d9aa0e085ea492b788c5cf54cfe6b7840c14cbe928" exitCode=6
Jul 17 21:52:28.051648 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:28.051630    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls" event={"ID":"8b8b6132-c3de-4a7c-873d-b55e9cdc83e1","Type":"ContainerDied","Data":"dc466319f7bedc1152e2b2d9aa0e085ea492b788c5cf54cfe6b7840c14cbe928"}
Jul 17 21:52:28.052150 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:28.051685    2573 scope.go:117] "RemoveContainer" containerID="1a8866421a8e739941d26e021259ab41867cf6fc872a60b82ca9a1fb10aa0a6a"
Jul 17 21:52:28.052150 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:28.051874    2573 scope.go:117] "RemoveContainer" containerID="dc466319f7bedc1152e2b2d9aa0e085ea492b788c5cf54cfe6b7840c14cbe928"
Jul 17 21:52:28.052150 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:52:28.052104    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls_odh-ai-gateway-infra(8b8b6132-c3de-4a7c-873d-b55e9cdc83e1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls" podUID="8b8b6132-c3de-4a7c-873d-b55e9cdc83e1"
Jul 17 21:52:28.529954 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:28.529922    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1eca942-91b0-458b-ba7e-dd8d9ab1f107" path="/var/lib/kubelet/pods/c1eca942-91b0-458b-ba7e-dd8d9ab1f107/volumes"
Jul 17 21:52:29.057524 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:29.057492    2573 scope.go:117] "RemoveContainer" containerID="dc466319f7bedc1152e2b2d9aa0e085ea492b788c5cf54cfe6b7840c14cbe928"
Jul 17 21:52:29.057876 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:52:29.057676    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls_odh-ai-gateway-infra(8b8b6132-c3de-4a7c-873d-b55e9cdc83e1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls" podUID="8b8b6132-c3de-4a7c-873d-b55e9cdc83e1"
Jul 17 21:52:31.983182 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:31.983150    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"]
Jul 17 21:52:31.983648 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:31.983447    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w" podUID="b6cafea6-3c01-4096-8747-e32639fdcb4f" containerName="istio-proxy" containerID="cri-o://650d86edadcdf879c29babbc873a88b9e2c67d5f0d7cbd597f8686f770c78f55" gracePeriod=30
Jul 17 21:52:33.524589 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:33.524561    2573 scope.go:117] "RemoveContainer" containerID="c98e702537685e13766c5d84da8f65028d53ffb86a142794dd7d98fe5003a5f3"
Jul 17 21:52:34.078400 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:34.078371    2573 generic.go:356] "Generic (PLEG): container finished" podID="ef719948-beff-42e9-acb1-7d1b78834816" containerID="7380908e01739674985a208fcd255a2cd984f5d3c22a25ff83f92a338d350f96" exitCode=6
Jul 17 21:52:34.078499 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:34.078439    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8" event={"ID":"ef719948-beff-42e9-acb1-7d1b78834816","Type":"ContainerDied","Data":"7380908e01739674985a208fcd255a2cd984f5d3c22a25ff83f92a338d350f96"}
Jul 17 21:52:34.078499 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:34.078495    2573 scope.go:117] "RemoveContainer" containerID="c98e702537685e13766c5d84da8f65028d53ffb86a142794dd7d98fe5003a5f3"
Jul 17 21:52:34.078783 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:34.078765    2573 scope.go:117] "RemoveContainer" containerID="7380908e01739674985a208fcd255a2cd984f5d3c22a25ff83f92a338d350f96"
Jul 17 21:52:34.078976 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:52:34.078958    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8_odh-ai-gateway-infra(ef719948-beff-42e9-acb1-7d1b78834816)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8" podUID="ef719948-beff-42e9-acb1-7d1b78834816"
Jul 17 21:52:35.102397 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:35.102366    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8"]
Jul 17 21:52:35.235154 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:35.235131    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8"
Jul 17 21:52:35.386647 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:35.386577    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef719948-beff-42e9-acb1-7d1b78834816-maas-api-service-ca\") pod \"ef719948-beff-42e9-acb1-7d1b78834816\" (UID: \"ef719948-beff-42e9-acb1-7d1b78834816\") "
Jul 17 21:52:35.386905 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:35.386884    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef719948-beff-42e9-acb1-7d1b78834816-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef719948-beff-42e9-acb1-7d1b78834816" (UID: "ef719948-beff-42e9-acb1-7d1b78834816"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:52:35.488070 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:35.488041    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef719948-beff-42e9-acb1-7d1b78834816-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:52:36.088863 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:36.088831    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8"
Jul 17 21:52:36.089038 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:36.088865    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8" event={"ID":"ef719948-beff-42e9-acb1-7d1b78834816","Type":"ContainerDied","Data":"765e110f3686b7f45aebd37c7b9fb52f6d95e77b7d08aa6047463e66c2dec010"}
Jul 17 21:52:36.089038 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:36.088923    2573 scope.go:117] "RemoveContainer" containerID="7380908e01739674985a208fcd255a2cd984f5d3c22a25ff83f92a338d350f96"
Jul 17 21:52:36.111082 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:36.111059    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8"]
Jul 17 21:52:36.114274 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:36.114253    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-rqll8"]
Jul 17 21:52:36.529945 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:36.529869    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef719948-beff-42e9-acb1-7d1b78834816" path="/var/lib/kubelet/pods/ef719948-beff-42e9-acb1-7d1b78834816/volumes"
Jul 17 21:52:37.237593 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.237572    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:52:37.414560 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.414524    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-credential-socket\") pod \"b6cafea6-3c01-4096-8747-e32639fdcb4f\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") "
Jul 17 21:52:37.414750 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.414572    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-workload-socket\") pod \"b6cafea6-3c01-4096-8747-e32639fdcb4f\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") "
Jul 17 21:52:37.414750 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.414599    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-podinfo\") pod \"b6cafea6-3c01-4096-8747-e32639fdcb4f\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") "
Jul 17 21:52:37.414750 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.414676    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-envoy\") pod \"b6cafea6-3c01-4096-8747-e32639fdcb4f\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") "
Jul 17 21:52:37.414750 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.414715    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wg67x\" (UniqueName: \"kubernetes.io/projected/b6cafea6-3c01-4096-8747-e32639fdcb4f-kube-api-access-wg67x\") pod \"b6cafea6-3c01-4096-8747-e32639fdcb4f\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") "
Jul 17 21:52:37.414750 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.414737    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cafea6-3c01-4096-8747-e32639fdcb4f-istiod-ca-cert\") pod \"b6cafea6-3c01-4096-8747-e32639fdcb4f\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") "
Jul 17 21:52:37.415038 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.414760    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-token\") pod \"b6cafea6-3c01-4096-8747-e32639fdcb4f\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") "
Jul 17 21:52:37.415038 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.414785    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-workload-certs\") pod \"b6cafea6-3c01-4096-8747-e32639fdcb4f\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") "
Jul 17 21:52:37.415038 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.414871    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-data\") pod \"b6cafea6-3c01-4096-8747-e32639fdcb4f\" (UID: \"b6cafea6-3c01-4096-8747-e32639fdcb4f\") "
Jul 17 21:52:37.415038 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.414816    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b6cafea6-3c01-4096-8747-e32639fdcb4f" (UID: "b6cafea6-3c01-4096-8747-e32639fdcb4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:52:37.415252 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.414841    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b6cafea6-3c01-4096-8747-e32639fdcb4f" (UID: "b6cafea6-3c01-4096-8747-e32639fdcb4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:52:37.415252 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.415025    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b6cafea6-3c01-4096-8747-e32639fdcb4f" (UID: "b6cafea6-3c01-4096-8747-e32639fdcb4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:52:37.415252 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.415135    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:52:37.415252 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.415154    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:52:37.415252 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.415168    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:52:37.415540 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.415409    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6cafea6-3c01-4096-8747-e32639fdcb4f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b6cafea6-3c01-4096-8747-e32639fdcb4f" (UID: "b6cafea6-3c01-4096-8747-e32639fdcb4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:52:37.415796 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.415758    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-data" (OuterVolumeSpecName: "istio-data") pod "b6cafea6-3c01-4096-8747-e32639fdcb4f" (UID: "b6cafea6-3c01-4096-8747-e32639fdcb4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:52:37.417033 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.417013    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b6cafea6-3c01-4096-8747-e32639fdcb4f" (UID: "b6cafea6-3c01-4096-8747-e32639fdcb4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 21:52:37.417140 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.417047    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cafea6-3c01-4096-8747-e32639fdcb4f-kube-api-access-wg67x" (OuterVolumeSpecName: "kube-api-access-wg67x") pod "b6cafea6-3c01-4096-8747-e32639fdcb4f" (UID: "b6cafea6-3c01-4096-8747-e32639fdcb4f"). InnerVolumeSpecName "kube-api-access-wg67x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:52:37.417140 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.417104    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-token" (OuterVolumeSpecName: "istio-token") pod "b6cafea6-3c01-4096-8747-e32639fdcb4f" (UID: "b6cafea6-3c01-4096-8747-e32639fdcb4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:52:37.417140 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.417125    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b6cafea6-3c01-4096-8747-e32639fdcb4f" (UID: "b6cafea6-3c01-4096-8747-e32639fdcb4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:52:37.516201 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.516179    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:52:37.516324 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.516203    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg67x\" (UniqueName: \"kubernetes.io/projected/b6cafea6-3c01-4096-8747-e32639fdcb4f-kube-api-access-wg67x\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:52:37.516324 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.516218    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6cafea6-3c01-4096-8747-e32639fdcb4f-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:52:37.516324 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.516229    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:52:37.516324 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.516239    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:52:37.516324 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:37.516251    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6cafea6-3c01-4096-8747-e32639fdcb4f-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:52:38.100573 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:38.100539    2573 generic.go:356] "Generic (PLEG): container finished" podID="b6cafea6-3c01-4096-8747-e32639fdcb4f" containerID="650d86edadcdf879c29babbc873a88b9e2c67d5f0d7cbd597f8686f770c78f55" exitCode=0
Jul 17 21:52:38.100741 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:38.100581    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"
Jul 17 21:52:38.100741 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:38.100619    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w" event={"ID":"b6cafea6-3c01-4096-8747-e32639fdcb4f","Type":"ContainerDied","Data":"650d86edadcdf879c29babbc873a88b9e2c67d5f0d7cbd597f8686f770c78f55"}
Jul 17 21:52:38.100741 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:38.100671    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w" event={"ID":"b6cafea6-3c01-4096-8747-e32639fdcb4f","Type":"ContainerDied","Data":"7b8578453db05a2af788061196bc4f30e67847a55d531f971296fc184d758aa0"}
Jul 17 21:52:38.100741 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:38.100694    2573 scope.go:117] "RemoveContainer" containerID="650d86edadcdf879c29babbc873a88b9e2c67d5f0d7cbd597f8686f770c78f55"
Jul 17 21:52:38.109334 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:38.109319    2573 scope.go:117] "RemoveContainer" containerID="6814ce10ad3ee65df5255b0b0e8fddff288c96b38a5a3ae23b6a522ac18731f8"
Jul 17 21:52:38.117027 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:38.117012    2573 scope.go:117] "RemoveContainer" containerID="650d86edadcdf879c29babbc873a88b9e2c67d5f0d7cbd597f8686f770c78f55"
Jul 17 21:52:38.117265 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:52:38.117249    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650d86edadcdf879c29babbc873a88b9e2c67d5f0d7cbd597f8686f770c78f55\": container with ID starting with 650d86edadcdf879c29babbc873a88b9e2c67d5f0d7cbd597f8686f770c78f55 not found: ID does not exist" containerID="650d86edadcdf879c29babbc873a88b9e2c67d5f0d7cbd597f8686f770c78f55"
Jul 17 21:52:38.117330 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:38.117272    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650d86edadcdf879c29babbc873a88b9e2c67d5f0d7cbd597f8686f770c78f55"} err="failed to get container status \"650d86edadcdf879c29babbc873a88b9e2c67d5f0d7cbd597f8686f770c78f55\": rpc error: code = NotFound desc = could not find container \"650d86edadcdf879c29babbc873a88b9e2c67d5f0d7cbd597f8686f770c78f55\": container with ID starting with 650d86edadcdf879c29babbc873a88b9e2c67d5f0d7cbd597f8686f770c78f55 not found: ID does not exist"
Jul 17 21:52:38.117330 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:38.117301    2573 scope.go:117] "RemoveContainer" containerID="6814ce10ad3ee65df5255b0b0e8fddff288c96b38a5a3ae23b6a522ac18731f8"
Jul 17 21:52:38.117501 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:52:38.117484    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6814ce10ad3ee65df5255b0b0e8fddff288c96b38a5a3ae23b6a522ac18731f8\": container with ID starting with 6814ce10ad3ee65df5255b0b0e8fddff288c96b38a5a3ae23b6a522ac18731f8 not found: ID does not exist" containerID="6814ce10ad3ee65df5255b0b0e8fddff288c96b38a5a3ae23b6a522ac18731f8"
Jul 17 21:52:38.117544 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:38.117509    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6814ce10ad3ee65df5255b0b0e8fddff288c96b38a5a3ae23b6a522ac18731f8"} err="failed to get container status \"6814ce10ad3ee65df5255b0b0e8fddff288c96b38a5a3ae23b6a522ac18731f8\": rpc error: code = NotFound desc = could not find container \"6814ce10ad3ee65df5255b0b0e8fddff288c96b38a5a3ae23b6a522ac18731f8\": container with ID starting with 6814ce10ad3ee65df5255b0b0e8fddff288c96b38a5a3ae23b6a522ac18731f8 not found: ID does not exist"
Jul 17 21:52:38.121896 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:38.121877    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"]
Jul 17 21:52:38.124137 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:38.124118    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-62f6c7d6-openshift-default-6fb7fb9c75-6dq6w"]
Jul 17 21:52:38.530001 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:38.529928    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6cafea6-3c01-4096-8747-e32639fdcb4f" path="/var/lib/kubelet/pods/b6cafea6-3c01-4096-8747-e32639fdcb4f/volumes"
Jul 17 21:52:41.524456 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:41.524429    2573 scope.go:117] "RemoveContainer" containerID="dc466319f7bedc1152e2b2d9aa0e085ea492b788c5cf54cfe6b7840c14cbe928"
Jul 17 21:52:42.118406 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:42.118373    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls" event={"ID":"8b8b6132-c3de-4a7c-873d-b55e9cdc83e1","Type":"ContainerStarted","Data":"30df91e3afcc8977c1b1c89e35f5bde93690769c8a7df449d88914546cce40d9"}
Jul 17 21:52:43.124235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:43.124197    2573 generic.go:356] "Generic (PLEG): container finished" podID="8b8b6132-c3de-4a7c-873d-b55e9cdc83e1" containerID="30df91e3afcc8977c1b1c89e35f5bde93690769c8a7df449d88914546cce40d9" exitCode=6
Jul 17 21:52:43.124639 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:43.124261    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls" event={"ID":"8b8b6132-c3de-4a7c-873d-b55e9cdc83e1","Type":"ContainerDied","Data":"30df91e3afcc8977c1b1c89e35f5bde93690769c8a7df449d88914546cce40d9"}
Jul 17 21:52:43.124639 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:43.124325    2573 scope.go:117] "RemoveContainer" containerID="dc466319f7bedc1152e2b2d9aa0e085ea492b788c5cf54cfe6b7840c14cbe928"
Jul 17 21:52:43.124639 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:43.124556    2573 scope.go:117] "RemoveContainer" containerID="30df91e3afcc8977c1b1c89e35f5bde93690769c8a7df449d88914546cce40d9"
Jul 17 21:52:43.124786 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:52:43.124762    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls_odh-ai-gateway-infra(8b8b6132-c3de-4a7c-873d-b55e9cdc83e1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls" podUID="8b8b6132-c3de-4a7c-873d-b55e9cdc83e1"
Jul 17 21:52:43.150836 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:43.150804    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls"]
Jul 17 21:52:44.250059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:44.250033    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls"
Jul 17 21:52:44.370418 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:44.370387    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8b6132-c3de-4a7c-873d-b55e9cdc83e1-maas-api-service-ca\") pod \"8b8b6132-c3de-4a7c-873d-b55e9cdc83e1\" (UID: \"8b8b6132-c3de-4a7c-873d-b55e9cdc83e1\") "
Jul 17 21:52:44.370721 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:44.370701    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8b6132-c3de-4a7c-873d-b55e9cdc83e1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8b8b6132-c3de-4a7c-873d-b55e9cdc83e1" (UID: "8b8b6132-c3de-4a7c-873d-b55e9cdc83e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:52:44.471909 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:44.471828    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b8b6132-c3de-4a7c-873d-b55e9cdc83e1-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:52:45.134637 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:45.134612    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls"
Jul 17 21:52:45.134637 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:45.134615    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls" event={"ID":"8b8b6132-c3de-4a7c-873d-b55e9cdc83e1","Type":"ContainerDied","Data":"7a86e287fad266369da1becd79a315fe55e5fd4b2d07e92ecd90289ad90aeb6c"}
Jul 17 21:52:45.134874 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:45.134671    2573 scope.go:117] "RemoveContainer" containerID="30df91e3afcc8977c1b1c89e35f5bde93690769c8a7df449d88914546cce40d9"
Jul 17 21:52:45.149839 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:45.149816    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls"]
Jul 17 21:52:45.157565 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:45.157546    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-w2dls"]
Jul 17 21:52:46.529054 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:46.529024    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b8b6132-c3de-4a7c-873d-b55e9cdc83e1" path="/var/lib/kubelet/pods/8b8b6132-c3de-4a7c-873d-b55e9cdc83e1/volumes"
Jul 17 21:52:50.574608 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574580    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb"]
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574834    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b8b6132-c3de-4a7c-873d-b55e9cdc83e1" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574845    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b8b6132-c3de-4a7c-873d-b55e9cdc83e1" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574855    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1eca942-91b0-458b-ba7e-dd8d9ab1f107" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574860    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1eca942-91b0-458b-ba7e-dd8d9ab1f107" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574869    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6cafea6-3c01-4096-8747-e32639fdcb4f" containerName="istio-proxy"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574874    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6cafea6-3c01-4096-8747-e32639fdcb4f" containerName="istio-proxy"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574881    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef719948-beff-42e9-acb1-7d1b78834816" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574886    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef719948-beff-42e9-acb1-7d1b78834816" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574895    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6cafea6-3c01-4096-8747-e32639fdcb4f" containerName="istio-proxy"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574901    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6cafea6-3c01-4096-8747-e32639fdcb4f" containerName="istio-proxy"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574911    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1eca942-91b0-458b-ba7e-dd8d9ab1f107" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574916    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1eca942-91b0-458b-ba7e-dd8d9ab1f107" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574922    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef719948-beff-42e9-acb1-7d1b78834816" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574929    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef719948-beff-42e9-acb1-7d1b78834816" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574938    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef719948-beff-42e9-acb1-7d1b78834816" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574946    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef719948-beff-42e9-acb1-7d1b78834816" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574959    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b8b6132-c3de-4a7c-873d-b55e9cdc83e1" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.574964    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b8b6132-c3de-4a7c-873d-b55e9cdc83e1" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.575007    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6cafea6-3c01-4096-8747-e32639fdcb4f" containerName="istio-proxy"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.575017    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1eca942-91b0-458b-ba7e-dd8d9ab1f107" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.575027    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef719948-beff-42e9-acb1-7d1b78834816" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.575032    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef719948-beff-42e9-acb1-7d1b78834816" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.575038    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1eca942-91b0-458b-ba7e-dd8d9ab1f107" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.575044    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1eca942-91b0-458b-ba7e-dd8d9ab1f107" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.575050    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef719948-beff-42e9-acb1-7d1b78834816" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.575055    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b8b6132-c3de-4a7c-873d-b55e9cdc83e1" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.575061    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b8b6132-c3de-4a7c-873d-b55e9cdc83e1" containerName="revoke-keys"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.575067    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6cafea6-3c01-4096-8747-e32639fdcb4f" containerName="istio-proxy"
Jul 17 21:52:50.575059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.575072    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b8b6132-c3de-4a7c-873d-b55e9cdc83e1" containerName="revoke-keys"
Jul 17 21:52:50.579321 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.579299    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb"
Jul 17 21:52:50.581748 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.581726    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:52:50.581866 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.581812    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:52:50.585237 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.585205    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb"]
Jul 17 21:52:50.616874 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.616852    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bd1946d-bebf-4d26-bc30-ee19b61e827e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-tqrtb\" (UID: \"3bd1946d-bebf-4d26-bc30-ee19b61e827e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb"
Jul 17 21:52:50.717470 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.717443    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bd1946d-bebf-4d26-bc30-ee19b61e827e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-tqrtb\" (UID: \"3bd1946d-bebf-4d26-bc30-ee19b61e827e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb"
Jul 17 21:52:50.717956 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.717940    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bd1946d-bebf-4d26-bc30-ee19b61e827e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-tqrtb\" (UID: \"3bd1946d-bebf-4d26-bc30-ee19b61e827e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb"
Jul 17 21:52:50.891272 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:50.891250    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb"
Jul 17 21:52:51.015481 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.015456    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb"]
Jul 17 21:52:51.017258 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:52:51.017235    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd1946d_bebf_4d26_bc30_ee19b61e827e.slice/crio-652e5cdf6a592f0b8122b7f51284af073c8785c0424ca89763c68c1603eb173e WatchSource:0}: Error finding container 652e5cdf6a592f0b8122b7f51284af073c8785c0424ca89763c68c1603eb173e: Status 404 returned error can't find the container with id 652e5cdf6a592f0b8122b7f51284af073c8785c0424ca89763c68c1603eb173e
Jul 17 21:52:51.160358 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.160323    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb" event={"ID":"3bd1946d-bebf-4d26-bc30-ee19b61e827e","Type":"ContainerStarted","Data":"652e5cdf6a592f0b8122b7f51284af073c8785c0424ca89763c68c1603eb173e"}
Jul 17 21:52:51.519417 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.519344    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"]
Jul 17 21:52:51.519759 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.519741    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1eca942-91b0-458b-ba7e-dd8d9ab1f107" containerName="revoke-keys"
Jul 17 21:52:51.519866 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.519760    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1eca942-91b0-458b-ba7e-dd8d9ab1f107" containerName="revoke-keys"
Jul 17 21:52:51.519866 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.519792    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b8b6132-c3de-4a7c-873d-b55e9cdc83e1" containerName="revoke-keys"
Jul 17 21:52:51.519866 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.519801    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b8b6132-c3de-4a7c-873d-b55e9cdc83e1" containerName="revoke-keys"
Jul 17 21:52:51.523013 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.522990    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.523367 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.523341    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-envoy\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.523367 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.523371    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istiod-ca-cert\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.523539 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.523422    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-workload-socket\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.523539 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.523448    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-workload-certs\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.523642 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.523539    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-credential-socket\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.523642 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.523569    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2xqs\" (UniqueName: \"kubernetes.io/projected/abfebe49-260e-453a-88eb-1ce39cc2ad8a-kube-api-access-h2xqs\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.523642 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.523588    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-podinfo\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.523642 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.523608    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-data\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.523642 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.523625    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-token\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.525426 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.525409    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c39f8651-openshift-default-dockercfg-f5qmx\""
Jul 17 21:52:51.533077 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.533058    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"]
Jul 17 21:52:51.623911 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.623885    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-data\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.624310 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.623916    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-token\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.624310 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.623945    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-envoy\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.624310 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.624058    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istiod-ca-cert\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.624310 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.624105    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-workload-socket\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.624310 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.624170    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-workload-certs\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.624310 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.624241    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-credential-socket\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.624310 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.624269    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2xqs\" (UniqueName: \"kubernetes.io/projected/abfebe49-260e-453a-88eb-1ce39cc2ad8a-kube-api-access-h2xqs\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.624697 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.624327    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-data\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.624697 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.624367    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-podinfo\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.624697 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.624485    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-workload-socket\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.624697 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.624565    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-workload-certs\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.624697 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.624597    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-credential-socket\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.624697 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.624674    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istiod-ca-cert\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.626260 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.626238    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-envoy\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.626668 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.626652    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-podinfo\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.632385 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.632364    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-token\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.632463 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.632396    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2xqs\" (UniqueName: \"kubernetes.io/projected/abfebe49-260e-453a-88eb-1ce39cc2ad8a-kube-api-access-h2xqs\") pod \"e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") " pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.833791 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.833715    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:51.968976 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.968941    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"]
Jul 17 21:52:51.972777 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:52:51.972753    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabfebe49_260e_453a_88eb_1ce39cc2ad8a.slice/crio-abdff5e1080700d7b67ff104d48fd73e7c5914f8f088fedd920cc6366efcca0a WatchSource:0}: Error finding container abdff5e1080700d7b67ff104d48fd73e7c5914f8f088fedd920cc6366efcca0a: Status 404 returned error can't find the container with id abdff5e1080700d7b67ff104d48fd73e7c5914f8f088fedd920cc6366efcca0a
Jul 17 21:52:51.974999 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.974965    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:52:51.975073 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.975046    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:52:51.975109 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:51.975087    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:52:52.166827 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:52.166793    2573 generic.go:356] "Generic (PLEG): container finished" podID="3bd1946d-bebf-4d26-bc30-ee19b61e827e" containerID="ccff0a109031568ab418e5f9833398a271ecd0ac954e36d7efef22cfee324a18" exitCode=6
Jul 17 21:52:52.167015 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:52.166859    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb" event={"ID":"3bd1946d-bebf-4d26-bc30-ee19b61e827e","Type":"ContainerDied","Data":"ccff0a109031568ab418e5f9833398a271ecd0ac954e36d7efef22cfee324a18"}
Jul 17 21:52:52.167116 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:52.167102    2573 scope.go:117] "RemoveContainer" containerID="ccff0a109031568ab418e5f9833398a271ecd0ac954e36d7efef22cfee324a18"
Jul 17 21:52:52.173311 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:52.169128    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt" event={"ID":"abfebe49-260e-453a-88eb-1ce39cc2ad8a","Type":"ContainerStarted","Data":"5a439c15f4266e573c2e7cbd65b2404df868697488ffa2ebcfca07e0d6742e36"}
Jul 17 21:52:52.173311 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:52.169231    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt" event={"ID":"abfebe49-260e-453a-88eb-1ce39cc2ad8a","Type":"ContainerStarted","Data":"abdff5e1080700d7b67ff104d48fd73e7c5914f8f088fedd920cc6366efcca0a"}
Jul 17 21:52:52.213726 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:52.213679    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt" podStartSLOduration=1.213662926 podStartE2EDuration="1.213662926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:52:52.212829045 +0000 UTC m=+2672.176949656" watchObservedRunningTime="2026-07-17 21:52:52.213662926 +0000 UTC m=+2672.177783593"
Jul 17 21:52:52.834043 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:52.834020    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:53.174761 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:53.174729    2573 generic.go:356] "Generic (PLEG): container finished" podID="3bd1946d-bebf-4d26-bc30-ee19b61e827e" containerID="d47b1d86a3a2a5e4c89f77c473e718e92cda23bc9a8ecee69169f837b6644169" exitCode=6
Jul 17 21:52:53.174909 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:53.174814    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb" event={"ID":"3bd1946d-bebf-4d26-bc30-ee19b61e827e","Type":"ContainerDied","Data":"d47b1d86a3a2a5e4c89f77c473e718e92cda23bc9a8ecee69169f837b6644169"}
Jul 17 21:52:53.174909 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:53.174877    2573 scope.go:117] "RemoveContainer" containerID="ccff0a109031568ab418e5f9833398a271ecd0ac954e36d7efef22cfee324a18"
Jul 17 21:52:53.175098 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:53.175082    2573 scope.go:117] "RemoveContainer" containerID="d47b1d86a3a2a5e4c89f77c473e718e92cda23bc9a8ecee69169f837b6644169"
Jul 17 21:52:53.175332 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:52:53.175310    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-tqrtb_odh-ai-gateway-infra(3bd1946d-bebf-4d26-bc30-ee19b61e827e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb" podUID="3bd1946d-bebf-4d26-bc30-ee19b61e827e"
Jul 17 21:52:53.839653 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:53.839629    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:54.180737 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:54.180655    2573 scope.go:117] "RemoveContainer" containerID="d47b1d86a3a2a5e4c89f77c473e718e92cda23bc9a8ecee69169f837b6644169"
Jul 17 21:52:54.180737 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:54.180726    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:54.180951 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:52:54.180813    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-tqrtb_odh-ai-gateway-infra(3bd1946d-bebf-4d26-bc30-ee19b61e827e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb" podUID="3bd1946d-bebf-4d26-bc30-ee19b61e827e"
Jul 17 21:52:54.181649 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:54.181630    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:52:57.526903 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:57.526869    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg"]
Jul 17 21:52:57.531223 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:57.531205    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg"
Jul 17 21:52:57.536941 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:57.536916    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg"]
Jul 17 21:52:57.569797 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:57.569760    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c4db03c-28f6-44bc-a93f-ae8427b9662c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg\" (UID: \"2c4db03c-28f6-44bc-a93f-ae8427b9662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg"
Jul 17 21:52:57.670641 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:57.670607    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c4db03c-28f6-44bc-a93f-ae8427b9662c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg\" (UID: \"2c4db03c-28f6-44bc-a93f-ae8427b9662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg"
Jul 17 21:52:57.671190 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:57.671172    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c4db03c-28f6-44bc-a93f-ae8427b9662c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg\" (UID: \"2c4db03c-28f6-44bc-a93f-ae8427b9662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg"
Jul 17 21:52:57.843689 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:57.843665    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg"
Jul 17 21:52:57.969907 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:57.969884    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg"]
Jul 17 21:52:57.971942 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:52:57.971918    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4db03c_28f6_44bc_a93f_ae8427b9662c.slice/crio-6f5f61a1953045b37108a7991f6e826134fd4a9ba0c25b605b1afbbb35be9bd7 WatchSource:0}: Error finding container 6f5f61a1953045b37108a7991f6e826134fd4a9ba0c25b605b1afbbb35be9bd7: Status 404 returned error can't find the container with id 6f5f61a1953045b37108a7991f6e826134fd4a9ba0c25b605b1afbbb35be9bd7
Jul 17 21:52:58.197480 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:58.197390    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg" event={"ID":"2c4db03c-28f6-44bc-a93f-ae8427b9662c","Type":"ContainerStarted","Data":"6f5f61a1953045b37108a7991f6e826134fd4a9ba0c25b605b1afbbb35be9bd7"}
Jul 17 21:52:59.203120 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:59.203034    2573 generic.go:356] "Generic (PLEG): container finished" podID="2c4db03c-28f6-44bc-a93f-ae8427b9662c" containerID="6f865f028e6f60730469c622cdbb20b11877ae85da8e367ccdfda88ebcea8337" exitCode=6
Jul 17 21:52:59.203120 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:59.203099    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg" event={"ID":"2c4db03c-28f6-44bc-a93f-ae8427b9662c","Type":"ContainerDied","Data":"6f865f028e6f60730469c622cdbb20b11877ae85da8e367ccdfda88ebcea8337"}
Jul 17 21:52:59.203562 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:52:59.203347    2573 scope.go:117] "RemoveContainer" containerID="6f865f028e6f60730469c622cdbb20b11877ae85da8e367ccdfda88ebcea8337"
Jul 17 21:53:00.209511 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:00.209474    2573 generic.go:356] "Generic (PLEG): container finished" podID="2c4db03c-28f6-44bc-a93f-ae8427b9662c" containerID="39882aa18850ed2bd15ae89fe09386632ae39089f855f3922619bdc68ea69f82" exitCode=6
Jul 17 21:53:00.209967 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:00.209552    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg" event={"ID":"2c4db03c-28f6-44bc-a93f-ae8427b9662c","Type":"ContainerDied","Data":"39882aa18850ed2bd15ae89fe09386632ae39089f855f3922619bdc68ea69f82"}
Jul 17 21:53:00.209967 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:00.209605    2573 scope.go:117] "RemoveContainer" containerID="6f865f028e6f60730469c622cdbb20b11877ae85da8e367ccdfda88ebcea8337"
Jul 17 21:53:00.209967 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:00.209834    2573 scope.go:117] "RemoveContainer" containerID="39882aa18850ed2bd15ae89fe09386632ae39089f855f3922619bdc68ea69f82"
Jul 17 21:53:00.210127 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:53:00.210063    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg_odh-ai-gateway-infra(2c4db03c-28f6-44bc-a93f-ae8427b9662c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg" podUID="2c4db03c-28f6-44bc-a93f-ae8427b9662c"
Jul 17 21:53:01.214355 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:01.214326    2573 scope.go:117] "RemoveContainer" containerID="39882aa18850ed2bd15ae89fe09386632ae39089f855f3922619bdc68ea69f82"
Jul 17 21:53:01.214839 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:53:01.214564    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg_odh-ai-gateway-infra(2c4db03c-28f6-44bc-a93f-ae8427b9662c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg" podUID="2c4db03c-28f6-44bc-a93f-ae8427b9662c"
Jul 17 21:53:05.231530 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:05.231459    2573 generic.go:356] "Generic (PLEG): container finished" podID="abfebe49-260e-453a-88eb-1ce39cc2ad8a" containerID="5a439c15f4266e573c2e7cbd65b2404df868697488ffa2ebcfca07e0d6742e36" exitCode=0
Jul 17 21:53:05.231913 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:05.231528    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt" event={"ID":"abfebe49-260e-453a-88eb-1ce39cc2ad8a","Type":"ContainerDied","Data":"5a439c15f4266e573c2e7cbd65b2404df868697488ffa2ebcfca07e0d6742e36"}
Jul 17 21:53:05.233035 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:05.232573    2573 scope.go:117] "RemoveContainer" containerID="5a439c15f4266e573c2e7cbd65b2404df868697488ffa2ebcfca07e0d6742e36"
Jul 17 21:53:05.237933 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:05.237816    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:53:05.238018 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:05.237999    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:53:05.238096 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:05.238076    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:53:06.240461 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:06.240422    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt" event={"ID":"abfebe49-260e-453a-88eb-1ce39cc2ad8a","Type":"ContainerStarted","Data":"993aa9134493fa6ab87af37e0e6835c96c9b504452ec235469a07302c0e358e6"}
Jul 17 21:53:06.524540 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:06.524469    2573 scope.go:117] "RemoveContainer" containerID="d47b1d86a3a2a5e4c89f77c473e718e92cda23bc9a8ecee69169f837b6644169"
Jul 17 21:53:06.834669 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:06.834578    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:53:06.835603 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:06.835577    2573 patch_prober.go:28] interesting pod/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 17 21:53:06.835744 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:06.835633    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt" podUID="abfebe49-260e-453a-88eb-1ce39cc2ad8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 17 21:53:07.251025 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:07.250990    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb" event={"ID":"3bd1946d-bebf-4d26-bc30-ee19b61e827e","Type":"ContainerStarted","Data":"480bb246348c776a97a07c2ba632b6cd52edb53e7e6b1eb6f7730e904cbebf9b"}
Jul 17 21:53:07.269362 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:07.269305    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb" podStartSLOduration=17.269290571 podStartE2EDuration="17.269290571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:07.267482975 +0000 UTC m=+2687.231603584" watchObservedRunningTime="2026-07-17 21:53:07.269290571 +0000 UTC m=+2687.233411172"
Jul 17 21:53:07.835036 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:07.834995    2573 patch_prober.go:28] interesting pod/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 17 21:53:07.835240 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:07.835061    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt" podUID="abfebe49-260e-453a-88eb-1ce39cc2ad8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 17 21:53:08.258556 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:08.258509    2573 generic.go:356] "Generic (PLEG): container finished" podID="3bd1946d-bebf-4d26-bc30-ee19b61e827e" containerID="480bb246348c776a97a07c2ba632b6cd52edb53e7e6b1eb6f7730e904cbebf9b" exitCode=6
Jul 17 21:53:08.258959 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:08.258577    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb" event={"ID":"3bd1946d-bebf-4d26-bc30-ee19b61e827e","Type":"ContainerDied","Data":"480bb246348c776a97a07c2ba632b6cd52edb53e7e6b1eb6f7730e904cbebf9b"}
Jul 17 21:53:08.258959 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:08.258646    2573 scope.go:117] "RemoveContainer" containerID="d47b1d86a3a2a5e4c89f77c473e718e92cda23bc9a8ecee69169f837b6644169"
Jul 17 21:53:08.258959 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:08.258927    2573 scope.go:117] "RemoveContainer" containerID="480bb246348c776a97a07c2ba632b6cd52edb53e7e6b1eb6f7730e904cbebf9b"
Jul 17 21:53:08.259213 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:53:08.259187    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-tqrtb_odh-ai-gateway-infra(3bd1946d-bebf-4d26-bc30-ee19b61e827e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb" podUID="3bd1946d-bebf-4d26-bc30-ee19b61e827e"
Jul 17 21:53:08.286062 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:08.286041    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb"]
Jul 17 21:53:08.835031 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:08.834984    2573 patch_prober.go:28] interesting pod/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 17 21:53:08.835242 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:08.835054    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt" podUID="abfebe49-260e-453a-88eb-1ce39cc2ad8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 17 21:53:09.181051 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:09.181020    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:53:09.404519 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:09.404497    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb"
Jul 17 21:53:09.471946 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:09.471868    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bd1946d-bebf-4d26-bc30-ee19b61e827e-maas-api-service-ca\") pod \"3bd1946d-bebf-4d26-bc30-ee19b61e827e\" (UID: \"3bd1946d-bebf-4d26-bc30-ee19b61e827e\") "
Jul 17 21:53:09.472213 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:09.472193    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd1946d-bebf-4d26-bc30-ee19b61e827e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3bd1946d-bebf-4d26-bc30-ee19b61e827e" (UID: "3bd1946d-bebf-4d26-bc30-ee19b61e827e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:53:09.572953 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:09.572911    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bd1946d-bebf-4d26-bc30-ee19b61e827e-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:53:09.834930 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:09.834832    2573 patch_prober.go:28] interesting pod/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 17 21:53:09.836371 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:09.835170    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt" podUID="abfebe49-260e-453a-88eb-1ce39cc2ad8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 17 21:53:10.270154 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:10.270119    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb" event={"ID":"3bd1946d-bebf-4d26-bc30-ee19b61e827e","Type":"ContainerDied","Data":"652e5cdf6a592f0b8122b7f51284af073c8785c0424ca89763c68c1603eb173e"}
Jul 17 21:53:10.270154 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:10.270140    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb"
Jul 17 21:53:10.270458 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:10.270208    2573 scope.go:117] "RemoveContainer" containerID="480bb246348c776a97a07c2ba632b6cd52edb53e7e6b1eb6f7730e904cbebf9b"
Jul 17 21:53:10.289825 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:10.289804    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb"]
Jul 17 21:53:10.293451 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:10.293431    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-tqrtb"]
Jul 17 21:53:10.534718 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:10.534628    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bd1946d-bebf-4d26-bc30-ee19b61e827e" path="/var/lib/kubelet/pods/3bd1946d-bebf-4d26-bc30-ee19b61e827e/volumes"
Jul 17 21:53:10.839295 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:10.839248    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:53:10.840317 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:10.840275    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:53:14.525108 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:14.525076    2573 scope.go:117] "RemoveContainer" containerID="39882aa18850ed2bd15ae89fe09386632ae39089f855f3922619bdc68ea69f82"
Jul 17 21:53:15.299088 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:15.299053    2573 generic.go:356] "Generic (PLEG): container finished" podID="2c4db03c-28f6-44bc-a93f-ae8427b9662c" containerID="967bfb73db62b3f0f5f3375ba47a82c48919650849dc029ab73b96c002a99aab" exitCode=6
Jul 17 21:53:15.299245 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:15.299127    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg" event={"ID":"2c4db03c-28f6-44bc-a93f-ae8427b9662c","Type":"ContainerDied","Data":"967bfb73db62b3f0f5f3375ba47a82c48919650849dc029ab73b96c002a99aab"}
Jul 17 21:53:15.299245 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:15.299188    2573 scope.go:117] "RemoveContainer" containerID="39882aa18850ed2bd15ae89fe09386632ae39089f855f3922619bdc68ea69f82"
Jul 17 21:53:15.299515 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:15.299501    2573 scope.go:117] "RemoveContainer" containerID="967bfb73db62b3f0f5f3375ba47a82c48919650849dc029ab73b96c002a99aab"
Jul 17 21:53:15.299713 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:53:15.299693    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg_odh-ai-gateway-infra(2c4db03c-28f6-44bc-a93f-ae8427b9662c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg" podUID="2c4db03c-28f6-44bc-a93f-ae8427b9662c"
Jul 17 21:53:16.322829 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:16.322797    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg"]
Jul 17 21:53:16.451136 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:16.451113    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg"
Jul 17 21:53:16.525214 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:16.525184    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c4db03c-28f6-44bc-a93f-ae8427b9662c-maas-api-service-ca\") pod \"2c4db03c-28f6-44bc-a93f-ae8427b9662c\" (UID: \"2c4db03c-28f6-44bc-a93f-ae8427b9662c\") "
Jul 17 21:53:16.525545 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:16.525522    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c4db03c-28f6-44bc-a93f-ae8427b9662c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c4db03c-28f6-44bc-a93f-ae8427b9662c" (UID: "2c4db03c-28f6-44bc-a93f-ae8427b9662c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:53:16.626001 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:16.625975    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c4db03c-28f6-44bc-a93f-ae8427b9662c-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:53:17.315925 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:17.315897    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg"
Jul 17 21:53:17.316094 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:17.315896    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg" event={"ID":"2c4db03c-28f6-44bc-a93f-ae8427b9662c","Type":"ContainerDied","Data":"6f5f61a1953045b37108a7991f6e826134fd4a9ba0c25b605b1afbbb35be9bd7"}
Jul 17 21:53:17.316094 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:17.316028    2573 scope.go:117] "RemoveContainer" containerID="967bfb73db62b3f0f5f3375ba47a82c48919650849dc029ab73b96c002a99aab"
Jul 17 21:53:17.331408 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:17.331389    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg"]
Jul 17 21:53:17.341071 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:17.341051    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9jpwg"]
Jul 17 21:53:18.529911 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:18.529882    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c4db03c-28f6-44bc-a93f-ae8427b9662c" path="/var/lib/kubelet/pods/2c4db03c-28f6-44bc-a93f-ae8427b9662c/volumes"
Jul 17 21:53:40.617923 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.617894    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6"]
Jul 17 21:53:40.618391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.618204    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c4db03c-28f6-44bc-a93f-ae8427b9662c" containerName="revoke-keys"
Jul 17 21:53:40.618391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.618215    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c4db03c-28f6-44bc-a93f-ae8427b9662c" containerName="revoke-keys"
Jul 17 21:53:40.618391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.618224    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd1946d-bebf-4d26-bc30-ee19b61e827e" containerName="revoke-keys"
Jul 17 21:53:40.618391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.618230    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd1946d-bebf-4d26-bc30-ee19b61e827e" containerName="revoke-keys"
Jul 17 21:53:40.618391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.618236    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c4db03c-28f6-44bc-a93f-ae8427b9662c" containerName="revoke-keys"
Jul 17 21:53:40.618391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.618242    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c4db03c-28f6-44bc-a93f-ae8427b9662c" containerName="revoke-keys"
Jul 17 21:53:40.618391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.618252    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd1946d-bebf-4d26-bc30-ee19b61e827e" containerName="revoke-keys"
Jul 17 21:53:40.618391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.618258    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd1946d-bebf-4d26-bc30-ee19b61e827e" containerName="revoke-keys"
Jul 17 21:53:40.618391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.618268    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd1946d-bebf-4d26-bc30-ee19b61e827e" containerName="revoke-keys"
Jul 17 21:53:40.618391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.618273    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd1946d-bebf-4d26-bc30-ee19b61e827e" containerName="revoke-keys"
Jul 17 21:53:40.618391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.618337    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c4db03c-28f6-44bc-a93f-ae8427b9662c" containerName="revoke-keys"
Jul 17 21:53:40.618391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.618344    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c4db03c-28f6-44bc-a93f-ae8427b9662c" containerName="revoke-keys"
Jul 17 21:53:40.618391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.618349    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bd1946d-bebf-4d26-bc30-ee19b61e827e" containerName="revoke-keys"
Jul 17 21:53:40.618391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.618355    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bd1946d-bebf-4d26-bc30-ee19b61e827e" containerName="revoke-keys"
Jul 17 21:53:40.618391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.618361    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bd1946d-bebf-4d26-bc30-ee19b61e827e" containerName="revoke-keys"
Jul 17 21:53:40.621121 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.621105    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6"
Jul 17 21:53:40.623463 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.623442    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:53:40.623620 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.623542    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:53:40.628001 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.627870    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6"]
Jul 17 21:53:40.712943 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.712917    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e410de0-b077-47c4-8de7-17c58aa1d11f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-82hd6\" (UID: \"8e410de0-b077-47c4-8de7-17c58aa1d11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6"
Jul 17 21:53:40.814256 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.814227    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e410de0-b077-47c4-8de7-17c58aa1d11f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-82hd6\" (UID: \"8e410de0-b077-47c4-8de7-17c58aa1d11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6"
Jul 17 21:53:40.814818 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.814799    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e410de0-b077-47c4-8de7-17c58aa1d11f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-82hd6\" (UID: \"8e410de0-b077-47c4-8de7-17c58aa1d11f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6"
Jul 17 21:53:40.931992 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:40.931933    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6"
Jul 17 21:53:41.056048 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:41.056023    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6"]
Jul 17 21:53:41.058186 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:53:41.058163    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e410de0_b077_47c4_8de7_17c58aa1d11f.slice/crio-8b7194c7511d02c0ada3d4a7e805480666f6a08a74ac9208870658276d39bb06 WatchSource:0}: Error finding container 8b7194c7511d02c0ada3d4a7e805480666f6a08a74ac9208870658276d39bb06: Status 404 returned error can't find the container with id 8b7194c7511d02c0ada3d4a7e805480666f6a08a74ac9208870658276d39bb06
Jul 17 21:53:41.418299 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:41.418250    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6" event={"ID":"8e410de0-b077-47c4-8de7-17c58aa1d11f","Type":"ContainerStarted","Data":"ed1c923d8cdcff2432d5d149dc2b58daafaa4172763750d9e25f83a66ce04a63"}
Jul 17 21:53:41.418491 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:41.418350    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6" event={"ID":"8e410de0-b077-47c4-8de7-17c58aa1d11f","Type":"ContainerStarted","Data":"8b7194c7511d02c0ada3d4a7e805480666f6a08a74ac9208870658276d39bb06"}
Jul 17 21:53:41.434206 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:41.434153    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6" podStartSLOduration=1.434139389 podStartE2EDuration="1.434139389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:41.432541643 +0000 UTC m=+2721.396662252" watchObservedRunningTime="2026-07-17 21:53:41.434139389 +0000 UTC m=+2721.398259999"
Jul 17 21:53:42.424009 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:42.423972    2573 generic.go:356] "Generic (PLEG): container finished" podID="8e410de0-b077-47c4-8de7-17c58aa1d11f" containerID="ed1c923d8cdcff2432d5d149dc2b58daafaa4172763750d9e25f83a66ce04a63" exitCode=6
Jul 17 21:53:42.424473 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:42.424043    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6" event={"ID":"8e410de0-b077-47c4-8de7-17c58aa1d11f","Type":"ContainerDied","Data":"ed1c923d8cdcff2432d5d149dc2b58daafaa4172763750d9e25f83a66ce04a63"}
Jul 17 21:53:42.424473 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:42.424251    2573 scope.go:117] "RemoveContainer" containerID="ed1c923d8cdcff2432d5d149dc2b58daafaa4172763750d9e25f83a66ce04a63"
Jul 17 21:53:43.430170 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:43.430137    2573 generic.go:356] "Generic (PLEG): container finished" podID="8e410de0-b077-47c4-8de7-17c58aa1d11f" containerID="3a0c78e3e02b83eff763bf5063a12065d9a4c6be2f74d572744e49c4de000c66" exitCode=6
Jul 17 21:53:43.430633 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:43.430235    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6" event={"ID":"8e410de0-b077-47c4-8de7-17c58aa1d11f","Type":"ContainerDied","Data":"3a0c78e3e02b83eff763bf5063a12065d9a4c6be2f74d572744e49c4de000c66"}
Jul 17 21:53:43.430633 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:43.430327    2573 scope.go:117] "RemoveContainer" containerID="ed1c923d8cdcff2432d5d149dc2b58daafaa4172763750d9e25f83a66ce04a63"
Jul 17 21:53:43.430633 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:43.430485    2573 scope.go:117] "RemoveContainer" containerID="3a0c78e3e02b83eff763bf5063a12065d9a4c6be2f74d572744e49c4de000c66"
Jul 17 21:53:43.430757 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:53:43.430647    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-82hd6_odh-ai-gateway-infra(8e410de0-b077-47c4-8de7-17c58aa1d11f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6" podUID="8e410de0-b077-47c4-8de7-17c58aa1d11f"
Jul 17 21:53:44.435933 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:44.435908    2573 scope.go:117] "RemoveContainer" containerID="3a0c78e3e02b83eff763bf5063a12065d9a4c6be2f74d572744e49c4de000c66"
Jul 17 21:53:44.436335 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:53:44.436077    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-82hd6_odh-ai-gateway-infra(8e410de0-b077-47c4-8de7-17c58aa1d11f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6" podUID="8e410de0-b077-47c4-8de7-17c58aa1d11f"
Jul 17 21:53:47.572398 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:47.572370    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk"]
Jul 17 21:53:47.572779 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:47.572650    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c4db03c-28f6-44bc-a93f-ae8427b9662c" containerName="revoke-keys"
Jul 17 21:53:47.572779 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:47.572662    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c4db03c-28f6-44bc-a93f-ae8427b9662c" containerName="revoke-keys"
Jul 17 21:53:47.572779 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:47.572713    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c4db03c-28f6-44bc-a93f-ae8427b9662c" containerName="revoke-keys"
Jul 17 21:53:47.576755 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:47.576738    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk"
Jul 17 21:53:47.583650 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:47.583330    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk"]
Jul 17 21:53:47.665074 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:47.665047    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e38153f-6e69-41e2-8e87-96ab9db0636d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk\" (UID: \"2e38153f-6e69-41e2-8e87-96ab9db0636d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk"
Jul 17 21:53:47.766510 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:47.766482    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e38153f-6e69-41e2-8e87-96ab9db0636d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk\" (UID: \"2e38153f-6e69-41e2-8e87-96ab9db0636d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk"
Jul 17 21:53:47.767109 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:47.767091    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e38153f-6e69-41e2-8e87-96ab9db0636d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk\" (UID: \"2e38153f-6e69-41e2-8e87-96ab9db0636d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk"
Jul 17 21:53:47.888061 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:47.888036    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk"
Jul 17 21:53:48.015314 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:48.015268    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk"]
Jul 17 21:53:48.457095 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:48.457059    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk" event={"ID":"2e38153f-6e69-41e2-8e87-96ab9db0636d","Type":"ContainerStarted","Data":"8205ac5a2d72c96d7ed4e0fcd3cc574c88b27872b08fe2564768b13cd77f0b05"}
Jul 17 21:53:48.457242 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:48.457112    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk" event={"ID":"2e38153f-6e69-41e2-8e87-96ab9db0636d","Type":"ContainerStarted","Data":"5cb054cd54a99bbaea00f500465c9ebb182d52def61cdaa2257d11a99ab3e288"}
Jul 17 21:53:48.473586 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:48.473540    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk" podStartSLOduration=1.473526045 podStartE2EDuration="1.473526045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:48.472482193 +0000 UTC m=+2728.436602806" watchObservedRunningTime="2026-07-17 21:53:48.473526045 +0000 UTC m=+2728.437646655"
Jul 17 21:53:49.462517 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:49.462484    2573 generic.go:356] "Generic (PLEG): container finished" podID="2e38153f-6e69-41e2-8e87-96ab9db0636d" containerID="8205ac5a2d72c96d7ed4e0fcd3cc574c88b27872b08fe2564768b13cd77f0b05" exitCode=6
Jul 17 21:53:49.462882 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:49.462564    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk" event={"ID":"2e38153f-6e69-41e2-8e87-96ab9db0636d","Type":"ContainerDied","Data":"8205ac5a2d72c96d7ed4e0fcd3cc574c88b27872b08fe2564768b13cd77f0b05"}
Jul 17 21:53:49.462882 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:49.462753    2573 scope.go:117] "RemoveContainer" containerID="8205ac5a2d72c96d7ed4e0fcd3cc574c88b27872b08fe2564768b13cd77f0b05"
Jul 17 21:53:50.468515 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:50.468480    2573 generic.go:356] "Generic (PLEG): container finished" podID="2e38153f-6e69-41e2-8e87-96ab9db0636d" containerID="2b9d3392316fa2056d0ad1a4a91a501ff7a40e25403c99d2c0b3e529201d501e" exitCode=6
Jul 17 21:53:50.468919 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:50.468562    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk" event={"ID":"2e38153f-6e69-41e2-8e87-96ab9db0636d","Type":"ContainerDied","Data":"2b9d3392316fa2056d0ad1a4a91a501ff7a40e25403c99d2c0b3e529201d501e"}
Jul 17 21:53:50.468919 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:50.468622    2573 scope.go:117] "RemoveContainer" containerID="8205ac5a2d72c96d7ed4e0fcd3cc574c88b27872b08fe2564768b13cd77f0b05"
Jul 17 21:53:50.468919 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:50.468863    2573 scope.go:117] "RemoveContainer" containerID="2b9d3392316fa2056d0ad1a4a91a501ff7a40e25403c99d2c0b3e529201d501e"
Jul 17 21:53:50.469060 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:53:50.469042    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk_odh-ai-gateway-infra(2e38153f-6e69-41e2-8e87-96ab9db0636d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk" podUID="2e38153f-6e69-41e2-8e87-96ab9db0636d"
Jul 17 21:53:51.474101 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:51.474076    2573 scope.go:117] "RemoveContainer" containerID="2b9d3392316fa2056d0ad1a4a91a501ff7a40e25403c99d2c0b3e529201d501e"
Jul 17 21:53:51.474490 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:53:51.474239    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk_odh-ai-gateway-infra(2e38153f-6e69-41e2-8e87-96ab9db0636d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk" podUID="2e38153f-6e69-41e2-8e87-96ab9db0636d"
Jul 17 21:53:56.529556 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:56.529531    2573 scope.go:117] "RemoveContainer" containerID="3a0c78e3e02b83eff763bf5063a12065d9a4c6be2f74d572744e49c4de000c66"
Jul 17 21:53:57.505256 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:57.505177    2573 generic.go:356] "Generic (PLEG): container finished" podID="8e410de0-b077-47c4-8de7-17c58aa1d11f" containerID="6ab607f4f33c7138891c4987c6706d35bfee7800e0b5fd99367fbba74fa30ae5" exitCode=6
Jul 17 21:53:57.505256 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:57.505219    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6" event={"ID":"8e410de0-b077-47c4-8de7-17c58aa1d11f","Type":"ContainerDied","Data":"6ab607f4f33c7138891c4987c6706d35bfee7800e0b5fd99367fbba74fa30ae5"}
Jul 17 21:53:57.505467 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:57.505264    2573 scope.go:117] "RemoveContainer" containerID="3a0c78e3e02b83eff763bf5063a12065d9a4c6be2f74d572744e49c4de000c66"
Jul 17 21:53:57.505524 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:57.505512    2573 scope.go:117] "RemoveContainer" containerID="6ab607f4f33c7138891c4987c6706d35bfee7800e0b5fd99367fbba74fa30ae5"
Jul 17 21:53:57.505733 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:53:57.505709    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-82hd6_odh-ai-gateway-infra(8e410de0-b077-47c4-8de7-17c58aa1d11f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6" podUID="8e410de0-b077-47c4-8de7-17c58aa1d11f"
Jul 17 21:53:58.531278 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:58.531248    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6"]
Jul 17 21:53:58.655653 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:58.655632    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6"
Jul 17 21:53:58.743569 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:58.743544    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e410de0-b077-47c4-8de7-17c58aa1d11f-maas-api-service-ca\") pod \"8e410de0-b077-47c4-8de7-17c58aa1d11f\" (UID: \"8e410de0-b077-47c4-8de7-17c58aa1d11f\") "
Jul 17 21:53:58.743862 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:58.743842    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e410de0-b077-47c4-8de7-17c58aa1d11f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8e410de0-b077-47c4-8de7-17c58aa1d11f" (UID: "8e410de0-b077-47c4-8de7-17c58aa1d11f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:53:58.844001 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:58.843977    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e410de0-b077-47c4-8de7-17c58aa1d11f-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:53:59.515687 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:59.515656    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6" event={"ID":"8e410de0-b077-47c4-8de7-17c58aa1d11f","Type":"ContainerDied","Data":"8b7194c7511d02c0ada3d4a7e805480666f6a08a74ac9208870658276d39bb06"}
Jul 17 21:53:59.515839 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:59.515709    2573 scope.go:117] "RemoveContainer" containerID="6ab607f4f33c7138891c4987c6706d35bfee7800e0b5fd99367fbba74fa30ae5"
Jul 17 21:53:59.515839 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:59.515662    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6"
Jul 17 21:53:59.538208 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:59.538186    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6"]
Jul 17 21:53:59.544685 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:53:59.544666    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-82hd6"]
Jul 17 21:54:00.534085 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:00.534051    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e410de0-b077-47c4-8de7-17c58aa1d11f" path="/var/lib/kubelet/pods/8e410de0-b077-47c4-8de7-17c58aa1d11f/volumes"
Jul 17 21:54:04.524835 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:04.524803    2573 scope.go:117] "RemoveContainer" containerID="2b9d3392316fa2056d0ad1a4a91a501ff7a40e25403c99d2c0b3e529201d501e"
Jul 17 21:54:05.540822 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:05.540742    2573 generic.go:356] "Generic (PLEG): container finished" podID="2e38153f-6e69-41e2-8e87-96ab9db0636d" containerID="d160642f4fea8196c0a956048d2b47094769cbacd54fee9972e87e9d3696e712" exitCode=6
Jul 17 21:54:05.540822 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:05.540792    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk" event={"ID":"2e38153f-6e69-41e2-8e87-96ab9db0636d","Type":"ContainerDied","Data":"d160642f4fea8196c0a956048d2b47094769cbacd54fee9972e87e9d3696e712"}
Jul 17 21:54:05.541311 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:05.540838    2573 scope.go:117] "RemoveContainer" containerID="2b9d3392316fa2056d0ad1a4a91a501ff7a40e25403c99d2c0b3e529201d501e"
Jul 17 21:54:05.541311 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:05.541084    2573 scope.go:117] "RemoveContainer" containerID="d160642f4fea8196c0a956048d2b47094769cbacd54fee9972e87e9d3696e712"
Jul 17 21:54:05.541311 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:54:05.541267    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk_odh-ai-gateway-infra(2e38153f-6e69-41e2-8e87-96ab9db0636d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk" podUID="2e38153f-6e69-41e2-8e87-96ab9db0636d"
Jul 17 21:54:06.563934 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:06.563900    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk"]
Jul 17 21:54:06.703955 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:06.703932    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk"
Jul 17 21:54:06.811637 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:06.811605    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e38153f-6e69-41e2-8e87-96ab9db0636d-maas-api-service-ca\") pod \"2e38153f-6e69-41e2-8e87-96ab9db0636d\" (UID: \"2e38153f-6e69-41e2-8e87-96ab9db0636d\") "
Jul 17 21:54:06.811996 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:06.811967    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e38153f-6e69-41e2-8e87-96ab9db0636d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2e38153f-6e69-41e2-8e87-96ab9db0636d" (UID: "2e38153f-6e69-41e2-8e87-96ab9db0636d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:54:06.912684 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:06.912655    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e38153f-6e69-41e2-8e87-96ab9db0636d-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:54:07.550814 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:07.550787    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk"
Jul 17 21:54:07.550978 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:07.550786    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk" event={"ID":"2e38153f-6e69-41e2-8e87-96ab9db0636d","Type":"ContainerDied","Data":"5cb054cd54a99bbaea00f500465c9ebb182d52def61cdaa2257d11a99ab3e288"}
Jul 17 21:54:07.550978 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:07.550915    2573 scope.go:117] "RemoveContainer" containerID="d160642f4fea8196c0a956048d2b47094769cbacd54fee9972e87e9d3696e712"
Jul 17 21:54:07.575018 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:07.574994    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk"]
Jul 17 21:54:07.581066 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:07.581045    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-r7vjk"]
Jul 17 21:54:08.529728 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:08.529697    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e38153f-6e69-41e2-8e87-96ab9db0636d" path="/var/lib/kubelet/pods/2e38153f-6e69-41e2-8e87-96ab9db0636d/volumes"
Jul 17 21:54:09.447916 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.447887    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f"]
Jul 17 21:54:09.448305 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.448161    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e410de0-b077-47c4-8de7-17c58aa1d11f" containerName="revoke-keys"
Jul 17 21:54:09.448305 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.448173    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e410de0-b077-47c4-8de7-17c58aa1d11f" containerName="revoke-keys"
Jul 17 21:54:09.448305 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.448180    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e38153f-6e69-41e2-8e87-96ab9db0636d" containerName="revoke-keys"
Jul 17 21:54:09.448305 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.448188    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e38153f-6e69-41e2-8e87-96ab9db0636d" containerName="revoke-keys"
Jul 17 21:54:09.448305 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.448196    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e410de0-b077-47c4-8de7-17c58aa1d11f" containerName="revoke-keys"
Jul 17 21:54:09.448305 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.448201    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e410de0-b077-47c4-8de7-17c58aa1d11f" containerName="revoke-keys"
Jul 17 21:54:09.448305 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.448229    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e38153f-6e69-41e2-8e87-96ab9db0636d" containerName="revoke-keys"
Jul 17 21:54:09.448305 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.448234    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e38153f-6e69-41e2-8e87-96ab9db0636d" containerName="revoke-keys"
Jul 17 21:54:09.448305 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.448245    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e38153f-6e69-41e2-8e87-96ab9db0636d" containerName="revoke-keys"
Jul 17 21:54:09.448305 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.448251    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e38153f-6e69-41e2-8e87-96ab9db0636d" containerName="revoke-keys"
Jul 17 21:54:09.448619 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.448320    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e38153f-6e69-41e2-8e87-96ab9db0636d" containerName="revoke-keys"
Jul 17 21:54:09.448619 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.448331    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e410de0-b077-47c4-8de7-17c58aa1d11f" containerName="revoke-keys"
Jul 17 21:54:09.448619 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.448341    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e38153f-6e69-41e2-8e87-96ab9db0636d" containerName="revoke-keys"
Jul 17 21:54:09.448619 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.448346    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e38153f-6e69-41e2-8e87-96ab9db0636d" containerName="revoke-keys"
Jul 17 21:54:09.448619 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.448354    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e410de0-b077-47c4-8de7-17c58aa1d11f" containerName="revoke-keys"
Jul 17 21:54:09.452270 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.452254    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f"
Jul 17 21:54:09.454466 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.454439    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:54:09.454620 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.454448    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:54:09.458767 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.458746    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f"]
Jul 17 21:54:09.631239 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.631204    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afec7be1-fbe1-4a09-86db-120b72f2a74a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f\" (UID: \"afec7be1-fbe1-4a09-86db-120b72f2a74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f"
Jul 17 21:54:09.732543 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.732469    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afec7be1-fbe1-4a09-86db-120b72f2a74a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f\" (UID: \"afec7be1-fbe1-4a09-86db-120b72f2a74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f"
Jul 17 21:54:09.733071 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.733053    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afec7be1-fbe1-4a09-86db-120b72f2a74a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f\" (UID: \"afec7be1-fbe1-4a09-86db-120b72f2a74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f"
Jul 17 21:54:09.763811 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.763787    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f"
Jul 17 21:54:09.917744 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:09.917708    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f"]
Jul 17 21:54:09.921205 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:54:09.921178    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafec7be1_fbe1_4a09_86db_120b72f2a74a.slice/crio-d0874db7def3f5e24073841dd56ccf47c47a119f9d88fed1698ab4d344b6420f WatchSource:0}: Error finding container d0874db7def3f5e24073841dd56ccf47c47a119f9d88fed1698ab4d344b6420f: Status 404 returned error can't find the container with id d0874db7def3f5e24073841dd56ccf47c47a119f9d88fed1698ab4d344b6420f
Jul 17 21:54:10.564340 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:10.564306    2573 generic.go:356] "Generic (PLEG): container finished" podID="afec7be1-fbe1-4a09-86db-120b72f2a74a" containerID="06ae9c3fe53772037d7e3d2c86eb8d094d47dfbe65957308acb91e6c9cd5baa1" exitCode=6
Jul 17 21:54:10.564685 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:10.564357    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f" event={"ID":"afec7be1-fbe1-4a09-86db-120b72f2a74a","Type":"ContainerDied","Data":"06ae9c3fe53772037d7e3d2c86eb8d094d47dfbe65957308acb91e6c9cd5baa1"}
Jul 17 21:54:10.564685 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:10.564392    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f" event={"ID":"afec7be1-fbe1-4a09-86db-120b72f2a74a","Type":"ContainerStarted","Data":"d0874db7def3f5e24073841dd56ccf47c47a119f9d88fed1698ab4d344b6420f"}
Jul 17 21:54:10.564685 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:10.564551    2573 scope.go:117] "RemoveContainer" containerID="06ae9c3fe53772037d7e3d2c86eb8d094d47dfbe65957308acb91e6c9cd5baa1"
Jul 17 21:54:11.570841 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:11.570767    2573 generic.go:356] "Generic (PLEG): container finished" podID="afec7be1-fbe1-4a09-86db-120b72f2a74a" containerID="2f1f8798a9a8860253e7c8521c7f1c7613054409b2aa03bbea26e02bf73c25bd" exitCode=6
Jul 17 21:54:11.570841 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:11.570800    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f" event={"ID":"afec7be1-fbe1-4a09-86db-120b72f2a74a","Type":"ContainerDied","Data":"2f1f8798a9a8860253e7c8521c7f1c7613054409b2aa03bbea26e02bf73c25bd"}
Jul 17 21:54:11.571372 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:11.570847    2573 scope.go:117] "RemoveContainer" containerID="06ae9c3fe53772037d7e3d2c86eb8d094d47dfbe65957308acb91e6c9cd5baa1"
Jul 17 21:54:11.571372 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:11.571063    2573 scope.go:117] "RemoveContainer" containerID="2f1f8798a9a8860253e7c8521c7f1c7613054409b2aa03bbea26e02bf73c25bd"
Jul 17 21:54:11.571372 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:54:11.571268    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f_odh-ai-gateway-infra(afec7be1-fbe1-4a09-86db-120b72f2a74a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f" podUID="afec7be1-fbe1-4a09-86db-120b72f2a74a"
Jul 17 21:54:12.575732 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:12.575707    2573 scope.go:117] "RemoveContainer" containerID="2f1f8798a9a8860253e7c8521c7f1c7613054409b2aa03bbea26e02bf73c25bd"
Jul 17 21:54:12.576100 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:54:12.575862    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f_odh-ai-gateway-infra(afec7be1-fbe1-4a09-86db-120b72f2a74a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f" podUID="afec7be1-fbe1-4a09-86db-120b72f2a74a"
Jul 17 21:54:23.525232 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:23.525202    2573 scope.go:117] "RemoveContainer" containerID="2f1f8798a9a8860253e7c8521c7f1c7613054409b2aa03bbea26e02bf73c25bd"
Jul 17 21:54:24.623691 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:24.623659    2573 generic.go:356] "Generic (PLEG): container finished" podID="afec7be1-fbe1-4a09-86db-120b72f2a74a" containerID="f57fbc3278c5df6eef5d565e3e42dfc15fa6ee71997a4f6cc0b6d74827eceeaf" exitCode=6
Jul 17 21:54:24.624095 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:24.623731    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f" event={"ID":"afec7be1-fbe1-4a09-86db-120b72f2a74a","Type":"ContainerDied","Data":"f57fbc3278c5df6eef5d565e3e42dfc15fa6ee71997a4f6cc0b6d74827eceeaf"}
Jul 17 21:54:24.624095 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:24.623789    2573 scope.go:117] "RemoveContainer" containerID="2f1f8798a9a8860253e7c8521c7f1c7613054409b2aa03bbea26e02bf73c25bd"
Jul 17 21:54:24.624095 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:24.624039    2573 scope.go:117] "RemoveContainer" containerID="f57fbc3278c5df6eef5d565e3e42dfc15fa6ee71997a4f6cc0b6d74827eceeaf"
Jul 17 21:54:24.624270 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:54:24.624246    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f_odh-ai-gateway-infra(afec7be1-fbe1-4a09-86db-120b72f2a74a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f" podUID="afec7be1-fbe1-4a09-86db-120b72f2a74a"
Jul 17 21:54:25.648395 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:25.648364    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f"]
Jul 17 21:54:25.776089 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:25.776063    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f"
Jul 17 21:54:25.845657 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:25.845632    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afec7be1-fbe1-4a09-86db-120b72f2a74a-maas-api-service-ca\") pod \"afec7be1-fbe1-4a09-86db-120b72f2a74a\" (UID: \"afec7be1-fbe1-4a09-86db-120b72f2a74a\") "
Jul 17 21:54:25.845985 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:25.845961    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afec7be1-fbe1-4a09-86db-120b72f2a74a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "afec7be1-fbe1-4a09-86db-120b72f2a74a" (UID: "afec7be1-fbe1-4a09-86db-120b72f2a74a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:54:25.946235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:25.946155    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afec7be1-fbe1-4a09-86db-120b72f2a74a-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:54:26.634828 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:26.634794    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f" event={"ID":"afec7be1-fbe1-4a09-86db-120b72f2a74a","Type":"ContainerDied","Data":"d0874db7def3f5e24073841dd56ccf47c47a119f9d88fed1698ab4d344b6420f"}
Jul 17 21:54:26.634828 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:26.634826    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f"
Jul 17 21:54:26.635039 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:26.634850    2573 scope.go:117] "RemoveContainer" containerID="f57fbc3278c5df6eef5d565e3e42dfc15fa6ee71997a4f6cc0b6d74827eceeaf"
Jul 17 21:54:26.649760 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:26.649740    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f"]
Jul 17 21:54:26.654001 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:26.653981    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-c5z6f"]
Jul 17 21:54:28.530469 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:28.530436    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afec7be1-fbe1-4a09-86db-120b72f2a74a" path="/var/lib/kubelet/pods/afec7be1-fbe1-4a09-86db-120b72f2a74a/volumes"
Jul 17 21:54:30.660416 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:30.660383    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c"]
Jul 17 21:54:30.660778 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:30.660646    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afec7be1-fbe1-4a09-86db-120b72f2a74a" containerName="revoke-keys"
Jul 17 21:54:30.660778 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:30.660656    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="afec7be1-fbe1-4a09-86db-120b72f2a74a" containerName="revoke-keys"
Jul 17 21:54:30.660778 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:30.660663    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afec7be1-fbe1-4a09-86db-120b72f2a74a" containerName="revoke-keys"
Jul 17 21:54:30.660778 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:30.660669    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="afec7be1-fbe1-4a09-86db-120b72f2a74a" containerName="revoke-keys"
Jul 17 21:54:30.660778 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:30.660677    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e410de0-b077-47c4-8de7-17c58aa1d11f" containerName="revoke-keys"
Jul 17 21:54:30.660778 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:30.660683    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e410de0-b077-47c4-8de7-17c58aa1d11f" containerName="revoke-keys"
Jul 17 21:54:30.660778 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:30.660734    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="afec7be1-fbe1-4a09-86db-120b72f2a74a" containerName="revoke-keys"
Jul 17 21:54:30.660778 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:30.660742    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e410de0-b077-47c4-8de7-17c58aa1d11f" containerName="revoke-keys"
Jul 17 21:54:30.660778 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:30.660748    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="afec7be1-fbe1-4a09-86db-120b72f2a74a" containerName="revoke-keys"
Jul 17 21:54:30.663465 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:30.663448    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c"
Jul 17 21:54:30.665883 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:30.665862    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:54:30.666015 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:30.665864    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:54:30.671059 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:30.671038    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c"]
Jul 17 21:54:30.789534 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:30.784974    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c4180e-7780-440d-8354-8ca02bbb892a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-c5j5c\" (UID: \"39c4180e-7780-440d-8354-8ca02bbb892a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c"
Jul 17 21:54:30.885770 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:30.885734    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c4180e-7780-440d-8354-8ca02bbb892a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-c5j5c\" (UID: \"39c4180e-7780-440d-8354-8ca02bbb892a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c"
Jul 17 21:54:30.886345 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:30.886326    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c4180e-7780-440d-8354-8ca02bbb892a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-c5j5c\" (UID: \"39c4180e-7780-440d-8354-8ca02bbb892a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c"
Jul 17 21:54:30.974152 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:30.974095    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c"
Jul 17 21:54:31.097752 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:31.097724    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c"]
Jul 17 21:54:31.099808 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:54:31.099783    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c4180e_7780_440d_8354_8ca02bbb892a.slice/crio-8e21277721045b83e85108249cfd9a0772989ad666de8db1b63f6ccab5069bb6 WatchSource:0}: Error finding container 8e21277721045b83e85108249cfd9a0772989ad666de8db1b63f6ccab5069bb6: Status 404 returned error can't find the container with id 8e21277721045b83e85108249cfd9a0772989ad666de8db1b63f6ccab5069bb6
Jul 17 21:54:31.657487 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:31.657410    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c" event={"ID":"39c4180e-7780-440d-8354-8ca02bbb892a","Type":"ContainerStarted","Data":"212b80e199da9b21af9ea0d2177783eee92d35aeac92cf17075f494453aa6ede"}
Jul 17 21:54:31.657487 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:31.657466    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c" event={"ID":"39c4180e-7780-440d-8354-8ca02bbb892a","Type":"ContainerStarted","Data":"8e21277721045b83e85108249cfd9a0772989ad666de8db1b63f6ccab5069bb6"}
Jul 17 21:54:31.673579 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:31.673536    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c" podStartSLOduration=1.673523011 podStartE2EDuration="1.673523011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:54:31.671769535 +0000 UTC m=+2771.635890145" watchObservedRunningTime="2026-07-17 21:54:31.673523011 +0000 UTC m=+2771.637643620"
Jul 17 21:54:32.662209 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:32.662174    2573 generic.go:356] "Generic (PLEG): container finished" podID="39c4180e-7780-440d-8354-8ca02bbb892a" containerID="212b80e199da9b21af9ea0d2177783eee92d35aeac92cf17075f494453aa6ede" exitCode=6
Jul 17 21:54:32.662485 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:32.662233    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c" event={"ID":"39c4180e-7780-440d-8354-8ca02bbb892a","Type":"ContainerDied","Data":"212b80e199da9b21af9ea0d2177783eee92d35aeac92cf17075f494453aa6ede"}
Jul 17 21:54:32.662485 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:32.662450    2573 scope.go:117] "RemoveContainer" containerID="212b80e199da9b21af9ea0d2177783eee92d35aeac92cf17075f494453aa6ede"
Jul 17 21:54:33.668712 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:33.668679    2573 generic.go:356] "Generic (PLEG): container finished" podID="39c4180e-7780-440d-8354-8ca02bbb892a" containerID="67f19699bba10f52c583ccb23bc1a0c2ef9339d022feabeab2149bd874db2f36" exitCode=6
Jul 17 21:54:33.669106 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:33.668716    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c" event={"ID":"39c4180e-7780-440d-8354-8ca02bbb892a","Type":"ContainerDied","Data":"67f19699bba10f52c583ccb23bc1a0c2ef9339d022feabeab2149bd874db2f36"}
Jul 17 21:54:33.669106 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:33.668758    2573 scope.go:117] "RemoveContainer" containerID="212b80e199da9b21af9ea0d2177783eee92d35aeac92cf17075f494453aa6ede"
Jul 17 21:54:33.669106 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:33.668928    2573 scope.go:117] "RemoveContainer" containerID="67f19699bba10f52c583ccb23bc1a0c2ef9339d022feabeab2149bd874db2f36"
Jul 17 21:54:33.669238 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:54:33.669140    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-c5j5c_odh-ai-gateway-infra(39c4180e-7780-440d-8354-8ca02bbb892a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c" podUID="39c4180e-7780-440d-8354-8ca02bbb892a"
Jul 17 21:54:34.674107 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:34.674080    2573 scope.go:117] "RemoveContainer" containerID="67f19699bba10f52c583ccb23bc1a0c2ef9339d022feabeab2149bd874db2f36"
Jul 17 21:54:34.674480 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:54:34.674265    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-c5j5c_odh-ai-gateway-infra(39c4180e-7780-440d-8354-8ca02bbb892a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c" podUID="39c4180e-7780-440d-8354-8ca02bbb892a"
Jul 17 21:54:38.763609 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:38.763576    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"]
Jul 17 21:54:38.764046 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:38.763773    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt" podUID="abfebe49-260e-453a-88eb-1ce39cc2ad8a" containerName="istio-proxy" containerID="cri-o://993aa9134493fa6ab87af37e0e6835c96c9b504452ec235469a07302c0e358e6" gracePeriod=30
Jul 17 21:54:40.841661 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:40.841625    2573 patch_prober.go:28] interesting pod/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 21:54:40.842030 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:40.841692    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt" podUID="abfebe49-260e-453a-88eb-1ce39cc2ad8a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 21:54:44.003733 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.003710    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:54:44.089457 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.089432    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-token\") pod \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") "
Jul 17 21:54:44.089618 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.089470    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-workload-certs\") pod \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") "
Jul 17 21:54:44.089618 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.089487    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-podinfo\") pod \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") "
Jul 17 21:54:44.089618 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.089519    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2xqs\" (UniqueName: \"kubernetes.io/projected/abfebe49-260e-453a-88eb-1ce39cc2ad8a-kube-api-access-h2xqs\") pod \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") "
Jul 17 21:54:44.089618 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.089542    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-workload-socket\") pod \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") "
Jul 17 21:54:44.089618 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.089571    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-data\") pod \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") "
Jul 17 21:54:44.089618 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.089596    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-envoy\") pod \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") "
Jul 17 21:54:44.089618 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.089614    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-credential-socket\") pod \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") "
Jul 17 21:54:44.089954 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.089702    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istiod-ca-cert\") pod \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\" (UID: \"abfebe49-260e-453a-88eb-1ce39cc2ad8a\") "
Jul 17 21:54:44.089954 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.089753    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "abfebe49-260e-453a-88eb-1ce39cc2ad8a" (UID: "abfebe49-260e-453a-88eb-1ce39cc2ad8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:54:44.090071 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.089970    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "abfebe49-260e-453a-88eb-1ce39cc2ad8a" (UID: "abfebe49-260e-453a-88eb-1ce39cc2ad8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:54:44.090071 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.089975    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:54:44.090179 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.090086    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "abfebe49-260e-453a-88eb-1ce39cc2ad8a" (UID: "abfebe49-260e-453a-88eb-1ce39cc2ad8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:54:44.090232 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.090184    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "abfebe49-260e-453a-88eb-1ce39cc2ad8a" (UID: "abfebe49-260e-453a-88eb-1ce39cc2ad8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:54:44.090962 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.090932    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-data" (OuterVolumeSpecName: "istio-data") pod "abfebe49-260e-453a-88eb-1ce39cc2ad8a" (UID: "abfebe49-260e-453a-88eb-1ce39cc2ad8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:54:44.091772 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.091748    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-token" (OuterVolumeSpecName: "istio-token") pod "abfebe49-260e-453a-88eb-1ce39cc2ad8a" (UID: "abfebe49-260e-453a-88eb-1ce39cc2ad8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:54:44.091867 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.091854    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "abfebe49-260e-453a-88eb-1ce39cc2ad8a" (UID: "abfebe49-260e-453a-88eb-1ce39cc2ad8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 21:54:44.091941 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.091917    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abfebe49-260e-453a-88eb-1ce39cc2ad8a-kube-api-access-h2xqs" (OuterVolumeSpecName: "kube-api-access-h2xqs") pod "abfebe49-260e-453a-88eb-1ce39cc2ad8a" (UID: "abfebe49-260e-453a-88eb-1ce39cc2ad8a"). InnerVolumeSpecName "kube-api-access-h2xqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:54:44.091941 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.091929    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "abfebe49-260e-453a-88eb-1ce39cc2ad8a" (UID: "abfebe49-260e-453a-88eb-1ce39cc2ad8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:54:44.190547 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.190524    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:54:44.190547 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.190543    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:54:44.190687 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.190559    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:54:44.190687 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.190569    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:54:44.190687 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.190577    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:54:44.190687 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.190584    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abfebe49-260e-453a-88eb-1ce39cc2ad8a-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:54:44.190687 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.190592    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2xqs\" (UniqueName: \"kubernetes.io/projected/abfebe49-260e-453a-88eb-1ce39cc2ad8a-kube-api-access-h2xqs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:54:44.190687 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.190600    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abfebe49-260e-453a-88eb-1ce39cc2ad8a-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:54:44.716365 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.716327    2573 generic.go:356] "Generic (PLEG): container finished" podID="abfebe49-260e-453a-88eb-1ce39cc2ad8a" containerID="993aa9134493fa6ab87af37e0e6835c96c9b504452ec235469a07302c0e358e6" exitCode=0
Jul 17 21:54:44.716549 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.716386    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"
Jul 17 21:54:44.716549 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.716416    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt" event={"ID":"abfebe49-260e-453a-88eb-1ce39cc2ad8a","Type":"ContainerDied","Data":"993aa9134493fa6ab87af37e0e6835c96c9b504452ec235469a07302c0e358e6"}
Jul 17 21:54:44.716549 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.716480    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt" event={"ID":"abfebe49-260e-453a-88eb-1ce39cc2ad8a","Type":"ContainerDied","Data":"abdff5e1080700d7b67ff104d48fd73e7c5914f8f088fedd920cc6366efcca0a"}
Jul 17 21:54:44.716549 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.716511    2573 scope.go:117] "RemoveContainer" containerID="993aa9134493fa6ab87af37e0e6835c96c9b504452ec235469a07302c0e358e6"
Jul 17 21:54:44.724596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.724580    2573 scope.go:117] "RemoveContainer" containerID="5a439c15f4266e573c2e7cbd65b2404df868697488ffa2ebcfca07e0d6742e36"
Jul 17 21:54:44.731713 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.731698    2573 scope.go:117] "RemoveContainer" containerID="993aa9134493fa6ab87af37e0e6835c96c9b504452ec235469a07302c0e358e6"
Jul 17 21:54:44.731937 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:54:44.731920    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"993aa9134493fa6ab87af37e0e6835c96c9b504452ec235469a07302c0e358e6\": container with ID starting with 993aa9134493fa6ab87af37e0e6835c96c9b504452ec235469a07302c0e358e6 not found: ID does not exist" containerID="993aa9134493fa6ab87af37e0e6835c96c9b504452ec235469a07302c0e358e6"
Jul 17 21:54:44.731994 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.731943    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"993aa9134493fa6ab87af37e0e6835c96c9b504452ec235469a07302c0e358e6"} err="failed to get container status \"993aa9134493fa6ab87af37e0e6835c96c9b504452ec235469a07302c0e358e6\": rpc error: code = NotFound desc = could not find container \"993aa9134493fa6ab87af37e0e6835c96c9b504452ec235469a07302c0e358e6\": container with ID starting with 993aa9134493fa6ab87af37e0e6835c96c9b504452ec235469a07302c0e358e6 not found: ID does not exist"
Jul 17 21:54:44.731994 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.731957    2573 scope.go:117] "RemoveContainer" containerID="5a439c15f4266e573c2e7cbd65b2404df868697488ffa2ebcfca07e0d6742e36"
Jul 17 21:54:44.732161 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:54:44.732145    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a439c15f4266e573c2e7cbd65b2404df868697488ffa2ebcfca07e0d6742e36\": container with ID starting with 5a439c15f4266e573c2e7cbd65b2404df868697488ffa2ebcfca07e0d6742e36 not found: ID does not exist" containerID="5a439c15f4266e573c2e7cbd65b2404df868697488ffa2ebcfca07e0d6742e36"
Jul 17 21:54:44.732203 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.732166    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a439c15f4266e573c2e7cbd65b2404df868697488ffa2ebcfca07e0d6742e36"} err="failed to get container status \"5a439c15f4266e573c2e7cbd65b2404df868697488ffa2ebcfca07e0d6742e36\": rpc error: code = NotFound desc = could not find container \"5a439c15f4266e573c2e7cbd65b2404df868697488ffa2ebcfca07e0d6742e36\": container with ID starting with 5a439c15f4266e573c2e7cbd65b2404df868697488ffa2ebcfca07e0d6742e36 not found: ID does not exist"
Jul 17 21:54:44.735193 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.735175    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"]
Jul 17 21:54:44.739321 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:44.739301    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c39f8651-openshift-default-c6b9c6cf9-nvmkt"]
Jul 17 21:54:46.529839 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:46.529802    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abfebe49-260e-453a-88eb-1ce39cc2ad8a" path="/var/lib/kubelet/pods/abfebe49-260e-453a-88eb-1ce39cc2ad8a/volumes"
Jul 17 21:54:47.621032 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:47.621002    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs"]
Jul 17 21:54:47.621392 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:47.621299    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afec7be1-fbe1-4a09-86db-120b72f2a74a" containerName="revoke-keys"
Jul 17 21:54:47.621392 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:47.621314    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="afec7be1-fbe1-4a09-86db-120b72f2a74a" containerName="revoke-keys"
Jul 17 21:54:47.621392 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:47.621335    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abfebe49-260e-453a-88eb-1ce39cc2ad8a" containerName="istio-proxy"
Jul 17 21:54:47.621392 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:47.621340    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="abfebe49-260e-453a-88eb-1ce39cc2ad8a" containerName="istio-proxy"
Jul 17 21:54:47.621580 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:47.621395    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="abfebe49-260e-453a-88eb-1ce39cc2ad8a" containerName="istio-proxy"
Jul 17 21:54:47.621580 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:47.621408    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="abfebe49-260e-453a-88eb-1ce39cc2ad8a" containerName="istio-proxy"
Jul 17 21:54:47.621580 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:47.621418    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="afec7be1-fbe1-4a09-86db-120b72f2a74a" containerName="revoke-keys"
Jul 17 21:54:47.625859 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:47.625843    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs"
Jul 17 21:54:47.631341 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:47.631315    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs"]
Jul 17 21:54:47.719663 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:47.719634    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e15970-601c-4380-8ae0-953d50d7f8cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs\" (UID: \"d4e15970-601c-4380-8ae0-953d50d7f8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs"
Jul 17 21:54:47.820759 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:47.820732    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e15970-601c-4380-8ae0-953d50d7f8cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs\" (UID: \"d4e15970-601c-4380-8ae0-953d50d7f8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs"
Jul 17 21:54:47.821303 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:47.821271    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e15970-601c-4380-8ae0-953d50d7f8cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs\" (UID: \"d4e15970-601c-4380-8ae0-953d50d7f8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs"
Jul 17 21:54:47.937589 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:47.937513    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs"
Jul 17 21:54:48.079617 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:48.079595    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs"]
Jul 17 21:54:48.081606 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:54:48.081574    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e15970_601c_4380_8ae0_953d50d7f8cb.slice/crio-05630833cf05ad7745d961e0e5f499ec01ad743bbce47738810ba07d6d38ee21 WatchSource:0}: Error finding container 05630833cf05ad7745d961e0e5f499ec01ad743bbce47738810ba07d6d38ee21: Status 404 returned error can't find the container with id 05630833cf05ad7745d961e0e5f499ec01ad743bbce47738810ba07d6d38ee21
Jul 17 21:54:48.736322 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:48.736278    2573 generic.go:356] "Generic (PLEG): container finished" podID="d4e15970-601c-4380-8ae0-953d50d7f8cb" containerID="c0c6f1f57313679ad4a31612d54cacdc442a47ce3dd3e2e3757ab6cf1516a45d" exitCode=6
Jul 17 21:54:48.736687 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:48.736316    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs" event={"ID":"d4e15970-601c-4380-8ae0-953d50d7f8cb","Type":"ContainerDied","Data":"c0c6f1f57313679ad4a31612d54cacdc442a47ce3dd3e2e3757ab6cf1516a45d"}
Jul 17 21:54:48.736687 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:48.736368    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs" event={"ID":"d4e15970-601c-4380-8ae0-953d50d7f8cb","Type":"ContainerStarted","Data":"05630833cf05ad7745d961e0e5f499ec01ad743bbce47738810ba07d6d38ee21"}
Jul 17 21:54:48.736687 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:48.736541    2573 scope.go:117] "RemoveContainer" containerID="c0c6f1f57313679ad4a31612d54cacdc442a47ce3dd3e2e3757ab6cf1516a45d"
Jul 17 21:54:49.524582 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:49.524557    2573 scope.go:117] "RemoveContainer" containerID="67f19699bba10f52c583ccb23bc1a0c2ef9339d022feabeab2149bd874db2f36"
Jul 17 21:54:49.742999 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:49.742963    2573 generic.go:356] "Generic (PLEG): container finished" podID="d4e15970-601c-4380-8ae0-953d50d7f8cb" containerID="3569de77fd232374c59126bd97a19173956d3f728d5ffe22b8c1e8ad3f1e45f5" exitCode=6
Jul 17 21:54:49.743440 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:49.743048    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs" event={"ID":"d4e15970-601c-4380-8ae0-953d50d7f8cb","Type":"ContainerDied","Data":"3569de77fd232374c59126bd97a19173956d3f728d5ffe22b8c1e8ad3f1e45f5"}
Jul 17 21:54:49.743440 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:49.743123    2573 scope.go:117] "RemoveContainer" containerID="c0c6f1f57313679ad4a31612d54cacdc442a47ce3dd3e2e3757ab6cf1516a45d"
Jul 17 21:54:49.743440 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:49.743316    2573 scope.go:117] "RemoveContainer" containerID="3569de77fd232374c59126bd97a19173956d3f728d5ffe22b8c1e8ad3f1e45f5"
Jul 17 21:54:49.743613 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:54:49.743526    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs_odh-ai-gateway-infra(d4e15970-601c-4380-8ae0-953d50d7f8cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs" podUID="d4e15970-601c-4380-8ae0-953d50d7f8cb"
Jul 17 21:54:50.749140 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:50.749109    2573 generic.go:356] "Generic (PLEG): container finished" podID="39c4180e-7780-440d-8354-8ca02bbb892a" containerID="4271250647ee91d5ad085a27a07c0fe73b59a37d2081a75e0e49bb4e358ea607" exitCode=6
Jul 17 21:54:50.749556 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:50.749182    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c" event={"ID":"39c4180e-7780-440d-8354-8ca02bbb892a","Type":"ContainerDied","Data":"4271250647ee91d5ad085a27a07c0fe73b59a37d2081a75e0e49bb4e358ea607"}
Jul 17 21:54:50.749556 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:50.749244    2573 scope.go:117] "RemoveContainer" containerID="67f19699bba10f52c583ccb23bc1a0c2ef9339d022feabeab2149bd874db2f36"
Jul 17 21:54:50.749556 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:50.749510    2573 scope.go:117] "RemoveContainer" containerID="4271250647ee91d5ad085a27a07c0fe73b59a37d2081a75e0e49bb4e358ea607"
Jul 17 21:54:50.749751 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:54:50.749711    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-c5j5c_odh-ai-gateway-infra(39c4180e-7780-440d-8354-8ca02bbb892a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c" podUID="39c4180e-7780-440d-8354-8ca02bbb892a"
Jul 17 21:54:50.750965 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:50.750911    2573 scope.go:117] "RemoveContainer" containerID="3569de77fd232374c59126bd97a19173956d3f728d5ffe22b8c1e8ad3f1e45f5"
Jul 17 21:54:50.751104 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:54:50.751088    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs_odh-ai-gateway-infra(d4e15970-601c-4380-8ae0-953d50d7f8cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs" podUID="d4e15970-601c-4380-8ae0-953d50d7f8cb"
Jul 17 21:54:51.774331 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:51.774301    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c"]
Jul 17 21:54:51.901469 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:51.901443    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c"
Jul 17 21:54:52.052950 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:52.052873    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c4180e-7780-440d-8354-8ca02bbb892a-maas-api-service-ca\") pod \"39c4180e-7780-440d-8354-8ca02bbb892a\" (UID: \"39c4180e-7780-440d-8354-8ca02bbb892a\") "
Jul 17 21:54:52.053265 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:52.053235    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c4180e-7780-440d-8354-8ca02bbb892a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "39c4180e-7780-440d-8354-8ca02bbb892a" (UID: "39c4180e-7780-440d-8354-8ca02bbb892a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:54:52.153962 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:52.153934    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c4180e-7780-440d-8354-8ca02bbb892a-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:54:52.761598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:52.761568    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c" event={"ID":"39c4180e-7780-440d-8354-8ca02bbb892a","Type":"ContainerDied","Data":"8e21277721045b83e85108249cfd9a0772989ad666de8db1b63f6ccab5069bb6"}
Jul 17 21:54:52.761598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:52.761588    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c"
Jul 17 21:54:52.761812 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:52.761627    2573 scope.go:117] "RemoveContainer" containerID="4271250647ee91d5ad085a27a07c0fe73b59a37d2081a75e0e49bb4e358ea607"
Jul 17 21:54:52.776782 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:52.776759    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c"]
Jul 17 21:54:52.778935 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:52.778914    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-c5j5c"]
Jul 17 21:54:54.529520 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:54:54.529486    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39c4180e-7780-440d-8354-8ca02bbb892a" path="/var/lib/kubelet/pods/39c4180e-7780-440d-8354-8ca02bbb892a/volumes"
Jul 17 21:55:03.525061 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:03.525031    2573 scope.go:117] "RemoveContainer" containerID="3569de77fd232374c59126bd97a19173956d3f728d5ffe22b8c1e8ad3f1e45f5"
Jul 17 21:55:03.807361 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:03.807263    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs" event={"ID":"d4e15970-601c-4380-8ae0-953d50d7f8cb","Type":"ContainerStarted","Data":"2bc3892cea8951424a856e38884edec7c1f4afaaec5017dc9d1618d10a87e419"}
Jul 17 21:55:03.822715 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:03.822656    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs" podStartSLOduration=16.8226371 podStartE2EDuration="16.8226371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:55:03.821623405 +0000 UTC m=+2803.785744016" watchObservedRunningTime="2026-07-17 21:55:03.8226371 +0000 UTC m=+2803.786757711"
Jul 17 21:55:04.813068 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:04.813031    2573 generic.go:356] "Generic (PLEG): container finished" podID="d4e15970-601c-4380-8ae0-953d50d7f8cb" containerID="2bc3892cea8951424a856e38884edec7c1f4afaaec5017dc9d1618d10a87e419" exitCode=6
Jul 17 21:55:04.813068 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:04.813070    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs" event={"ID":"d4e15970-601c-4380-8ae0-953d50d7f8cb","Type":"ContainerDied","Data":"2bc3892cea8951424a856e38884edec7c1f4afaaec5017dc9d1618d10a87e419"}
Jul 17 21:55:04.813617 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:04.813128    2573 scope.go:117] "RemoveContainer" containerID="3569de77fd232374c59126bd97a19173956d3f728d5ffe22b8c1e8ad3f1e45f5"
Jul 17 21:55:04.813617 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:04.813384    2573 scope.go:117] "RemoveContainer" containerID="2bc3892cea8951424a856e38884edec7c1f4afaaec5017dc9d1618d10a87e419"
Jul 17 21:55:04.813617 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:55:04.813573    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs_odh-ai-gateway-infra(d4e15970-601c-4380-8ae0-953d50d7f8cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs" podUID="d4e15970-601c-4380-8ae0-953d50d7f8cb"
Jul 17 21:55:04.832780 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:04.832758    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs"]
Jul 17 21:55:05.398850 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:05.398817    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54"]
Jul 17 21:55:05.399093 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:05.399082    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c4180e-7780-440d-8354-8ca02bbb892a" containerName="revoke-keys"
Jul 17 21:55:05.399142 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:05.399095    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c4180e-7780-440d-8354-8ca02bbb892a" containerName="revoke-keys"
Jul 17 21:55:05.399142 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:05.399114    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abfebe49-260e-453a-88eb-1ce39cc2ad8a" containerName="istio-proxy"
Jul 17 21:55:05.399142 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:05.399119    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="abfebe49-260e-453a-88eb-1ce39cc2ad8a" containerName="istio-proxy"
Jul 17 21:55:05.399142 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:05.399129    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c4180e-7780-440d-8354-8ca02bbb892a" containerName="revoke-keys"
Jul 17 21:55:05.399142 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:05.399135    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c4180e-7780-440d-8354-8ca02bbb892a" containerName="revoke-keys"
Jul 17 21:55:05.399322 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:05.399179    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c4180e-7780-440d-8354-8ca02bbb892a" containerName="revoke-keys"
Jul 17 21:55:05.399322 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:05.399187    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c4180e-7780-440d-8354-8ca02bbb892a" containerName="revoke-keys"
Jul 17 21:55:05.403145 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:05.403129    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54"
Jul 17 21:55:05.410831 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:05.410808    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54"]
Jul 17 21:55:05.557273 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:05.557237    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2784b49a-07bf-4e0b-850c-eb66c4c3c893-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-svs54\" (UID: \"2784b49a-07bf-4e0b-850c-eb66c4c3c893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54"
Jul 17 21:55:05.658134 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:05.658056    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2784b49a-07bf-4e0b-850c-eb66c4c3c893-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-svs54\" (UID: \"2784b49a-07bf-4e0b-850c-eb66c4c3c893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54"
Jul 17 21:55:05.658670 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:05.658652    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2784b49a-07bf-4e0b-850c-eb66c4c3c893-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-svs54\" (UID: \"2784b49a-07bf-4e0b-850c-eb66c4c3c893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54"
Jul 17 21:55:05.713620 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:05.713596    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54"
Jul 17 21:55:05.839248 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:05.839123    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54"]
Jul 17 21:55:05.841923 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:55:05.841896    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2784b49a_07bf_4e0b_850c_eb66c4c3c893.slice/crio-f1fe5a1f90ba1ac325fbce2ddac91383266c1164c8c3ce28a713edd4b6b82180 WatchSource:0}: Error finding container f1fe5a1f90ba1ac325fbce2ddac91383266c1164c8c3ce28a713edd4b6b82180: Status 404 returned error can't find the container with id f1fe5a1f90ba1ac325fbce2ddac91383266c1164c8c3ce28a713edd4b6b82180
Jul 17 21:55:05.952160 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:05.952140    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs"
Jul 17 21:55:06.060329 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:06.060299    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e15970-601c-4380-8ae0-953d50d7f8cb-maas-api-service-ca\") pod \"d4e15970-601c-4380-8ae0-953d50d7f8cb\" (UID: \"d4e15970-601c-4380-8ae0-953d50d7f8cb\") "
Jul 17 21:55:06.060644 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:06.060622    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e15970-601c-4380-8ae0-953d50d7f8cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4e15970-601c-4380-8ae0-953d50d7f8cb" (UID: "d4e15970-601c-4380-8ae0-953d50d7f8cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:55:06.161336 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:06.161306    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e15970-601c-4380-8ae0-953d50d7f8cb-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:55:06.825084 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:06.825052    2573 generic.go:356] "Generic (PLEG): container finished" podID="2784b49a-07bf-4e0b-850c-eb66c4c3c893" containerID="3e9c0897526bafb5deb7c3408657052e4670ebf3bd2ef35392a49e02d9edcd10" exitCode=6
Jul 17 21:55:06.825242 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:06.825143    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54" event={"ID":"2784b49a-07bf-4e0b-850c-eb66c4c3c893","Type":"ContainerDied","Data":"3e9c0897526bafb5deb7c3408657052e4670ebf3bd2ef35392a49e02d9edcd10"}
Jul 17 21:55:06.825242 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:06.825199    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54" event={"ID":"2784b49a-07bf-4e0b-850c-eb66c4c3c893","Type":"ContainerStarted","Data":"f1fe5a1f90ba1ac325fbce2ddac91383266c1164c8c3ce28a713edd4b6b82180"}
Jul 17 21:55:06.825436 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:06.825311    2573 scope.go:117] "RemoveContainer" containerID="3e9c0897526bafb5deb7c3408657052e4670ebf3bd2ef35392a49e02d9edcd10"
Jul 17 21:55:06.826537 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:06.826516    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs" event={"ID":"d4e15970-601c-4380-8ae0-953d50d7f8cb","Type":"ContainerDied","Data":"05630833cf05ad7745d961e0e5f499ec01ad743bbce47738810ba07d6d38ee21"}
Jul 17 21:55:06.826660 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:06.826542    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs"
Jul 17 21:55:06.826660 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:06.826579    2573 scope.go:117] "RemoveContainer" containerID="2bc3892cea8951424a856e38884edec7c1f4afaaec5017dc9d1618d10a87e419"
Jul 17 21:55:06.861657 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:06.861560    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs"]
Jul 17 21:55:06.865833 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:06.865812    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-j46zs"]
Jul 17 21:55:07.832855 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:07.832772    2573 generic.go:356] "Generic (PLEG): container finished" podID="2784b49a-07bf-4e0b-850c-eb66c4c3c893" containerID="35b1cd398568a06b0455dffe007b9af59716ab3f822289f0af8e94e5ad709429" exitCode=6
Jul 17 21:55:07.833024 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:07.832859    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54" event={"ID":"2784b49a-07bf-4e0b-850c-eb66c4c3c893","Type":"ContainerDied","Data":"35b1cd398568a06b0455dffe007b9af59716ab3f822289f0af8e94e5ad709429"}
Jul 17 21:55:07.833024 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:07.832914    2573 scope.go:117] "RemoveContainer" containerID="3e9c0897526bafb5deb7c3408657052e4670ebf3bd2ef35392a49e02d9edcd10"
Jul 17 21:55:07.833132 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:07.833023    2573 scope.go:117] "RemoveContainer" containerID="35b1cd398568a06b0455dffe007b9af59716ab3f822289f0af8e94e5ad709429"
Jul 17 21:55:07.833261 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:55:07.833239    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-svs54_odh-ai-gateway-infra(2784b49a-07bf-4e0b-850c-eb66c4c3c893)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54" podUID="2784b49a-07bf-4e0b-850c-eb66c4c3c893"
Jul 17 21:55:08.529979 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:08.529948    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4e15970-601c-4380-8ae0-953d50d7f8cb" path="/var/lib/kubelet/pods/d4e15970-601c-4380-8ae0-953d50d7f8cb/volumes"
Jul 17 21:55:08.838943 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:08.838908    2573 scope.go:117] "RemoveContainer" containerID="35b1cd398568a06b0455dffe007b9af59716ab3f822289f0af8e94e5ad709429"
Jul 17 21:55:08.839125 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:55:08.839102    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-svs54_odh-ai-gateway-infra(2784b49a-07bf-4e0b-850c-eb66c4c3c893)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54" podUID="2784b49a-07bf-4e0b-850c-eb66c4c3c893"
Jul 17 21:55:21.524672 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:21.524643    2573 scope.go:117] "RemoveContainer" containerID="35b1cd398568a06b0455dffe007b9af59716ab3f822289f0af8e94e5ad709429"
Jul 17 21:55:21.892146 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:21.892118    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54" event={"ID":"2784b49a-07bf-4e0b-850c-eb66c4c3c893","Type":"ContainerStarted","Data":"990091f9e0bed98a9d2245b393641cddacdecc0d3d36cf32e5f143bd0cc8cbb2"}
Jul 17 21:55:21.908774 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:21.908731    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54" podStartSLOduration=16.908717354 podStartE2EDuration="16.908717354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:55:21.906743501 +0000 UTC m=+2821.870864111" watchObservedRunningTime="2026-07-17 21:55:21.908717354 +0000 UTC m=+2821.872837963"
Jul 17 21:55:22.897600 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:22.897563    2573 generic.go:356] "Generic (PLEG): container finished" podID="2784b49a-07bf-4e0b-850c-eb66c4c3c893" containerID="990091f9e0bed98a9d2245b393641cddacdecc0d3d36cf32e5f143bd0cc8cbb2" exitCode=6
Jul 17 21:55:22.897995 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:22.897629    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54" event={"ID":"2784b49a-07bf-4e0b-850c-eb66c4c3c893","Type":"ContainerDied","Data":"990091f9e0bed98a9d2245b393641cddacdecc0d3d36cf32e5f143bd0cc8cbb2"}
Jul 17 21:55:22.897995 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:22.897687    2573 scope.go:117] "RemoveContainer" containerID="35b1cd398568a06b0455dffe007b9af59716ab3f822289f0af8e94e5ad709429"
Jul 17 21:55:22.897995 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:22.897899    2573 scope.go:117] "RemoveContainer" containerID="990091f9e0bed98a9d2245b393641cddacdecc0d3d36cf32e5f143bd0cc8cbb2"
Jul 17 21:55:22.898130 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:55:22.898107    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-svs54_odh-ai-gateway-infra(2784b49a-07bf-4e0b-850c-eb66c4c3c893)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54" podUID="2784b49a-07bf-4e0b-850c-eb66c4c3c893"
Jul 17 21:55:22.917707 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:22.917685    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54"]
Jul 17 21:55:24.029924 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:24.029897    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54"
Jul 17 21:55:24.091167 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:24.091138    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2784b49a-07bf-4e0b-850c-eb66c4c3c893-maas-api-service-ca\") pod \"2784b49a-07bf-4e0b-850c-eb66c4c3c893\" (UID: \"2784b49a-07bf-4e0b-850c-eb66c4c3c893\") "
Jul 17 21:55:24.091514 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:24.091491    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2784b49a-07bf-4e0b-850c-eb66c4c3c893-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2784b49a-07bf-4e0b-850c-eb66c4c3c893" (UID: "2784b49a-07bf-4e0b-850c-eb66c4c3c893"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:55:24.192104 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:24.192040    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2784b49a-07bf-4e0b-850c-eb66c4c3c893-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:55:24.907913 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:24.907884    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54"
Jul 17 21:55:24.907913 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:24.907889    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54" event={"ID":"2784b49a-07bf-4e0b-850c-eb66c4c3c893","Type":"ContainerDied","Data":"f1fe5a1f90ba1ac325fbce2ddac91383266c1164c8c3ce28a713edd4b6b82180"}
Jul 17 21:55:24.908154 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:24.907955    2573 scope.go:117] "RemoveContainer" containerID="990091f9e0bed98a9d2245b393641cddacdecc0d3d36cf32e5f143bd0cc8cbb2"
Jul 17 21:55:24.923151 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:24.923128    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54"]
Jul 17 21:55:24.929710 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:24.929690    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-svs54"]
Jul 17 21:55:26.530080 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:26.530052    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2784b49a-07bf-4e0b-850c-eb66c4c3c893" path="/var/lib/kubelet/pods/2784b49a-07bf-4e0b-850c-eb66c4c3c893/volumes"
Jul 17 21:55:30.710862 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.710827    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm"]
Jul 17 21:55:30.711235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.711096    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e15970-601c-4380-8ae0-953d50d7f8cb" containerName="revoke-keys"
Jul 17 21:55:30.711235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.711107    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e15970-601c-4380-8ae0-953d50d7f8cb" containerName="revoke-keys"
Jul 17 21:55:30.711235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.711116    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2784b49a-07bf-4e0b-850c-eb66c4c3c893" containerName="revoke-keys"
Jul 17 21:55:30.711235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.711121    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2784b49a-07bf-4e0b-850c-eb66c4c3c893" containerName="revoke-keys"
Jul 17 21:55:30.711235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.711131    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2784b49a-07bf-4e0b-850c-eb66c4c3c893" containerName="revoke-keys"
Jul 17 21:55:30.711235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.711136    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2784b49a-07bf-4e0b-850c-eb66c4c3c893" containerName="revoke-keys"
Jul 17 21:55:30.711235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.711144    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e15970-601c-4380-8ae0-953d50d7f8cb" containerName="revoke-keys"
Jul 17 21:55:30.711235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.711149    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e15970-601c-4380-8ae0-953d50d7f8cb" containerName="revoke-keys"
Jul 17 21:55:30.711235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.711160    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c4180e-7780-440d-8354-8ca02bbb892a" containerName="revoke-keys"
Jul 17 21:55:30.711235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.711165    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c4180e-7780-440d-8354-8ca02bbb892a" containerName="revoke-keys"
Jul 17 21:55:30.711235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.711210    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4e15970-601c-4380-8ae0-953d50d7f8cb" containerName="revoke-keys"
Jul 17 21:55:30.711235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.711217    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c4180e-7780-440d-8354-8ca02bbb892a" containerName="revoke-keys"
Jul 17 21:55:30.711235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.711225    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2784b49a-07bf-4e0b-850c-eb66c4c3c893" containerName="revoke-keys"
Jul 17 21:55:30.711235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.711230    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4e15970-601c-4380-8ae0-953d50d7f8cb" containerName="revoke-keys"
Jul 17 21:55:30.711235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.711235    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2784b49a-07bf-4e0b-850c-eb66c4c3c893" containerName="revoke-keys"
Jul 17 21:55:30.711235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.711240    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2784b49a-07bf-4e0b-850c-eb66c4c3c893" containerName="revoke-keys"
Jul 17 21:55:30.714000 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.713984    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm"
Jul 17 21:55:30.716443 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.716415    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:55:30.716724 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.716706    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:55:30.721878 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.721854    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm"]
Jul 17 21:55:30.738981 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.738953    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb4b2d7f-dca3-4a50-957f-719f6a2cdc95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-v5kpm\" (UID: \"bb4b2d7f-dca3-4a50-957f-719f6a2cdc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm"
Jul 17 21:55:30.839924 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.839904    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb4b2d7f-dca3-4a50-957f-719f6a2cdc95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-v5kpm\" (UID: \"bb4b2d7f-dca3-4a50-957f-719f6a2cdc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm"
Jul 17 21:55:30.840489 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:30.840472    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb4b2d7f-dca3-4a50-957f-719f6a2cdc95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-v5kpm\" (UID: \"bb4b2d7f-dca3-4a50-957f-719f6a2cdc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm"
Jul 17 21:55:31.025337 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:31.025255    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm"
Jul 17 21:55:31.148490 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:31.148459    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm"]
Jul 17 21:55:31.152441 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:55:31.152410    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4b2d7f_dca3_4a50_957f_719f6a2cdc95.slice/crio-14b23e1ee69a3415b528e9d6b56b76e864468cb1d4945a19d7444697fa7c2afe WatchSource:0}: Error finding container 14b23e1ee69a3415b528e9d6b56b76e864468cb1d4945a19d7444697fa7c2afe: Status 404 returned error can't find the container with id 14b23e1ee69a3415b528e9d6b56b76e864468cb1d4945a19d7444697fa7c2afe
Jul 17 21:55:31.940199 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:31.940170    2573 generic.go:356] "Generic (PLEG): container finished" podID="bb4b2d7f-dca3-4a50-957f-719f6a2cdc95" containerID="1389ffe3a0ebc7e85ad4efbaceeaefcd829ff36e253bd107db759e448075cc4e" exitCode=6
Jul 17 21:55:31.940626 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:31.940204    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm" event={"ID":"bb4b2d7f-dca3-4a50-957f-719f6a2cdc95","Type":"ContainerDied","Data":"1389ffe3a0ebc7e85ad4efbaceeaefcd829ff36e253bd107db759e448075cc4e"}
Jul 17 21:55:31.940626 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:31.940265    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm" event={"ID":"bb4b2d7f-dca3-4a50-957f-719f6a2cdc95","Type":"ContainerStarted","Data":"14b23e1ee69a3415b528e9d6b56b76e864468cb1d4945a19d7444697fa7c2afe"}
Jul 17 21:55:31.940626 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:31.940499    2573 scope.go:117] "RemoveContainer" containerID="1389ffe3a0ebc7e85ad4efbaceeaefcd829ff36e253bd107db759e448075cc4e"
Jul 17 21:55:32.946779 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:32.946749    2573 generic.go:356] "Generic (PLEG): container finished" podID="bb4b2d7f-dca3-4a50-957f-719f6a2cdc95" containerID="766fcbd6cb8d940d205644713b971e1109d525cad4a9bc0b9f3670b7ac84768a" exitCode=6
Jul 17 21:55:32.947226 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:32.946794    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm" event={"ID":"bb4b2d7f-dca3-4a50-957f-719f6a2cdc95","Type":"ContainerDied","Data":"766fcbd6cb8d940d205644713b971e1109d525cad4a9bc0b9f3670b7ac84768a"}
Jul 17 21:55:32.947226 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:32.946842    2573 scope.go:117] "RemoveContainer" containerID="1389ffe3a0ebc7e85ad4efbaceeaefcd829ff36e253bd107db759e448075cc4e"
Jul 17 21:55:32.947226 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:32.947041    2573 scope.go:117] "RemoveContainer" containerID="766fcbd6cb8d940d205644713b971e1109d525cad4a9bc0b9f3670b7ac84768a"
Jul 17 21:55:32.947408 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:55:32.947262    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-v5kpm_odh-ai-gateway-infra(bb4b2d7f-dca3-4a50-957f-719f6a2cdc95)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm" podUID="bb4b2d7f-dca3-4a50-957f-719f6a2cdc95"
Jul 17 21:55:33.068694 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.068669    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"]
Jul 17 21:55:33.068933 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.068921    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2784b49a-07bf-4e0b-850c-eb66c4c3c893" containerName="revoke-keys"
Jul 17 21:55:33.068975 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.068934    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2784b49a-07bf-4e0b-850c-eb66c4c3c893" containerName="revoke-keys"
Jul 17 21:55:33.068975 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.068946    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e15970-601c-4380-8ae0-953d50d7f8cb" containerName="revoke-keys"
Jul 17 21:55:33.068975 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.068953    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e15970-601c-4380-8ae0-953d50d7f8cb" containerName="revoke-keys"
Jul 17 21:55:33.069076 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.069011    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4e15970-601c-4380-8ae0-953d50d7f8cb" containerName="revoke-keys"
Jul 17 21:55:33.072700 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.072683    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.075108 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.075090    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ffd161-openshift-default-dockercfg-tsfqv\""
Jul 17 21:55:33.087485 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.087467    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"]
Jul 17 21:55:33.157890 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.157864    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istiod-ca-cert\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.158026 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.157913    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-podinfo\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.158026 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.157933    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-token\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.158100 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.158028    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-credential-socket\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.158100 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.158086    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-envoy\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.158170 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.158121    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twsgx\" (UniqueName: \"kubernetes.io/projected/dbc1079a-d4ac-44a9-abb0-82b759148e1b-kube-api-access-twsgx\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.158170 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.158150    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-workload-socket\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.158239 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.158193    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-workload-certs\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.158239 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.158221    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-data\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.259629 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.259558    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istiod-ca-cert\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.259629 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.259602    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-podinfo\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.259629 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.259621    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-token\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.259867 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.259647    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-credential-socket\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.259867 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.259686    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-envoy\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.259867 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.259709    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twsgx\" (UniqueName: \"kubernetes.io/projected/dbc1079a-d4ac-44a9-abb0-82b759148e1b-kube-api-access-twsgx\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.259867 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.259734    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-workload-socket\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.259867 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.259769    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-workload-certs\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.259867 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.259792    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-data\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.260181 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.260123    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-credential-socket\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.260181 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.260153    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-workload-socket\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.260321 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.260229    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-data\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.260360 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.260336    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-workload-certs\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.260406 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.260359    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istiod-ca-cert\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.261973 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.261952    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-envoy\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.262297 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.262268    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-podinfo\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.267516 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.267498    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-token\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.267901 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.267880    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twsgx\" (UniqueName: \"kubernetes.io/projected/dbc1079a-d4ac-44a9-abb0-82b759148e1b-kube-api-access-twsgx\") pod \"e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") " pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.382209 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.382178    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:33.507545 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.507523    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"]
Jul 17 21:55:33.509535 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:55:33.509500    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc1079a_d4ac_44a9_abb0_82b759148e1b.slice/crio-c73f6364c2e00fbd06db341b38c322826f1ad724f13e38cc1c5bd46a027536cf WatchSource:0}: Error finding container c73f6364c2e00fbd06db341b38c322826f1ad724f13e38cc1c5bd46a027536cf: Status 404 returned error can't find the container with id c73f6364c2e00fbd06db341b38c322826f1ad724f13e38cc1c5bd46a027536cf
Jul 17 21:55:33.511720 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.511687    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:55:33.511806 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.511761    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:55:33.511863 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.511805    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:55:33.952776 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.952738    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks" event={"ID":"dbc1079a-d4ac-44a9-abb0-82b759148e1b","Type":"ContainerStarted","Data":"b3f40b83d299ac564ea6daec6e343a689dc6800de71729ba32729452a6983e51"}
Jul 17 21:55:33.953232 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.952800    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks" event={"ID":"dbc1079a-d4ac-44a9-abb0-82b759148e1b","Type":"ContainerStarted","Data":"c73f6364c2e00fbd06db341b38c322826f1ad724f13e38cc1c5bd46a027536cf"}
Jul 17 21:55:33.954457 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.954437    2573 scope.go:117] "RemoveContainer" containerID="766fcbd6cb8d940d205644713b971e1109d525cad4a9bc0b9f3670b7ac84768a"
Jul 17 21:55:33.954639 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:55:33.954621    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-v5kpm_odh-ai-gateway-infra(bb4b2d7f-dca3-4a50-957f-719f6a2cdc95)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm" podUID="bb4b2d7f-dca3-4a50-957f-719f6a2cdc95"
Jul 17 21:55:33.971466 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:33.971427    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks" podStartSLOduration=0.97141362 podStartE2EDuration="971.41362ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:55:33.969660552 +0000 UTC m=+2833.933781161" watchObservedRunningTime="2026-07-17 21:55:33.97141362 +0000 UTC m=+2833.935534229"
Jul 17 21:55:34.382959 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:34.382924    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:34.387382 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:34.387357    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:34.958689 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:34.958661    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:34.959639 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:34.959620    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:40.914917 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:40.914878    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"]
Jul 17 21:55:40.918142 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:40.918125    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:40.920607 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:40.920581    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-786043-openshift-default-dockercfg-mff8m\""
Jul 17 21:55:40.931060 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:40.931036    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"]
Jul 17 21:55:41.020802 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.020765    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1577289-b438-4210-9cd3-ce8153574531-istio-token\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.020802 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.020815    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1577289-b438-4210-9cd3-ce8153574531-istio-podinfo\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.021029 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.020844    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-istio-envoy\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.021029 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.020897    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-credential-socket\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.021029 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.020926    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1577289-b438-4210-9cd3-ce8153574531-istiod-ca-cert\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.021029 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.020950    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2lmt\" (UniqueName: \"kubernetes.io/projected/f1577289-b438-4210-9cd3-ce8153574531-kube-api-access-h2lmt\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.021029 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.020983    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-workload-socket\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.021201 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.021071    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-workload-certs\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.021201 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.021115    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-istio-data\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.122142 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.122103    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1577289-b438-4210-9cd3-ce8153574531-istio-token\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.122142 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.122143    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1577289-b438-4210-9cd3-ce8153574531-istio-podinfo\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.122445 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.122162    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-istio-envoy\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.122445 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.122193    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-credential-socket\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.122445 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.122211    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1577289-b438-4210-9cd3-ce8153574531-istiod-ca-cert\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.122445 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.122226    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2lmt\" (UniqueName: \"kubernetes.io/projected/f1577289-b438-4210-9cd3-ce8153574531-kube-api-access-h2lmt\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.122445 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.122247    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-workload-socket\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.122445 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.122319    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-workload-certs\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.122445 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.122366    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-istio-data\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.122826 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.122725    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-credential-socket\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.122826 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.122746    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-istio-data\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.122826 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.122818    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-workload-socket\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.122993 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.122830    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-workload-certs\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.123167 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.123145    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1577289-b438-4210-9cd3-ce8153574531-istiod-ca-cert\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.124482 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.124458    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-istio-envoy\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.124779 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.124760    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1577289-b438-4210-9cd3-ce8153574531-istio-podinfo\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.131584 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.131554    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1577289-b438-4210-9cd3-ce8153574531-istio-token\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.131712 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.131687    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2lmt\" (UniqueName: \"kubernetes.io/projected/f1577289-b438-4210-9cd3-ce8153574531-kube-api-access-h2lmt\") pod \"e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") " pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.229141 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.229071    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:41.360235 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.360164    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"]
Jul 17 21:55:41.362461 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:55:41.362434    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1577289_b438_4210_9cd3_ce8153574531.slice/crio-12efaf46e35c542d87882b78a0b9edfc9876f6386d1b26d653ffbb16b8f8b0be WatchSource:0}: Error finding container 12efaf46e35c542d87882b78a0b9edfc9876f6386d1b26d653ffbb16b8f8b0be: Status 404 returned error can't find the container with id 12efaf46e35c542d87882b78a0b9edfc9876f6386d1b26d653ffbb16b8f8b0be
Jul 17 21:55:41.364473 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.364439    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:55:41.364540 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.364503    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:55:41.364540 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.364532    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:55:41.989948 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.989911    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc" event={"ID":"f1577289-b438-4210-9cd3-ce8153574531","Type":"ContainerStarted","Data":"c7a9cee97594436b882f151f8d704188684e2d36311ba03ca18b7f8807527633"}
Jul 17 21:55:41.990359 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:41.989968    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc" event={"ID":"f1577289-b438-4210-9cd3-ce8153574531","Type":"ContainerStarted","Data":"12efaf46e35c542d87882b78a0b9edfc9876f6386d1b26d653ffbb16b8f8b0be"}
Jul 17 21:55:42.011972 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:42.011917    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc" podStartSLOduration=2.011901757 podStartE2EDuration="2.011901757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:55:42.010359279 +0000 UTC m=+2841.974479889" watchObservedRunningTime="2026-07-17 21:55:42.011901757 +0000 UTC m=+2841.976022367"
Jul 17 21:55:42.230006 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:42.229970    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:42.235305 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:42.235261    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:42.994378 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:42.994341    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:42.995336 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:42.995317    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:55:48.525327 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:48.525278    2573 scope.go:117] "RemoveContainer" containerID="766fcbd6cb8d940d205644713b971e1109d525cad4a9bc0b9f3670b7ac84768a"
Jul 17 21:55:49.022464 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:49.022437    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm" event={"ID":"bb4b2d7f-dca3-4a50-957f-719f6a2cdc95","Type":"ContainerStarted","Data":"6494a3d46d6bb231899cbc54adc12adbfdf4fc4886754482074025043514e7b2"}
Jul 17 21:55:50.028341 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:50.028311    2573 generic.go:356] "Generic (PLEG): container finished" podID="bb4b2d7f-dca3-4a50-957f-719f6a2cdc95" containerID="6494a3d46d6bb231899cbc54adc12adbfdf4fc4886754482074025043514e7b2" exitCode=6
Jul 17 21:55:50.028787 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:50.028387    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm" event={"ID":"bb4b2d7f-dca3-4a50-957f-719f6a2cdc95","Type":"ContainerDied","Data":"6494a3d46d6bb231899cbc54adc12adbfdf4fc4886754482074025043514e7b2"}
Jul 17 21:55:50.028787 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:50.028460    2573 scope.go:117] "RemoveContainer" containerID="766fcbd6cb8d940d205644713b971e1109d525cad4a9bc0b9f3670b7ac84768a"
Jul 17 21:55:50.028787 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:50.028726    2573 scope.go:117] "RemoveContainer" containerID="6494a3d46d6bb231899cbc54adc12adbfdf4fc4886754482074025043514e7b2"
Jul 17 21:55:50.028974 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:55:50.028951    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-v5kpm_odh-ai-gateway-infra(bb4b2d7f-dca3-4a50-957f-719f6a2cdc95)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm" podUID="bb4b2d7f-dca3-4a50-957f-719f6a2cdc95"
Jul 17 21:55:50.054422 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:50.054395    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm"]
Jul 17 21:55:51.001510 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:51.001465    2573 patch_prober.go:28] interesting pod/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 17 21:55:51.001776 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:51.001534    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks" podUID="dbc1079a-d4ac-44a9-abb0-82b759148e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 17 21:55:51.162596 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:51.162572    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm"
Jul 17 21:55:51.308327 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:51.308228    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb4b2d7f-dca3-4a50-957f-719f6a2cdc95-maas-api-service-ca\") pod \"bb4b2d7f-dca3-4a50-957f-719f6a2cdc95\" (UID: \"bb4b2d7f-dca3-4a50-957f-719f6a2cdc95\") "
Jul 17 21:55:51.308680 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:51.308654    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb4b2d7f-dca3-4a50-957f-719f6a2cdc95-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bb4b2d7f-dca3-4a50-957f-719f6a2cdc95" (UID: "bb4b2d7f-dca3-4a50-957f-719f6a2cdc95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:55:51.409348 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:51.409318    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb4b2d7f-dca3-4a50-957f-719f6a2cdc95-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:55:52.039673 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:52.039641    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm"
Jul 17 21:55:52.039673 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:52.039652    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm" event={"ID":"bb4b2d7f-dca3-4a50-957f-719f6a2cdc95","Type":"ContainerDied","Data":"14b23e1ee69a3415b528e9d6b56b76e864468cb1d4945a19d7444697fa7c2afe"}
Jul 17 21:55:52.039932 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:52.039736    2573 scope.go:117] "RemoveContainer" containerID="6494a3d46d6bb231899cbc54adc12adbfdf4fc4886754482074025043514e7b2"
Jul 17 21:55:52.061707 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:52.061683    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm"]
Jul 17 21:55:52.072771 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:52.072740    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-v5kpm"]
Jul 17 21:55:52.530356 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:52.530002    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb4b2d7f-dca3-4a50-957f-719f6a2cdc95" path="/var/lib/kubelet/pods/bb4b2d7f-dca3-4a50-957f-719f6a2cdc95/volumes"
Jul 17 21:55:53.048774 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:53.048709    2573 generic.go:356] "Generic (PLEG): container finished" podID="dbc1079a-d4ac-44a9-abb0-82b759148e1b" containerID="b3f40b83d299ac564ea6daec6e343a689dc6800de71729ba32729452a6983e51" exitCode=0
Jul 17 21:55:53.049078 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:53.049026    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks" event={"ID":"dbc1079a-d4ac-44a9-abb0-82b759148e1b","Type":"ContainerDied","Data":"b3f40b83d299ac564ea6daec6e343a689dc6800de71729ba32729452a6983e51"}
Jul 17 21:55:53.049806 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:53.049765    2573 scope.go:117] "RemoveContainer" containerID="b3f40b83d299ac564ea6daec6e343a689dc6800de71729ba32729452a6983e51"
Jul 17 21:55:53.052299 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:53.051961    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:55:53.052299 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:53.052048    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:55:53.052299 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:53.052096    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:55:54.057330 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:54.057265    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks" event={"ID":"dbc1079a-d4ac-44a9-abb0-82b759148e1b","Type":"ContainerStarted","Data":"5d380283841834c27d588cc0c5a3730eca34ba3501f84d96bd0e7b6907d4c258"}
Jul 17 21:55:54.383125 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:54.383087    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:54.383360 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:54.383278    2573 patch_prober.go:28] interesting pod/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 17 21:55:54.383451 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:54.383360    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks" podUID="dbc1079a-d4ac-44a9-abb0-82b759148e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 17 21:55:55.383362 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:55.383322    2573 patch_prober.go:28] interesting pod/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 17 21:55:55.383868 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:55.383381    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks" podUID="dbc1079a-d4ac-44a9-abb0-82b759148e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 17 21:55:56.383139 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:56.383099    2573 patch_prober.go:28] interesting pod/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 17 21:55:56.383351 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:56.383167    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks" podUID="dbc1079a-d4ac-44a9-abb0-82b759148e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 17 21:55:57.383507 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:57.383470    2573 patch_prober.go:28] interesting pod/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 17 21:55:57.383982 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:57.383570    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks" podUID="dbc1079a-d4ac-44a9-abb0-82b759148e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 17 21:55:58.383542 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:58.383502    2573 patch_prober.go:28] interesting pod/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 17 21:55:58.383997 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:58.383567    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks" podUID="dbc1079a-d4ac-44a9-abb0-82b759148e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 17 21:55:59.036458 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:59.036413    2573 patch_prober.go:28] interesting pod/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 17 21:55:59.036677 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:59.036485    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc" podUID="f1577289-b438-4210-9cd3-ce8153574531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 17 21:55:59.389734 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:59.389705    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:59.391899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:59.390099    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:55:59.391899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:55:59.390891    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:56:00.087179 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:00.087146    2573 generic.go:356] "Generic (PLEG): container finished" podID="f1577289-b438-4210-9cd3-ce8153574531" containerID="c7a9cee97594436b882f151f8d704188684e2d36311ba03ca18b7f8807527633" exitCode=0
Jul 17 21:56:00.087388 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:00.087222    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc" event={"ID":"f1577289-b438-4210-9cd3-ce8153574531","Type":"ContainerDied","Data":"c7a9cee97594436b882f151f8d704188684e2d36311ba03ca18b7f8807527633"}
Jul 17 21:56:00.087741 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:00.087725    2573 scope.go:117] "RemoveContainer" containerID="c7a9cee97594436b882f151f8d704188684e2d36311ba03ca18b7f8807527633"
Jul 17 21:56:00.088662 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:00.088638    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:56:00.088716 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:00.088703    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:56:00.088758 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:00.088736    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:56:01.093547 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:01.093515    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc" event={"ID":"f1577289-b438-4210-9cd3-ce8153574531","Type":"ContainerStarted","Data":"f62982683388221947317e13eea77099e8fafc50d8562d9e449b8b034cb45d60"}
Jul 17 21:56:01.238508 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:01.238466    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:56:01.239329 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:01.239276    2573 patch_prober.go:28] interesting pod/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 17 21:56:01.239724 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:01.239693    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc" podUID="f1577289-b438-4210-9cd3-ce8153574531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 17 21:56:02.230417 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:02.230379    2573 patch_prober.go:28] interesting pod/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 17 21:56:02.230924 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:02.230449    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc" podUID="f1577289-b438-4210-9cd3-ce8153574531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 17 21:56:03.231080 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:03.230586    2573 patch_prober.go:28] interesting pod/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 17 21:56:03.231080 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:03.230658    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc" podUID="f1577289-b438-4210-9cd3-ce8153574531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 17 21:56:04.230469 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:04.230432    2573 patch_prober.go:28] interesting pod/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 17 21:56:04.230768 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:04.230738    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc" podUID="f1577289-b438-4210-9cd3-ce8153574531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 17 21:56:05.233279 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:05.233250    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:56:05.233726 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:05.233709    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:56:05.234396 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:56:05.234359    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:57:08.179584 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:08.179551    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffd161-fsth4"]
Jul 17 21:57:08.180046 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:08.179846    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb4b2d7f-dca3-4a50-957f-719f6a2cdc95" containerName="revoke-keys"
Jul 17 21:57:08.180046 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:08.179860    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb4b2d7f-dca3-4a50-957f-719f6a2cdc95" containerName="revoke-keys"
Jul 17 21:57:08.180046 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:08.179878    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb4b2d7f-dca3-4a50-957f-719f6a2cdc95" containerName="revoke-keys"
Jul 17 21:57:08.180046 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:08.179885    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb4b2d7f-dca3-4a50-957f-719f6a2cdc95" containerName="revoke-keys"
Jul 17 21:57:08.180046 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:08.179894    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb4b2d7f-dca3-4a50-957f-719f6a2cdc95" containerName="revoke-keys"
Jul 17 21:57:08.180046 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:08.179900    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb4b2d7f-dca3-4a50-957f-719f6a2cdc95" containerName="revoke-keys"
Jul 17 21:57:08.180046 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:08.179951    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb4b2d7f-dca3-4a50-957f-719f6a2cdc95" containerName="revoke-keys"
Jul 17 21:57:08.180046 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:08.179958    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb4b2d7f-dca3-4a50-957f-719f6a2cdc95" containerName="revoke-keys"
Jul 17 21:57:08.182869 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:08.182851    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffd161-fsth4"
Jul 17 21:57:08.189955 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:08.189931    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:57:08.190221 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:08.190196    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:57:08.192003 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:08.191981    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffd161-fsth4"]
Jul 17 21:57:08.323356 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:08.323323    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76dd8293-4d21-4387-bf38-8b63670fb88e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ffd161-fsth4\" (UID: \"76dd8293-4d21-4387-bf38-8b63670fb88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffd161-fsth4"
Jul 17 21:57:08.424138 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:08.424104    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76dd8293-4d21-4387-bf38-8b63670fb88e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ffd161-fsth4\" (UID: \"76dd8293-4d21-4387-bf38-8b63670fb88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffd161-fsth4"
Jul 17 21:57:08.424688 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:08.424670    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76dd8293-4d21-4387-bf38-8b63670fb88e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ffd161-fsth4\" (UID: \"76dd8293-4d21-4387-bf38-8b63670fb88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffd161-fsth4"
Jul 17 21:57:08.500250 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:08.500197    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffd161-fsth4"
Jul 17 21:57:08.623790 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:08.623756    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffd161-fsth4"]
Jul 17 21:57:08.626678 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:57:08.626653    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76dd8293_4d21_4387_bf38_8b63670fb88e.slice/crio-a75006d07d7e75487a9e73658fa084b2c0cf23613bd01068c2e7724b711746de WatchSource:0}: Error finding container a75006d07d7e75487a9e73658fa084b2c0cf23613bd01068c2e7724b711746de: Status 404 returned error can't find the container with id a75006d07d7e75487a9e73658fa084b2c0cf23613bd01068c2e7724b711746de
Jul 17 21:57:09.375865 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:09.375829    2573 generic.go:356] "Generic (PLEG): container finished" podID="76dd8293-4d21-4387-bf38-8b63670fb88e" containerID="561ab3bdfdb22f836ab09c7a1b75c2c655523a351f65fedbba890c87d1b48ba3" exitCode=0
Jul 17 21:57:09.376231 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:09.375919    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffd161-fsth4" event={"ID":"76dd8293-4d21-4387-bf38-8b63670fb88e","Type":"ContainerDied","Data":"561ab3bdfdb22f836ab09c7a1b75c2c655523a351f65fedbba890c87d1b48ba3"}
Jul 17 21:57:09.376231 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:09.375987    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffd161-fsth4" event={"ID":"76dd8293-4d21-4387-bf38-8b63670fb88e","Type":"ContainerStarted","Data":"a75006d07d7e75487a9e73658fa084b2c0cf23613bd01068c2e7724b711746de"}
Jul 17 21:57:10.504092 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:10.504064    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffd161-fsth4"
Jul 17 21:57:10.641823 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:10.641741    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76dd8293-4d21-4387-bf38-8b63670fb88e-maas-api-service-ca\") pod \"76dd8293-4d21-4387-bf38-8b63670fb88e\" (UID: \"76dd8293-4d21-4387-bf38-8b63670fb88e\") "
Jul 17 21:57:10.642070 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:10.642039    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76dd8293-4d21-4387-bf38-8b63670fb88e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "76dd8293-4d21-4387-bf38-8b63670fb88e" (UID: "76dd8293-4d21-4387-bf38-8b63670fb88e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:57:10.743179 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:10.743147    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76dd8293-4d21-4387-bf38-8b63670fb88e-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:57:11.385401 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:11.385375    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffd161-fsth4"
Jul 17 21:57:11.385401 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:11.385389    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ffd161-fsth4" event={"ID":"76dd8293-4d21-4387-bf38-8b63670fb88e","Type":"ContainerDied","Data":"a75006d07d7e75487a9e73658fa084b2c0cf23613bd01068c2e7724b711746de"}
Jul 17 21:57:11.385624 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:11.385439    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a75006d07d7e75487a9e73658fa084b2c0cf23613bd01068c2e7724b711746de"
Jul 17 21:57:33.969395 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:33.969364    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"]
Jul 17 21:57:33.969805 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:33.969581    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks" podUID="dbc1079a-d4ac-44a9-abb0-82b759148e1b" containerName="istio-proxy" containerID="cri-o://5d380283841834c27d588cc0c5a3730eca34ba3501f84d96bd0e7b6907d4c258" gracePeriod=30
Jul 17 21:57:39.212645 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.212621    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:57:39.274987 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.274918    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-credential-socket\") pod \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") "
Jul 17 21:57:39.274987 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.274966    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-podinfo\") pod \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") "
Jul 17 21:57:39.274987 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.274985    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-data\") pod \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") "
Jul 17 21:57:39.275254 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.275002    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-workload-socket\") pod \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") "
Jul 17 21:57:39.275254 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.275124    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-envoy\") pod \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") "
Jul 17 21:57:39.275254 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.275176    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istiod-ca-cert\") pod \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") "
Jul 17 21:57:39.275254 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.275201    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twsgx\" (UniqueName: \"kubernetes.io/projected/dbc1079a-d4ac-44a9-abb0-82b759148e1b-kube-api-access-twsgx\") pod \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") "
Jul 17 21:57:39.275254 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.275225    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-workload-certs\") pod \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") "
Jul 17 21:57:39.275539 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.275259    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-token\") pod \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\" (UID: \"dbc1079a-d4ac-44a9-abb0-82b759148e1b\") "
Jul 17 21:57:39.275539 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.275272    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dbc1079a-d4ac-44a9-abb0-82b759148e1b" (UID: "dbc1079a-d4ac-44a9-abb0-82b759148e1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:57:39.275539 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.275330    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dbc1079a-d4ac-44a9-abb0-82b759148e1b" (UID: "dbc1079a-d4ac-44a9-abb0-82b759148e1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:57:39.275539 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.275493    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dbc1079a-d4ac-44a9-abb0-82b759148e1b" (UID: "dbc1079a-d4ac-44a9-abb0-82b759148e1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:57:39.275539 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.275525    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dbc1079a-d4ac-44a9-abb0-82b759148e1b" (UID: "dbc1079a-d4ac-44a9-abb0-82b759148e1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:57:39.275792 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.275604    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:57:39.275792 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.275622    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:57:39.275792 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.275637    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:57:39.275951 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.275840    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-data" (OuterVolumeSpecName: "istio-data") pod "dbc1079a-d4ac-44a9-abb0-82b759148e1b" (UID: "dbc1079a-d4ac-44a9-abb0-82b759148e1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:57:39.277163 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.277137    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dbc1079a-d4ac-44a9-abb0-82b759148e1b" (UID: "dbc1079a-d4ac-44a9-abb0-82b759148e1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 21:57:39.277343 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.277319    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc1079a-d4ac-44a9-abb0-82b759148e1b-kube-api-access-twsgx" (OuterVolumeSpecName: "kube-api-access-twsgx") pod "dbc1079a-d4ac-44a9-abb0-82b759148e1b" (UID: "dbc1079a-d4ac-44a9-abb0-82b759148e1b"). InnerVolumeSpecName "kube-api-access-twsgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:57:39.277468 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.277430    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-token" (OuterVolumeSpecName: "istio-token") pod "dbc1079a-d4ac-44a9-abb0-82b759148e1b" (UID: "dbc1079a-d4ac-44a9-abb0-82b759148e1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:57:39.277468 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.277456    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dbc1079a-d4ac-44a9-abb0-82b759148e1b" (UID: "dbc1079a-d4ac-44a9-abb0-82b759148e1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:57:39.376842 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.376811    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:57:39.376842 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.376837    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:57:39.376842 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.376849    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:57:39.377049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.376860    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:57:39.377049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.376872    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twsgx\" (UniqueName: \"kubernetes.io/projected/dbc1079a-d4ac-44a9-abb0-82b759148e1b-kube-api-access-twsgx\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:57:39.377049 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.376883    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbc1079a-d4ac-44a9-abb0-82b759148e1b-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:57:39.507841 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.507807    2573 generic.go:356] "Generic (PLEG): container finished" podID="dbc1079a-d4ac-44a9-abb0-82b759148e1b" containerID="5d380283841834c27d588cc0c5a3730eca34ba3501f84d96bd0e7b6907d4c258" exitCode=0
Jul 17 21:57:39.507998 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.507865    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"
Jul 17 21:57:39.507998 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.507866    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks" event={"ID":"dbc1079a-d4ac-44a9-abb0-82b759148e1b","Type":"ContainerDied","Data":"5d380283841834c27d588cc0c5a3730eca34ba3501f84d96bd0e7b6907d4c258"}
Jul 17 21:57:39.508086 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.507999    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks" event={"ID":"dbc1079a-d4ac-44a9-abb0-82b759148e1b","Type":"ContainerDied","Data":"c73f6364c2e00fbd06db341b38c322826f1ad724f13e38cc1c5bd46a027536cf"}
Jul 17 21:57:39.508086 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.508021    2573 scope.go:117] "RemoveContainer" containerID="5d380283841834c27d588cc0c5a3730eca34ba3501f84d96bd0e7b6907d4c258"
Jul 17 21:57:39.516945 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.516928    2573 scope.go:117] "RemoveContainer" containerID="b3f40b83d299ac564ea6daec6e343a689dc6800de71729ba32729452a6983e51"
Jul 17 21:57:39.524349 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.524331    2573 scope.go:117] "RemoveContainer" containerID="5d380283841834c27d588cc0c5a3730eca34ba3501f84d96bd0e7b6907d4c258"
Jul 17 21:57:39.524584 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:57:39.524562    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d380283841834c27d588cc0c5a3730eca34ba3501f84d96bd0e7b6907d4c258\": container with ID starting with 5d380283841834c27d588cc0c5a3730eca34ba3501f84d96bd0e7b6907d4c258 not found: ID does not exist" containerID="5d380283841834c27d588cc0c5a3730eca34ba3501f84d96bd0e7b6907d4c258"
Jul 17 21:57:39.524668 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.524590    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d380283841834c27d588cc0c5a3730eca34ba3501f84d96bd0e7b6907d4c258"} err="failed to get container status \"5d380283841834c27d588cc0c5a3730eca34ba3501f84d96bd0e7b6907d4c258\": rpc error: code = NotFound desc = could not find container \"5d380283841834c27d588cc0c5a3730eca34ba3501f84d96bd0e7b6907d4c258\": container with ID starting with 5d380283841834c27d588cc0c5a3730eca34ba3501f84d96bd0e7b6907d4c258 not found: ID does not exist"
Jul 17 21:57:39.524668 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.524606    2573 scope.go:117] "RemoveContainer" containerID="b3f40b83d299ac564ea6daec6e343a689dc6800de71729ba32729452a6983e51"
Jul 17 21:57:39.524840 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:57:39.524821    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f40b83d299ac564ea6daec6e343a689dc6800de71729ba32729452a6983e51\": container with ID starting with b3f40b83d299ac564ea6daec6e343a689dc6800de71729ba32729452a6983e51 not found: ID does not exist" containerID="b3f40b83d299ac564ea6daec6e343a689dc6800de71729ba32729452a6983e51"
Jul 17 21:57:39.524883 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.524846    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f40b83d299ac564ea6daec6e343a689dc6800de71729ba32729452a6983e51"} err="failed to get container status \"b3f40b83d299ac564ea6daec6e343a689dc6800de71729ba32729452a6983e51\": rpc error: code = NotFound desc = could not find container \"b3f40b83d299ac564ea6daec6e343a689dc6800de71729ba32729452a6983e51\": container with ID starting with b3f40b83d299ac564ea6daec6e343a689dc6800de71729ba32729452a6983e51 not found: ID does not exist"
Jul 17 21:57:39.529315 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.529255    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"]
Jul 17 21:57:39.533093 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:39.533073    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ffd161-openshift-default-69b88ffcc-cgrks"]
Jul 17 21:57:40.530711 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:40.530679    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbc1079a-d4ac-44a9-abb0-82b759148e1b" path="/var/lib/kubelet/pods/dbc1079a-d4ac-44a9-abb0-82b759148e1b/volumes"
Jul 17 21:57:45.533106 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:45.533028    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc"]
Jul 17 21:57:45.533504 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:45.533443    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76dd8293-4d21-4387-bf38-8b63670fb88e" containerName="revoke-keys"
Jul 17 21:57:45.533504 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:45.533459    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="76dd8293-4d21-4387-bf38-8b63670fb88e" containerName="revoke-keys"
Jul 17 21:57:45.533504 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:45.533476    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc1079a-d4ac-44a9-abb0-82b759148e1b" containerName="istio-proxy"
Jul 17 21:57:45.533504 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:45.533485    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc1079a-d4ac-44a9-abb0-82b759148e1b" containerName="istio-proxy"
Jul 17 21:57:45.533660 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:45.533575    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="76dd8293-4d21-4387-bf38-8b63670fb88e" containerName="revoke-keys"
Jul 17 21:57:45.533660 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:45.533587    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb4b2d7f-dca3-4a50-957f-719f6a2cdc95" containerName="revoke-keys"
Jul 17 21:57:45.533660 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:45.533598    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbc1079a-d4ac-44a9-abb0-82b759148e1b" containerName="istio-proxy"
Jul 17 21:57:45.533660 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:45.533608    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbc1079a-d4ac-44a9-abb0-82b759148e1b" containerName="istio-proxy"
Jul 17 21:57:45.538154 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:45.538135    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc"
Jul 17 21:57:45.540535 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:45.540514    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:57:45.540674 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:45.540519    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:57:45.543701 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:45.543679    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc"]
Jul 17 21:57:45.615581 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:45.615550    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29b2c2d-0276-4a71-8841-94a333748961-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc\" (UID: \"d29b2c2d-0276-4a71-8841-94a333748961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc"
Jul 17 21:57:45.716562 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:45.716536    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29b2c2d-0276-4a71-8841-94a333748961-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc\" (UID: \"d29b2c2d-0276-4a71-8841-94a333748961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc"
Jul 17 21:57:45.717096 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:45.717077    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29b2c2d-0276-4a71-8841-94a333748961-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc\" (UID: \"d29b2c2d-0276-4a71-8841-94a333748961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc"
Jul 17 21:57:45.849564 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:45.849543    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc"
Jul 17 21:57:46.180958 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:46.180936    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc"]
Jul 17 21:57:46.183566 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:57:46.183537    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29b2c2d_0276_4a71_8841_94a333748961.slice/crio-e876e5838e1813d3eba35e7b1217ad910793f850d3728ccd0148e87222e74eaa WatchSource:0}: Error finding container e876e5838e1813d3eba35e7b1217ad910793f850d3728ccd0148e87222e74eaa: Status 404 returned error can't find the container with id e876e5838e1813d3eba35e7b1217ad910793f850d3728ccd0148e87222e74eaa
Jul 17 21:57:46.539934 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:46.539893    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc" event={"ID":"d29b2c2d-0276-4a71-8841-94a333748961","Type":"ContainerStarted","Data":"3361d78ba221b97fc9e38297b7a6e028604e50d28278beee2f01cc9898e980a1"}
Jul 17 21:57:46.540303 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:46.539941    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc" event={"ID":"d29b2c2d-0276-4a71-8841-94a333748961","Type":"ContainerStarted","Data":"e876e5838e1813d3eba35e7b1217ad910793f850d3728ccd0148e87222e74eaa"}
Jul 17 21:57:46.556370 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:46.556321    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc" podStartSLOduration=1.5563062269999999 podStartE2EDuration="1.556306227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:57:46.554689219 +0000 UTC m=+2966.518809829" watchObservedRunningTime="2026-07-17 21:57:46.556306227 +0000 UTC m=+2966.520426834"
Jul 17 21:57:47.545619 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:47.545589    2573 generic.go:356] "Generic (PLEG): container finished" podID="d29b2c2d-0276-4a71-8841-94a333748961" containerID="3361d78ba221b97fc9e38297b7a6e028604e50d28278beee2f01cc9898e980a1" exitCode=6
Jul 17 21:57:47.546076 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:47.545653    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc" event={"ID":"d29b2c2d-0276-4a71-8841-94a333748961","Type":"ContainerDied","Data":"3361d78ba221b97fc9e38297b7a6e028604e50d28278beee2f01cc9898e980a1"}
Jul 17 21:57:47.546076 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:47.545913    2573 scope.go:117] "RemoveContainer" containerID="3361d78ba221b97fc9e38297b7a6e028604e50d28278beee2f01cc9898e980a1"
Jul 17 21:57:48.550813 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:48.550720    2573 generic.go:356] "Generic (PLEG): container finished" podID="d29b2c2d-0276-4a71-8841-94a333748961" containerID="2438c62c566e63d7c2a58b5ab7479290815de6baccffc13b590700b524ff9a71" exitCode=6
Jul 17 21:57:48.550813 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:48.550785    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc" event={"ID":"d29b2c2d-0276-4a71-8841-94a333748961","Type":"ContainerDied","Data":"2438c62c566e63d7c2a58b5ab7479290815de6baccffc13b590700b524ff9a71"}
Jul 17 21:57:48.551240 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:48.550834    2573 scope.go:117] "RemoveContainer" containerID="3361d78ba221b97fc9e38297b7a6e028604e50d28278beee2f01cc9898e980a1"
Jul 17 21:57:48.551240 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:48.550997    2573 scope.go:117] "RemoveContainer" containerID="2438c62c566e63d7c2a58b5ab7479290815de6baccffc13b590700b524ff9a71"
Jul 17 21:57:48.551240 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:57:48.551183    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc_odh-ai-gateway-infra(d29b2c2d-0276-4a71-8841-94a333748961)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc" podUID="d29b2c2d-0276-4a71-8841-94a333748961"
Jul 17 21:57:49.556533 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:57:49.556502    2573 scope.go:117] "RemoveContainer" containerID="2438c62c566e63d7c2a58b5ab7479290815de6baccffc13b590700b524ff9a71"
Jul 17 21:57:49.557090 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:57:49.556732    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc_odh-ai-gateway-infra(d29b2c2d-0276-4a71-8841-94a333748961)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc" podUID="d29b2c2d-0276-4a71-8841-94a333748961"
Jul 17 21:58:00.079432 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:00.079391    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"]
Jul 17 21:58:00.079836 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:00.079618    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc" podUID="f1577289-b438-4210-9cd3-ce8153574531" containerName="istio-proxy" containerID="cri-o://f62982683388221947317e13eea77099e8fafc50d8562d9e449b8b034cb45d60" gracePeriod=30
Jul 17 21:58:02.529354 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:02.529330    2573 scope.go:117] "RemoveContainer" containerID="2438c62c566e63d7c2a58b5ab7479290815de6baccffc13b590700b524ff9a71"
Jul 17 21:58:03.612484 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:03.612453    2573 generic.go:356] "Generic (PLEG): container finished" podID="d29b2c2d-0276-4a71-8841-94a333748961" containerID="420c8be079668e829460bd87b56a9c600581bf54862f1223559fa9db5b7ee9ea" exitCode=6
Jul 17 21:58:03.612908 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:03.612531    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc" event={"ID":"d29b2c2d-0276-4a71-8841-94a333748961","Type":"ContainerDied","Data":"420c8be079668e829460bd87b56a9c600581bf54862f1223559fa9db5b7ee9ea"}
Jul 17 21:58:03.612908 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:03.612611    2573 scope.go:117] "RemoveContainer" containerID="2438c62c566e63d7c2a58b5ab7479290815de6baccffc13b590700b524ff9a71"
Jul 17 21:58:03.613022 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:03.612938    2573 scope.go:117] "RemoveContainer" containerID="420c8be079668e829460bd87b56a9c600581bf54862f1223559fa9db5b7ee9ea"
Jul 17 21:58:03.613167 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:58:03.613145    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc_odh-ai-gateway-infra(d29b2c2d-0276-4a71-8841-94a333748961)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc" podUID="d29b2c2d-0276-4a71-8841-94a333748961"
Jul 17 21:58:04.636748 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:04.636716    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc"]
Jul 17 21:58:04.767790 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:04.767764    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc"
Jul 17 21:58:04.868364 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:04.868329    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29b2c2d-0276-4a71-8841-94a333748961-maas-api-service-ca\") pod \"d29b2c2d-0276-4a71-8841-94a333748961\" (UID: \"d29b2c2d-0276-4a71-8841-94a333748961\") "
Jul 17 21:58:04.868712 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:04.868691    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d29b2c2d-0276-4a71-8841-94a333748961-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d29b2c2d-0276-4a71-8841-94a333748961" (UID: "d29b2c2d-0276-4a71-8841-94a333748961"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:58:04.969242 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:04.969171    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d29b2c2d-0276-4a71-8841-94a333748961-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:58:05.235118 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.235042    2573 patch_prober.go:28] interesting pod/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 17 21:58:05.235245 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.235109    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc" podUID="f1577289-b438-4210-9cd3-ce8153574531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 17 21:58:05.337317 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.337277    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:58:05.372022 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.372000    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-credential-socket\") pod \"f1577289-b438-4210-9cd3-ce8153574531\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") "
Jul 17 21:58:05.372147 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.372031    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1577289-b438-4210-9cd3-ce8153574531-istio-podinfo\") pod \"f1577289-b438-4210-9cd3-ce8153574531\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") "
Jul 17 21:58:05.372147 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.372053    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-istio-data\") pod \"f1577289-b438-4210-9cd3-ce8153574531\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") "
Jul 17 21:58:05.372147 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.372073    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2lmt\" (UniqueName: \"kubernetes.io/projected/f1577289-b438-4210-9cd3-ce8153574531-kube-api-access-h2lmt\") pod \"f1577289-b438-4210-9cd3-ce8153574531\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") "
Jul 17 21:58:05.372147 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.372101    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-istio-envoy\") pod \"f1577289-b438-4210-9cd3-ce8153574531\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") "
Jul 17 21:58:05.372147 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.372139    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-workload-socket\") pod \"f1577289-b438-4210-9cd3-ce8153574531\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") "
Jul 17 21:58:05.372455 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.372191    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1577289-b438-4210-9cd3-ce8153574531-istio-token\") pod \"f1577289-b438-4210-9cd3-ce8153574531\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") "
Jul 17 21:58:05.372455 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.372235    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-workload-certs\") pod \"f1577289-b438-4210-9cd3-ce8153574531\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") "
Jul 17 21:58:05.372455 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.372260    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1577289-b438-4210-9cd3-ce8153574531-istiod-ca-cert\") pod \"f1577289-b438-4210-9cd3-ce8153574531\" (UID: \"f1577289-b438-4210-9cd3-ce8153574531\") "
Jul 17 21:58:05.372455 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.372268    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f1577289-b438-4210-9cd3-ce8153574531" (UID: "f1577289-b438-4210-9cd3-ce8153574531"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:58:05.372674 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.372547    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:58:05.372674 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.372559    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f1577289-b438-4210-9cd3-ce8153574531" (UID: "f1577289-b438-4210-9cd3-ce8153574531"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:58:05.372674 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.372629    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f1577289-b438-4210-9cd3-ce8153574531" (UID: "f1577289-b438-4210-9cd3-ce8153574531"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:58:05.372843 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.372806    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1577289-b438-4210-9cd3-ce8153574531-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f1577289-b438-4210-9cd3-ce8153574531" (UID: "f1577289-b438-4210-9cd3-ce8153574531"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:58:05.373140 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.373114    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-istio-data" (OuterVolumeSpecName: "istio-data") pod "f1577289-b438-4210-9cd3-ce8153574531" (UID: "f1577289-b438-4210-9cd3-ce8153574531"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:58:05.374366 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.374346    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1577289-b438-4210-9cd3-ce8153574531-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f1577289-b438-4210-9cd3-ce8153574531" (UID: "f1577289-b438-4210-9cd3-ce8153574531"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 21:58:05.374507 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.374488    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f1577289-b438-4210-9cd3-ce8153574531" (UID: "f1577289-b438-4210-9cd3-ce8153574531"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:58:05.374583 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.374555    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1577289-b438-4210-9cd3-ce8153574531-istio-token" (OuterVolumeSpecName: "istio-token") pod "f1577289-b438-4210-9cd3-ce8153574531" (UID: "f1577289-b438-4210-9cd3-ce8153574531"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:58:05.374793 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.374776    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1577289-b438-4210-9cd3-ce8153574531-kube-api-access-h2lmt" (OuterVolumeSpecName: "kube-api-access-h2lmt") pod "f1577289-b438-4210-9cd3-ce8153574531" (UID: "f1577289-b438-4210-9cd3-ce8153574531"). InnerVolumeSpecName "kube-api-access-h2lmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:58:05.473544 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.473518    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1577289-b438-4210-9cd3-ce8153574531-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:58:05.473544 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.473542    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:58:05.473679 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.473555    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1577289-b438-4210-9cd3-ce8153574531-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:58:05.473679 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.473566    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1577289-b438-4210-9cd3-ce8153574531-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:58:05.473679 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.473578    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:58:05.473679 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.473590    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2lmt\" (UniqueName: \"kubernetes.io/projected/f1577289-b438-4210-9cd3-ce8153574531-kube-api-access-h2lmt\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:58:05.473679 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.473604    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:58:05.473679 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.473617    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1577289-b438-4210-9cd3-ce8153574531-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:58:05.623622 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.623594    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc"
Jul 17 21:58:05.623766 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.623636    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc" event={"ID":"d29b2c2d-0276-4a71-8841-94a333748961","Type":"ContainerDied","Data":"e876e5838e1813d3eba35e7b1217ad910793f850d3728ccd0148e87222e74eaa"}
Jul 17 21:58:05.623766 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.623696    2573 scope.go:117] "RemoveContainer" containerID="420c8be079668e829460bd87b56a9c600581bf54862f1223559fa9db5b7ee9ea"
Jul 17 21:58:05.625365 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.625336    2573 generic.go:356] "Generic (PLEG): container finished" podID="f1577289-b438-4210-9cd3-ce8153574531" containerID="f62982683388221947317e13eea77099e8fafc50d8562d9e449b8b034cb45d60" exitCode=0
Jul 17 21:58:05.625476 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.625373    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc" event={"ID":"f1577289-b438-4210-9cd3-ce8153574531","Type":"ContainerDied","Data":"f62982683388221947317e13eea77099e8fafc50d8562d9e449b8b034cb45d60"}
Jul 17 21:58:05.625476 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.625384    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"
Jul 17 21:58:05.625476 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.625403    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc" event={"ID":"f1577289-b438-4210-9cd3-ce8153574531","Type":"ContainerDied","Data":"12efaf46e35c542d87882b78a0b9edfc9876f6386d1b26d653ffbb16b8f8b0be"}
Jul 17 21:58:05.632441 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.632426    2573 scope.go:117] "RemoveContainer" containerID="f62982683388221947317e13eea77099e8fafc50d8562d9e449b8b034cb45d60"
Jul 17 21:58:05.640318 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.640141    2573 scope.go:117] "RemoveContainer" containerID="c7a9cee97594436b882f151f8d704188684e2d36311ba03ca18b7f8807527633"
Jul 17 21:58:05.647197 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.647179    2573 scope.go:117] "RemoveContainer" containerID="f62982683388221947317e13eea77099e8fafc50d8562d9e449b8b034cb45d60"
Jul 17 21:58:05.647456 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:58:05.647437    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f62982683388221947317e13eea77099e8fafc50d8562d9e449b8b034cb45d60\": container with ID starting with f62982683388221947317e13eea77099e8fafc50d8562d9e449b8b034cb45d60 not found: ID does not exist" containerID="f62982683388221947317e13eea77099e8fafc50d8562d9e449b8b034cb45d60"
Jul 17 21:58:05.647520 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.647463    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f62982683388221947317e13eea77099e8fafc50d8562d9e449b8b034cb45d60"} err="failed to get container status \"f62982683388221947317e13eea77099e8fafc50d8562d9e449b8b034cb45d60\": rpc error: code = NotFound desc = could not find container \"f62982683388221947317e13eea77099e8fafc50d8562d9e449b8b034cb45d60\": container with ID starting with f62982683388221947317e13eea77099e8fafc50d8562d9e449b8b034cb45d60 not found: ID does not exist"
Jul 17 21:58:05.647520 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.647480    2573 scope.go:117] "RemoveContainer" containerID="c7a9cee97594436b882f151f8d704188684e2d36311ba03ca18b7f8807527633"
Jul 17 21:58:05.647723 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:58:05.647702    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a9cee97594436b882f151f8d704188684e2d36311ba03ca18b7f8807527633\": container with ID starting with c7a9cee97594436b882f151f8d704188684e2d36311ba03ca18b7f8807527633 not found: ID does not exist" containerID="c7a9cee97594436b882f151f8d704188684e2d36311ba03ca18b7f8807527633"
Jul 17 21:58:05.647856 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.647729    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a9cee97594436b882f151f8d704188684e2d36311ba03ca18b7f8807527633"} err="failed to get container status \"c7a9cee97594436b882f151f8d704188684e2d36311ba03ca18b7f8807527633\": rpc error: code = NotFound desc = could not find container \"c7a9cee97594436b882f151f8d704188684e2d36311ba03ca18b7f8807527633\": container with ID starting with c7a9cee97594436b882f151f8d704188684e2d36311ba03ca18b7f8807527633 not found: ID does not exist"
Jul 17 21:58:05.649763 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.649744    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"]
Jul 17 21:58:05.657168 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.657147    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-786043-openshift-default-67f5f475d7-v2hlc"]
Jul 17 21:58:05.666648 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.666629    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc"]
Jul 17 21:58:05.672307 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:05.672275    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nghdc"]
Jul 17 21:58:06.530709 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:06.530678    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d29b2c2d-0276-4a71-8841-94a333748961" path="/var/lib/kubelet/pods/d29b2c2d-0276-4a71-8841-94a333748961/volumes"
Jul 17 21:58:06.531181 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:06.531158    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1577289-b438-4210-9cd3-ce8153574531" path="/var/lib/kubelet/pods/f1577289-b438-4210-9cd3-ce8153574531/volumes"
Jul 17 21:58:09.170171 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.170140    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"]
Jul 17 21:58:09.170670 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.170578    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29b2c2d-0276-4a71-8841-94a333748961" containerName="revoke-keys"
Jul 17 21:58:09.170670 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.170596    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29b2c2d-0276-4a71-8841-94a333748961" containerName="revoke-keys"
Jul 17 21:58:09.170670 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.170610    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29b2c2d-0276-4a71-8841-94a333748961" containerName="revoke-keys"
Jul 17 21:58:09.170670 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.170618    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29b2c2d-0276-4a71-8841-94a333748961" containerName="revoke-keys"
Jul 17 21:58:09.170670 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.170627    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29b2c2d-0276-4a71-8841-94a333748961" containerName="revoke-keys"
Jul 17 21:58:09.170670 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.170658    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29b2c2d-0276-4a71-8841-94a333748961" containerName="revoke-keys"
Jul 17 21:58:09.170969 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.170673    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1577289-b438-4210-9cd3-ce8153574531" containerName="istio-proxy"
Jul 17 21:58:09.170969 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.170682    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1577289-b438-4210-9cd3-ce8153574531" containerName="istio-proxy"
Jul 17 21:58:09.170969 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.170692    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbc1079a-d4ac-44a9-abb0-82b759148e1b" containerName="istio-proxy"
Jul 17 21:58:09.170969 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.170702    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbc1079a-d4ac-44a9-abb0-82b759148e1b" containerName="istio-proxy"
Jul 17 21:58:09.170969 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.170785    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d29b2c2d-0276-4a71-8841-94a333748961" containerName="revoke-keys"
Jul 17 21:58:09.170969 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.170797    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1577289-b438-4210-9cd3-ce8153574531" containerName="istio-proxy"
Jul 17 21:58:09.170969 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.170808    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d29b2c2d-0276-4a71-8841-94a333748961" containerName="revoke-keys"
Jul 17 21:58:09.170969 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.170817    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1577289-b438-4210-9cd3-ce8153574531" containerName="istio-proxy"
Jul 17 21:58:09.175247 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.175228    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.177644 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.177627    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dd0e35-openshift-default-dockercfg-jtdkt\""
Jul 17 21:58:09.186209 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.186190    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"]
Jul 17 21:58:09.301769 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.301740    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-workload-socket\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.301916 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.301788    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-credential-socket\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.301916 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.301811    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-token\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.301916 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.301873    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-podinfo\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.302031 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.301917    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istiod-ca-cert\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.302031 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.301940    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c4jp\" (UniqueName: \"kubernetes.io/projected/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-kube-api-access-7c4jp\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.302031 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.301977    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-envoy\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.302031 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.302005    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-data\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.302031 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.302024    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-workload-certs\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.402908 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.402876    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-credential-socket\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.403063 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.402916    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-token\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.403063 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.402953    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-podinfo\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.403063 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.402979    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istiod-ca-cert\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.403063 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.403006    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c4jp\" (UniqueName: \"kubernetes.io/projected/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-kube-api-access-7c4jp\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.403312 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.403152    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-envoy\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.403312 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.403192    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-data\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.403312 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.403225    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-workload-certs\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.403312 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.403261    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-workload-socket\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.403526 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.403316    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-credential-socket\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.403582 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.403538    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-data\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.403632 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.403600    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-workload-certs\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.403632 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.403619    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-workload-socket\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.403746 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.403727    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istiod-ca-cert\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.405361 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.405341    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-envoy\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.405598 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.405580    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-podinfo\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.410745 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.410719    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c4jp\" (UniqueName: \"kubernetes.io/projected/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-kube-api-access-7c4jp\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.411001 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.410984    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-token\") pod \"e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") " pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.485325 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.485270    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:09.616895 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.616869    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"]
Jul 17 21:58:09.619176 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:58:09.619148    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30cfba07_06c4_4cf3_a0ee_c95bc09e833e.slice/crio-935980dac507f4ca34714b1d667ea19b6e96eed4966b2b8d207264cee0c668e9 WatchSource:0}: Error finding container 935980dac507f4ca34714b1d667ea19b6e96eed4966b2b8d207264cee0c668e9: Status 404 returned error can't find the container with id 935980dac507f4ca34714b1d667ea19b6e96eed4966b2b8d207264cee0c668e9
Jul 17 21:58:09.621116 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.621082    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:58:09.621273 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.621145    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:58:09.621273 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.621177    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:58:09.643519 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:09.643493    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs" event={"ID":"30cfba07-06c4-4cf3-a0ee-c95bc09e833e","Type":"ContainerStarted","Data":"935980dac507f4ca34714b1d667ea19b6e96eed4966b2b8d207264cee0c668e9"}
Jul 17 21:58:10.648773 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:10.648736    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs" event={"ID":"30cfba07-06c4-4cf3-a0ee-c95bc09e833e","Type":"ContainerStarted","Data":"c9f1db2eb95152130f87ee2b410921ef5f6c06db703200add27292d407b1a2d7"}
Jul 17 21:58:10.669575 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:10.669525    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs" podStartSLOduration=1.669510746 podStartE2EDuration="1.669510746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:58:10.669452915 +0000 UTC m=+2990.633573524" watchObservedRunningTime="2026-07-17 21:58:10.669510746 +0000 UTC m=+2990.633631439"
Jul 17 21:58:11.486414 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:11.486383    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:11.490843 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:11.490818    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:11.652952 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:11.652924    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:11.653714 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:11.653695    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:20.861007 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:20.860976    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls"]
Jul 17 21:58:20.861397 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:20.861249    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1577289-b438-4210-9cd3-ce8153574531" containerName="istio-proxy"
Jul 17 21:58:20.861397 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:20.861260    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1577289-b438-4210-9cd3-ce8153574531" containerName="istio-proxy"
Jul 17 21:58:20.861397 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:20.861335    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d29b2c2d-0276-4a71-8841-94a333748961" containerName="revoke-keys"
Jul 17 21:58:20.864905 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:20.864888    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls"
Jul 17 21:58:20.867189 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:20.867171    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:58:20.867280 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:20.867202    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:58:20.871998 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:20.871772    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls"]
Jul 17 21:58:20.997594 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:20.997568    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-wbdls\" (UID: \"54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls"
Jul 17 21:58:21.098197 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:21.098174    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-wbdls\" (UID: \"54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls"
Jul 17 21:58:21.098783 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:21.098763    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-wbdls\" (UID: \"54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls"
Jul 17 21:58:21.176179 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:21.176156    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls"
Jul 17 21:58:21.303863 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:21.303821    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls"]
Jul 17 21:58:21.305949 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:58:21.305923    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ef5a35_8b3e_4a3a_aa4a_a413940b2bf5.slice/crio-10416a1179e25e77372372095d8a12e7117661ee4f6cc9b1ce6515768704e213 WatchSource:0}: Error finding container 10416a1179e25e77372372095d8a12e7117661ee4f6cc9b1ce6515768704e213: Status 404 returned error can't find the container with id 10416a1179e25e77372372095d8a12e7117661ee4f6cc9b1ce6515768704e213
Jul 17 21:58:21.697248 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:21.697216    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls" event={"ID":"54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5","Type":"ContainerStarted","Data":"1824f0216090ba287cd579cec2b1cb4297a969a35b199840f4c779a04a50bff9"}
Jul 17 21:58:21.697376 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:21.697271    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls" event={"ID":"54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5","Type":"ContainerStarted","Data":"10416a1179e25e77372372095d8a12e7117661ee4f6cc9b1ce6515768704e213"}
Jul 17 21:58:21.713620 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:21.713572    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls" podStartSLOduration=1.713558877 podStartE2EDuration="1.713558877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:58:21.711666763 +0000 UTC m=+3001.675787373" watchObservedRunningTime="2026-07-17 21:58:21.713558877 +0000 UTC m=+3001.677679487"
Jul 17 21:58:22.702728 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:22.702694    2573 generic.go:356] "Generic (PLEG): container finished" podID="54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5" containerID="1824f0216090ba287cd579cec2b1cb4297a969a35b199840f4c779a04a50bff9" exitCode=6
Jul 17 21:58:22.703118 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:22.702781    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls" event={"ID":"54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5","Type":"ContainerDied","Data":"1824f0216090ba287cd579cec2b1cb4297a969a35b199840f4c779a04a50bff9"}
Jul 17 21:58:22.703118 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:22.702896    2573 scope.go:117] "RemoveContainer" containerID="1824f0216090ba287cd579cec2b1cb4297a969a35b199840f4c779a04a50bff9"
Jul 17 21:58:23.708321 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:23.708264    2573 generic.go:356] "Generic (PLEG): container finished" podID="54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5" containerID="1611e9ca57c526e7468bdc043910d210860620c333e9f9b79383854b14e21ecc" exitCode=6
Jul 17 21:58:23.708743 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:23.708336    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls" event={"ID":"54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5","Type":"ContainerDied","Data":"1611e9ca57c526e7468bdc043910d210860620c333e9f9b79383854b14e21ecc"}
Jul 17 21:58:23.708743 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:23.708400    2573 scope.go:117] "RemoveContainer" containerID="1824f0216090ba287cd579cec2b1cb4297a969a35b199840f4c779a04a50bff9"
Jul 17 21:58:23.708743 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:23.708573    2573 scope.go:117] "RemoveContainer" containerID="1611e9ca57c526e7468bdc043910d210860620c333e9f9b79383854b14e21ecc"
Jul 17 21:58:23.708887 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:58:23.708782    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-wbdls_odh-ai-gateway-infra(54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls" podUID="54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5"
Jul 17 21:58:24.714115 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:24.714091    2573 scope.go:117] "RemoveContainer" containerID="1611e9ca57c526e7468bdc043910d210860620c333e9f9b79383854b14e21ecc"
Jul 17 21:58:24.714533 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:58:24.714275    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-wbdls_odh-ai-gateway-infra(54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls" podUID="54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5"
Jul 17 21:58:24.984360 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:24.984253    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380122-mcwr7"]
Jul 17 21:58:24.988605 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:24.988589    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380122-mcwr7"
Jul 17 21:58:24.993737 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:24.993711    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380122-mcwr7"]
Jul 17 21:58:25.142639 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:25.142590    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b807a2a-edb2-4d29-b12b-08bac7aac22f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-380122-mcwr7\" (UID: \"4b807a2a-edb2-4d29-b12b-08bac7aac22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380122-mcwr7"
Jul 17 21:58:25.250509 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:25.243084    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b807a2a-edb2-4d29-b12b-08bac7aac22f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-380122-mcwr7\" (UID: \"4b807a2a-edb2-4d29-b12b-08bac7aac22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380122-mcwr7"
Jul 17 21:58:25.250509 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:25.243897    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b807a2a-edb2-4d29-b12b-08bac7aac22f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-380122-mcwr7\" (UID: \"4b807a2a-edb2-4d29-b12b-08bac7aac22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380122-mcwr7"
Jul 17 21:58:25.300371 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:25.300328    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380122-mcwr7"
Jul 17 21:58:25.453306 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:25.445087    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380122-mcwr7"]
Jul 17 21:58:25.456740 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:58:25.456714    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b807a2a_edb2_4d29_b12b_08bac7aac22f.slice/crio-17e2e3339819931e52cf3ed4331f195b573b078e5796d0eed427e5b052eb0450 WatchSource:0}: Error finding container 17e2e3339819931e52cf3ed4331f195b573b078e5796d0eed427e5b052eb0450: Status 404 returned error can't find the container with id 17e2e3339819931e52cf3ed4331f195b573b078e5796d0eed427e5b052eb0450
Jul 17 21:58:25.718787 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:25.718755    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380122-mcwr7" event={"ID":"4b807a2a-edb2-4d29-b12b-08bac7aac22f","Type":"ContainerStarted","Data":"17e2e3339819931e52cf3ed4331f195b573b078e5796d0eed427e5b052eb0450"}
Jul 17 21:58:26.724561 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:26.724465    2573 generic.go:356] "Generic (PLEG): container finished" podID="4b807a2a-edb2-4d29-b12b-08bac7aac22f" containerID="9f6d67a5bd62f987d398ddc0051c206c0deab9d377e371a7df00c292c8d4a708" exitCode=0
Jul 17 21:58:26.724561 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:26.724549    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380122-mcwr7" event={"ID":"4b807a2a-edb2-4d29-b12b-08bac7aac22f","Type":"ContainerDied","Data":"9f6d67a5bd62f987d398ddc0051c206c0deab9d377e371a7df00c292c8d4a708"}
Jul 17 21:58:27.807179 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:27.807143    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj"]
Jul 17 21:58:27.810487 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:27.810465    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj"
Jul 17 21:58:27.820878 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:27.820654    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj"]
Jul 17 21:58:27.898386 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:27.898360    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380122-mcwr7"
Jul 17 21:58:27.974306 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:27.974249    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4956964e-e585-40f9-bccb-575be73a53d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj\" (UID: \"4956964e-e585-40f9-bccb-575be73a53d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj"
Jul 17 21:58:28.075547 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:28.075471    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b807a2a-edb2-4d29-b12b-08bac7aac22f-maas-api-service-ca\") pod \"4b807a2a-edb2-4d29-b12b-08bac7aac22f\" (UID: \"4b807a2a-edb2-4d29-b12b-08bac7aac22f\") "
Jul 17 21:58:28.075672 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:28.075656    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4956964e-e585-40f9-bccb-575be73a53d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj\" (UID: \"4956964e-e585-40f9-bccb-575be73a53d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj"
Jul 17 21:58:28.075793 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:28.075772    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b807a2a-edb2-4d29-b12b-08bac7aac22f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4b807a2a-edb2-4d29-b12b-08bac7aac22f" (UID: "4b807a2a-edb2-4d29-b12b-08bac7aac22f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:58:28.076265 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:28.076248    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4956964e-e585-40f9-bccb-575be73a53d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj\" (UID: \"4956964e-e585-40f9-bccb-575be73a53d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj"
Jul 17 21:58:28.122583 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:28.122564    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj"
Jul 17 21:58:28.176152 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:28.176056    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b807a2a-edb2-4d29-b12b-08bac7aac22f-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:58:28.282159 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:28.282134    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj"]
Jul 17 21:58:28.284442 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:58:28.284415    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4956964e_e585_40f9_bccb_575be73a53d5.slice/crio-3cbd2be29d6c0f9ff71ae66c725637700cfccd9551a4fafd95c1aedae4f820b5 WatchSource:0}: Error finding container 3cbd2be29d6c0f9ff71ae66c725637700cfccd9551a4fafd95c1aedae4f820b5: Status 404 returned error can't find the container with id 3cbd2be29d6c0f9ff71ae66c725637700cfccd9551a4fafd95c1aedae4f820b5
Jul 17 21:58:28.734857 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:28.734771    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj" event={"ID":"4956964e-e585-40f9-bccb-575be73a53d5","Type":"ContainerStarted","Data":"623cb41ce2c011afcb13267844338810dd468d9f5375886aeac167a5212aa2b5"}
Jul 17 21:58:28.734857 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:28.734829    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj" event={"ID":"4956964e-e585-40f9-bccb-575be73a53d5","Type":"ContainerStarted","Data":"3cbd2be29d6c0f9ff71ae66c725637700cfccd9551a4fafd95c1aedae4f820b5"}
Jul 17 21:58:28.736013 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:28.735992    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380122-mcwr7" event={"ID":"4b807a2a-edb2-4d29-b12b-08bac7aac22f","Type":"ContainerDied","Data":"17e2e3339819931e52cf3ed4331f195b573b078e5796d0eed427e5b052eb0450"}
Jul 17 21:58:28.736128 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:28.736024    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-380122-mcwr7"
Jul 17 21:58:28.736128 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:28.736029    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17e2e3339819931e52cf3ed4331f195b573b078e5796d0eed427e5b052eb0450"
Jul 17 21:58:28.752297 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:28.752250    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj" podStartSLOduration=1.7522367110000001 podStartE2EDuration="1.752236711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:58:28.751496234 +0000 UTC m=+3008.715616847" watchObservedRunningTime="2026-07-17 21:58:28.752236711 +0000 UTC m=+3008.716357359"
Jul 17 21:58:29.742507 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:29.742471    2573 generic.go:356] "Generic (PLEG): container finished" podID="4956964e-e585-40f9-bccb-575be73a53d5" containerID="623cb41ce2c011afcb13267844338810dd468d9f5375886aeac167a5212aa2b5" exitCode=6
Jul 17 21:58:29.742976 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:29.742565    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj" event={"ID":"4956964e-e585-40f9-bccb-575be73a53d5","Type":"ContainerDied","Data":"623cb41ce2c011afcb13267844338810dd468d9f5375886aeac167a5212aa2b5"}
Jul 17 21:58:29.742976 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:29.742795    2573 scope.go:117] "RemoveContainer" containerID="623cb41ce2c011afcb13267844338810dd468d9f5375886aeac167a5212aa2b5"
Jul 17 21:58:30.748029 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:30.747989    2573 generic.go:356] "Generic (PLEG): container finished" podID="4956964e-e585-40f9-bccb-575be73a53d5" containerID="3d9e4924f2a1bf1cb65ddfc095061094e39bf0970f202286576d3389c8fbcd59" exitCode=6
Jul 17 21:58:30.748537 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:30.748077    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj" event={"ID":"4956964e-e585-40f9-bccb-575be73a53d5","Type":"ContainerDied","Data":"3d9e4924f2a1bf1cb65ddfc095061094e39bf0970f202286576d3389c8fbcd59"}
Jul 17 21:58:30.748537 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:30.748137    2573 scope.go:117] "RemoveContainer" containerID="623cb41ce2c011afcb13267844338810dd468d9f5375886aeac167a5212aa2b5"
Jul 17 21:58:30.748537 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:30.748388    2573 scope.go:117] "RemoveContainer" containerID="3d9e4924f2a1bf1cb65ddfc095061094e39bf0970f202286576d3389c8fbcd59"
Jul 17 21:58:30.748724 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:58:30.748579    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj_odh-ai-gateway-infra(4956964e-e585-40f9-bccb-575be73a53d5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj" podUID="4956964e-e585-40f9-bccb-575be73a53d5"
Jul 17 21:58:31.754109 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:31.754078    2573 generic.go:356] "Generic (PLEG): container finished" podID="30cfba07-06c4-4cf3-a0ee-c95bc09e833e" containerID="c9f1db2eb95152130f87ee2b410921ef5f6c06db703200add27292d407b1a2d7" exitCode=0
Jul 17 21:58:31.754579 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:31.754150    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs" event={"ID":"30cfba07-06c4-4cf3-a0ee-c95bc09e833e","Type":"ContainerDied","Data":"c9f1db2eb95152130f87ee2b410921ef5f6c06db703200add27292d407b1a2d7"}
Jul 17 21:58:31.754635 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:31.754578    2573 scope.go:117] "RemoveContainer" containerID="c9f1db2eb95152130f87ee2b410921ef5f6c06db703200add27292d407b1a2d7"
Jul 17 21:58:31.755690 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:31.755654    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:58:31.755790 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:31.755731    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:58:31.755790 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:31.755764    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:58:31.755999 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:31.755985    2573 scope.go:117] "RemoveContainer" containerID="3d9e4924f2a1bf1cb65ddfc095061094e39bf0970f202286576d3389c8fbcd59"
Jul 17 21:58:31.756156 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:58:31.756138    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj_odh-ai-gateway-infra(4956964e-e585-40f9-bccb-575be73a53d5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj" podUID="4956964e-e585-40f9-bccb-575be73a53d5"
Jul 17 21:58:32.761514 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:32.761481    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs" event={"ID":"30cfba07-06c4-4cf3-a0ee-c95bc09e833e","Type":"ContainerStarted","Data":"2a5b4159d9e9ad2a68798e1990952b297fd06b748eacc5293c14593f3793d50e"}
Jul 17 21:58:33.486086 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:33.486058    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:33.486270 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:33.486227    2573 patch_prober.go:28] interesting pod/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Jul 17 21:58:33.486336 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:33.486272    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs" podUID="30cfba07-06c4-4cf3-a0ee-c95bc09e833e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Jul 17 21:58:34.486163 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:34.486127    2573 patch_prober.go:28] interesting pod/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Jul 17 21:58:34.486587 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:34.486183    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs" podUID="30cfba07-06c4-4cf3-a0ee-c95bc09e833e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Jul 17 21:58:35.485929 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:35.485895    2573 patch_prober.go:28] interesting pod/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Jul 17 21:58:35.486105 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:35.485948    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs" podUID="30cfba07-06c4-4cf3-a0ee-c95bc09e833e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Jul 17 21:58:36.486673 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:36.486633    2573 patch_prober.go:28] interesting pod/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Jul 17 21:58:36.487053 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:36.486693    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs" podUID="30cfba07-06c4-4cf3-a0ee-c95bc09e833e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Jul 17 21:58:37.490019 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:37.489991    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:37.490517 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:37.490310    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:37.491068 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:37.491042    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:58:38.524661 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:38.524632    2573 scope.go:117] "RemoveContainer" containerID="1611e9ca57c526e7468bdc043910d210860620c333e9f9b79383854b14e21ecc"
Jul 17 21:58:38.789146 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:38.789087    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls" event={"ID":"54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5","Type":"ContainerStarted","Data":"43b66a400ad8daaa6d35976f95befe937412bbba5baaa173914e9038bfa88fb2"}
Jul 17 21:58:39.794394 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:39.794358    2573 generic.go:356] "Generic (PLEG): container finished" podID="54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5" containerID="43b66a400ad8daaa6d35976f95befe937412bbba5baaa173914e9038bfa88fb2" exitCode=6
Jul 17 21:58:39.794851 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:39.794429    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls" event={"ID":"54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5","Type":"ContainerDied","Data":"43b66a400ad8daaa6d35976f95befe937412bbba5baaa173914e9038bfa88fb2"}
Jul 17 21:58:39.794851 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:39.794488    2573 scope.go:117] "RemoveContainer" containerID="1611e9ca57c526e7468bdc043910d210860620c333e9f9b79383854b14e21ecc"
Jul 17 21:58:39.794851 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:39.794726    2573 scope.go:117] "RemoveContainer" containerID="43b66a400ad8daaa6d35976f95befe937412bbba5baaa173914e9038bfa88fb2"
Jul 17 21:58:39.794978 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:58:39.794926    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-wbdls_odh-ai-gateway-infra(54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls" podUID="54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5"
Jul 17 21:58:39.821132 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:39.821108    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls"]
Jul 17 21:58:40.930215 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:40.930192    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls"
Jul 17 21:58:40.985772 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:40.985745    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5-maas-api-service-ca\") pod \"54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5\" (UID: \"54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5\") "
Jul 17 21:58:40.988364 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:40.986405    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5" (UID: "54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:58:41.087239 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:41.087218    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:58:41.805407 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:41.805377    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls"
Jul 17 21:58:41.805407 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:41.805387    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls" event={"ID":"54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5","Type":"ContainerDied","Data":"10416a1179e25e77372372095d8a12e7117661ee4f6cc9b1ce6515768704e213"}
Jul 17 21:58:41.805645 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:41.805450    2573 scope.go:117] "RemoveContainer" containerID="43b66a400ad8daaa6d35976f95befe937412bbba5baaa173914e9038bfa88fb2"
Jul 17 21:58:41.825698 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:41.825673    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls"]
Jul 17 21:58:41.831328 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:41.831305    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wbdls"]
Jul 17 21:58:42.529100 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:42.529071    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5" path="/var/lib/kubelet/pods/54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5/volumes"
Jul 17 21:58:45.579933 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:45.579894    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7"]
Jul 17 21:58:45.580433 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:45.580339    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5" containerName="revoke-keys"
Jul 17 21:58:45.580433 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:45.580358    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5" containerName="revoke-keys"
Jul 17 21:58:45.580433 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:45.580378    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b807a2a-edb2-4d29-b12b-08bac7aac22f" containerName="revoke-keys"
Jul 17 21:58:45.580433 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:45.580386    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b807a2a-edb2-4d29-b12b-08bac7aac22f" containerName="revoke-keys"
Jul 17 21:58:45.580433 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:45.580410    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5" containerName="revoke-keys"
Jul 17 21:58:45.580433 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:45.580418    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5" containerName="revoke-keys"
Jul 17 21:58:45.580743 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:45.580499    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b807a2a-edb2-4d29-b12b-08bac7aac22f" containerName="revoke-keys"
Jul 17 21:58:45.580743 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:45.580511    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5" containerName="revoke-keys"
Jul 17 21:58:45.580743 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:45.580521    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5" containerName="revoke-keys"
Jul 17 21:58:45.584853 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:45.584832    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7"
Jul 17 21:58:45.589669 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:45.589647    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7"]
Jul 17 21:58:45.620555 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:45.620528    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ffe57a-0f5d-47ec-984c-c4bc34c3a490-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7\" (UID: \"13ffe57a-0f5d-47ec-984c-c4bc34c3a490\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7"
Jul 17 21:58:45.721902 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:45.721875    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ffe57a-0f5d-47ec-984c-c4bc34c3a490-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7\" (UID: \"13ffe57a-0f5d-47ec-984c-c4bc34c3a490\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7"
Jul 17 21:58:45.722485 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:45.722468    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ffe57a-0f5d-47ec-984c-c4bc34c3a490-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7\" (UID: \"13ffe57a-0f5d-47ec-984c-c4bc34c3a490\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7"
Jul 17 21:58:45.896260 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:45.896237    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7"
Jul 17 21:58:46.021156 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:46.021133    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7"]
Jul 17 21:58:46.023734 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:58:46.023702    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13ffe57a_0f5d_47ec_984c_c4bc34c3a490.slice/crio-ba1beb9bd912f37913129dccaac6faa8f664ef010eb16207938860d0867d91d2 WatchSource:0}: Error finding container ba1beb9bd912f37913129dccaac6faa8f664ef010eb16207938860d0867d91d2: Status 404 returned error can't find the container with id ba1beb9bd912f37913129dccaac6faa8f664ef010eb16207938860d0867d91d2
Jul 17 21:58:46.524522 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:46.524461    2573 scope.go:117] "RemoveContainer" containerID="3d9e4924f2a1bf1cb65ddfc095061094e39bf0970f202286576d3389c8fbcd59"
Jul 17 21:58:46.832326 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:46.832259    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj" event={"ID":"4956964e-e585-40f9-bccb-575be73a53d5","Type":"ContainerStarted","Data":"1055307c5b40c83c67f42285b378fd0b3f777a1ef80713580957526aba51286a"}
Jul 17 21:58:46.833790 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:46.833764    2573 generic.go:356] "Generic (PLEG): container finished" podID="13ffe57a-0f5d-47ec-984c-c4bc34c3a490" containerID="bf977e7c62961d8944dd9063c6f28f2a87540fdbcdc9f611e3e2fc2d6eb68d8a" exitCode=6
Jul 17 21:58:46.833889 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:46.833815    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7" event={"ID":"13ffe57a-0f5d-47ec-984c-c4bc34c3a490","Type":"ContainerDied","Data":"bf977e7c62961d8944dd9063c6f28f2a87540fdbcdc9f611e3e2fc2d6eb68d8a"}
Jul 17 21:58:46.833889 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:46.833846    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7" event={"ID":"13ffe57a-0f5d-47ec-984c-c4bc34c3a490","Type":"ContainerStarted","Data":"ba1beb9bd912f37913129dccaac6faa8f664ef010eb16207938860d0867d91d2"}
Jul 17 21:58:46.833989 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:46.833979    2573 scope.go:117] "RemoveContainer" containerID="bf977e7c62961d8944dd9063c6f28f2a87540fdbcdc9f611e3e2fc2d6eb68d8a"
Jul 17 21:58:47.840928 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:47.840894    2573 generic.go:356] "Generic (PLEG): container finished" podID="4956964e-e585-40f9-bccb-575be73a53d5" containerID="1055307c5b40c83c67f42285b378fd0b3f777a1ef80713580957526aba51286a" exitCode=6
Jul 17 21:58:47.841375 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:47.840967    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj" event={"ID":"4956964e-e585-40f9-bccb-575be73a53d5","Type":"ContainerDied","Data":"1055307c5b40c83c67f42285b378fd0b3f777a1ef80713580957526aba51286a"}
Jul 17 21:58:47.841375 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:47.841030    2573 scope.go:117] "RemoveContainer" containerID="3d9e4924f2a1bf1cb65ddfc095061094e39bf0970f202286576d3389c8fbcd59"
Jul 17 21:58:47.841375 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:47.841357    2573 scope.go:117] "RemoveContainer" containerID="1055307c5b40c83c67f42285b378fd0b3f777a1ef80713580957526aba51286a"
Jul 17 21:58:47.841592 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:58:47.841574    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj_odh-ai-gateway-infra(4956964e-e585-40f9-bccb-575be73a53d5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj" podUID="4956964e-e585-40f9-bccb-575be73a53d5"
Jul 17 21:58:47.842915 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:47.842895    2573 generic.go:356] "Generic (PLEG): container finished" podID="13ffe57a-0f5d-47ec-984c-c4bc34c3a490" containerID="336affc549bd3530363f34dd7a1dfe044743e1fa7369abe760971b4a7100c1fd" exitCode=6
Jul 17 21:58:47.843026 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:47.842958    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7" event={"ID":"13ffe57a-0f5d-47ec-984c-c4bc34c3a490","Type":"ContainerDied","Data":"336affc549bd3530363f34dd7a1dfe044743e1fa7369abe760971b4a7100c1fd"}
Jul 17 21:58:47.843146 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:47.843135    2573 scope.go:117] "RemoveContainer" containerID="336affc549bd3530363f34dd7a1dfe044743e1fa7369abe760971b4a7100c1fd"
Jul 17 21:58:47.843354 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:58:47.843337    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7_odh-ai-gateway-infra(13ffe57a-0f5d-47ec-984c-c4bc34c3a490)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7" podUID="13ffe57a-0f5d-47ec-984c-c4bc34c3a490"
Jul 17 21:58:47.851147 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:47.851130    2573 scope.go:117] "RemoveContainer" containerID="bf977e7c62961d8944dd9063c6f28f2a87540fdbcdc9f611e3e2fc2d6eb68d8a"
Jul 17 21:58:47.856554 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:47.856529    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj"]
Jul 17 21:58:48.848730 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:48.848707    2573 scope.go:117] "RemoveContainer" containerID="336affc549bd3530363f34dd7a1dfe044743e1fa7369abe760971b4a7100c1fd"
Jul 17 21:58:48.849197 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:58:48.848919    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7_odh-ai-gateway-infra(13ffe57a-0f5d-47ec-984c-c4bc34c3a490)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7" podUID="13ffe57a-0f5d-47ec-984c-c4bc34c3a490"
Jul 17 21:58:48.980047 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:48.980025    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj"
Jul 17 21:58:49.048267 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.048241    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4956964e-e585-40f9-bccb-575be73a53d5-maas-api-service-ca\") pod \"4956964e-e585-40f9-bccb-575be73a53d5\" (UID: \"4956964e-e585-40f9-bccb-575be73a53d5\") "
Jul 17 21:58:49.048567 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.048545    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4956964e-e585-40f9-bccb-575be73a53d5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4956964e-e585-40f9-bccb-575be73a53d5" (UID: "4956964e-e585-40f9-bccb-575be73a53d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:58:49.149101 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.149048    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4956964e-e585-40f9-bccb-575be73a53d5-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:58:49.724993 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.724960    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb"]
Jul 17 21:58:49.725270 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.725257    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4956964e-e585-40f9-bccb-575be73a53d5" containerName="revoke-keys"
Jul 17 21:58:49.725352 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.725272    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4956964e-e585-40f9-bccb-575be73a53d5" containerName="revoke-keys"
Jul 17 21:58:49.725352 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.725305    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5" containerName="revoke-keys"
Jul 17 21:58:49.725352 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.725312    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5" containerName="revoke-keys"
Jul 17 21:58:49.725352 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.725323    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4956964e-e585-40f9-bccb-575be73a53d5" containerName="revoke-keys"
Jul 17 21:58:49.725352 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.725331    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4956964e-e585-40f9-bccb-575be73a53d5" containerName="revoke-keys"
Jul 17 21:58:49.725519 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.725382    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4956964e-e585-40f9-bccb-575be73a53d5" containerName="revoke-keys"
Jul 17 21:58:49.725519 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.725391    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="54ef5a35-8b3e-4a3a-aa4a-a413940b2bf5" containerName="revoke-keys"
Jul 17 21:58:49.725519 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.725400    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4956964e-e585-40f9-bccb-575be73a53d5" containerName="revoke-keys"
Jul 17 21:58:49.728193 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.728177    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb"
Jul 17 21:58:49.734728 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.734594    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb"]
Jul 17 21:58:49.754491 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.754468    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab04f272-9386-417d-9220-870a3ff71842-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb\" (UID: \"ab04f272-9386-417d-9220-870a3ff71842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb"
Jul 17 21:58:49.854859 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.854833    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab04f272-9386-417d-9220-870a3ff71842-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb\" (UID: \"ab04f272-9386-417d-9220-870a3ff71842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb"
Jul 17 21:58:49.855268 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.855249    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj"
Jul 17 21:58:49.855394 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.855280    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj" event={"ID":"4956964e-e585-40f9-bccb-575be73a53d5","Type":"ContainerDied","Data":"3cbd2be29d6c0f9ff71ae66c725637700cfccd9551a4fafd95c1aedae4f820b5"}
Jul 17 21:58:49.855394 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.855354    2573 scope.go:117] "RemoveContainer" containerID="1055307c5b40c83c67f42285b378fd0b3f777a1ef80713580957526aba51286a"
Jul 17 21:58:49.855562 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.855532    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab04f272-9386-417d-9220-870a3ff71842-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb\" (UID: \"ab04f272-9386-417d-9220-870a3ff71842\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb"
Jul 17 21:58:49.876223 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.876196    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj"]
Jul 17 21:58:49.880406 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:49.880387    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-nkcxj"]
Jul 17 21:58:50.039418 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:50.039344    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb"
Jul 17 21:58:50.161878 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:50.161853    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb"]
Jul 17 21:58:50.164303 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:58:50.164258    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab04f272_9386_417d_9220_870a3ff71842.slice/crio-c358aace7ff651c5972ff42d1612fd8b2b59333064e9c5b13943d615055c8f6e WatchSource:0}: Error finding container c358aace7ff651c5972ff42d1612fd8b2b59333064e9c5b13943d615055c8f6e: Status 404 returned error can't find the container with id c358aace7ff651c5972ff42d1612fd8b2b59333064e9c5b13943d615055c8f6e
Jul 17 21:58:50.530695 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:50.530659    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4956964e-e585-40f9-bccb-575be73a53d5" path="/var/lib/kubelet/pods/4956964e-e585-40f9-bccb-575be73a53d5/volumes"
Jul 17 21:58:50.869129 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:50.869099    2573 generic.go:356] "Generic (PLEG): container finished" podID="ab04f272-9386-417d-9220-870a3ff71842" containerID="dfff1202214776a22be42ffb145f4efca6d1179e4a3a85f537963ff1fa3e02f5" exitCode=6
Jul 17 21:58:50.869521 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:50.869157    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb" event={"ID":"ab04f272-9386-417d-9220-870a3ff71842","Type":"ContainerDied","Data":"dfff1202214776a22be42ffb145f4efca6d1179e4a3a85f537963ff1fa3e02f5"}
Jul 17 21:58:50.869521 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:50.869227    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb" event={"ID":"ab04f272-9386-417d-9220-870a3ff71842","Type":"ContainerStarted","Data":"c358aace7ff651c5972ff42d1612fd8b2b59333064e9c5b13943d615055c8f6e"}
Jul 17 21:58:50.869521 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:50.869356    2573 scope.go:117] "RemoveContainer" containerID="dfff1202214776a22be42ffb145f4efca6d1179e4a3a85f537963ff1fa3e02f5"
Jul 17 21:58:51.110075 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:51.110043    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8"]
Jul 17 21:58:51.110362 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:51.110349    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4956964e-e585-40f9-bccb-575be73a53d5" containerName="revoke-keys"
Jul 17 21:58:51.110422 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:51.110363    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4956964e-e585-40f9-bccb-575be73a53d5" containerName="revoke-keys"
Jul 17 21:58:51.110422 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:51.110416    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4956964e-e585-40f9-bccb-575be73a53d5" containerName="revoke-keys"
Jul 17 21:58:51.114330 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:51.114314    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8"
Jul 17 21:58:51.122914 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:51.122884    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8"]
Jul 17 21:58:51.169077 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:51.169056    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8\" (UID: \"55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8"
Jul 17 21:58:51.269507 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:51.269479    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8\" (UID: \"55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8"
Jul 17 21:58:51.270146 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:51.270121    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8\" (UID: \"55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8"
Jul 17 21:58:51.425616 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:51.425584    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8"
Jul 17 21:58:51.553859 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:51.553818    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8"]
Jul 17 21:58:51.585103 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:58:51.585073    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b3fcf9_6a2e_423e_934d_4b6fdcf9fb89.slice/crio-de62b3bd8fb2ea733e9bb433be0131389d9c73c93b4be6891433576c57d0496e WatchSource:0}: Error finding container de62b3bd8fb2ea733e9bb433be0131389d9c73c93b4be6891433576c57d0496e: Status 404 returned error can't find the container with id de62b3bd8fb2ea733e9bb433be0131389d9c73c93b4be6891433576c57d0496e
Jul 17 21:58:51.875737 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:51.875703    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8" event={"ID":"55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89","Type":"ContainerStarted","Data":"2a4f04e2a73e8f2cb2954641fe3b5800f2b4eb21bff6b5b9bd48cd417c09c4e6"}
Jul 17 21:58:51.876202 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:51.875768    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8" event={"ID":"55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89","Type":"ContainerStarted","Data":"de62b3bd8fb2ea733e9bb433be0131389d9c73c93b4be6891433576c57d0496e"}
Jul 17 21:58:51.877557 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:51.877528    2573 generic.go:356] "Generic (PLEG): container finished" podID="ab04f272-9386-417d-9220-870a3ff71842" containerID="fc1587150872f14b824cd50cc2d653733d63c591e0df3f229a178689781e9f0c" exitCode=6
Jul 17 21:58:51.877681 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:51.877593    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb" event={"ID":"ab04f272-9386-417d-9220-870a3ff71842","Type":"ContainerDied","Data":"fc1587150872f14b824cd50cc2d653733d63c591e0df3f229a178689781e9f0c"}
Jul 17 21:58:51.877681 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:51.877651    2573 scope.go:117] "RemoveContainer" containerID="dfff1202214776a22be42ffb145f4efca6d1179e4a3a85f537963ff1fa3e02f5"
Jul 17 21:58:51.877775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:51.877752    2573 scope.go:117] "RemoveContainer" containerID="fc1587150872f14b824cd50cc2d653733d63c591e0df3f229a178689781e9f0c"
Jul 17 21:58:51.877947 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:58:51.877920    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb_odh-ai-gateway-infra(ab04f272-9386-417d-9220-870a3ff71842)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb" podUID="ab04f272-9386-417d-9220-870a3ff71842"
Jul 17 21:58:51.891440 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:51.891398    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8" podStartSLOduration=0.891385188 podStartE2EDuration="891.385188ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:58:51.891348474 +0000 UTC m=+3031.855469085" watchObservedRunningTime="2026-07-17 21:58:51.891385188 +0000 UTC m=+3031.855505797"
Jul 17 21:58:52.883441 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:52.883417    2573 scope.go:117] "RemoveContainer" containerID="fc1587150872f14b824cd50cc2d653733d63c591e0df3f229a178689781e9f0c"
Jul 17 21:58:52.883904 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:58:52.883583    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb_odh-ai-gateway-infra(ab04f272-9386-417d-9220-870a3ff71842)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb" podUID="ab04f272-9386-417d-9220-870a3ff71842"
Jul 17 21:58:52.884658 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:52.884630    2573 generic.go:356] "Generic (PLEG): container finished" podID="55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89" containerID="2a4f04e2a73e8f2cb2954641fe3b5800f2b4eb21bff6b5b9bd48cd417c09c4e6" exitCode=0
Jul 17 21:58:52.884762 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:52.884698    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8" event={"ID":"55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89","Type":"ContainerDied","Data":"2a4f04e2a73e8f2cb2954641fe3b5800f2b4eb21bff6b5b9bd48cd417c09c4e6"}
Jul 17 21:58:54.010812 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:54.010789    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8"
Jul 17 21:58:54.092229 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:54.092203    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89-maas-api-service-ca\") pod \"55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89\" (UID: \"55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89\") "
Jul 17 21:58:54.092540 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:54.092520    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89" (UID: "55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:58:54.193540 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:54.193468    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:58:54.893883 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:54.893856    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8"
Jul 17 21:58:54.893883 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:54.893872    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8" event={"ID":"55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89","Type":"ContainerDied","Data":"de62b3bd8fb2ea733e9bb433be0131389d9c73c93b4be6891433576c57d0496e"}
Jul 17 21:58:54.894156 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:58:54.893913    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de62b3bd8fb2ea733e9bb433be0131389d9c73c93b4be6891433576c57d0496e"
Jul 17 21:59:02.524614 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:02.524585    2573 scope.go:117] "RemoveContainer" containerID="336affc549bd3530363f34dd7a1dfe044743e1fa7369abe760971b4a7100c1fd"
Jul 17 21:59:02.935875 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:02.935847    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7" event={"ID":"13ffe57a-0f5d-47ec-984c-c4bc34c3a490","Type":"ContainerStarted","Data":"0047045605a8b1afaea21b3985e6c74549c3e962c9616e7388da10c2e56b538a"}
Jul 17 21:59:03.942090 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:03.942056    2573 generic.go:356] "Generic (PLEG): container finished" podID="13ffe57a-0f5d-47ec-984c-c4bc34c3a490" containerID="0047045605a8b1afaea21b3985e6c74549c3e962c9616e7388da10c2e56b538a" exitCode=6
Jul 17 21:59:03.942527 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:03.942129    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7" event={"ID":"13ffe57a-0f5d-47ec-984c-c4bc34c3a490","Type":"ContainerDied","Data":"0047045605a8b1afaea21b3985e6c74549c3e962c9616e7388da10c2e56b538a"}
Jul 17 21:59:03.942527 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:03.942186    2573 scope.go:117] "RemoveContainer" containerID="336affc549bd3530363f34dd7a1dfe044743e1fa7369abe760971b4a7100c1fd"
Jul 17 21:59:03.942527 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:03.942436    2573 scope.go:117] "RemoveContainer" containerID="0047045605a8b1afaea21b3985e6c74549c3e962c9616e7388da10c2e56b538a"
Jul 17 21:59:03.942693 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:59:03.942670    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7_odh-ai-gateway-infra(13ffe57a-0f5d-47ec-984c-c4bc34c3a490)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7" podUID="13ffe57a-0f5d-47ec-984c-c4bc34c3a490"
Jul 17 21:59:03.959115 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:03.959072    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7" podStartSLOduration=18.959058882 podStartE2EDuration="18.959058882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:59:02.949618824 +0000 UTC m=+3042.913739434" watchObservedRunningTime="2026-07-17 21:59:03.959058882 +0000 UTC m=+3043.923179485"
Jul 17 21:59:03.959329 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:03.959316    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7"]
Jul 17 21:59:05.099117 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:05.099087    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7"
Jul 17 21:59:05.182898 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:05.182872    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ffe57a-0f5d-47ec-984c-c4bc34c3a490-maas-api-service-ca\") pod \"13ffe57a-0f5d-47ec-984c-c4bc34c3a490\" (UID: \"13ffe57a-0f5d-47ec-984c-c4bc34c3a490\") "
Jul 17 21:59:05.183194 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:05.183171    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13ffe57a-0f5d-47ec-984c-c4bc34c3a490-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "13ffe57a-0f5d-47ec-984c-c4bc34c3a490" (UID: "13ffe57a-0f5d-47ec-984c-c4bc34c3a490"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:59:05.283641 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:05.283584    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13ffe57a-0f5d-47ec-984c-c4bc34c3a490-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:59:05.953901 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:05.953870    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7"
Jul 17 21:59:05.953901 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:05.953884    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7" event={"ID":"13ffe57a-0f5d-47ec-984c-c4bc34c3a490","Type":"ContainerDied","Data":"ba1beb9bd912f37913129dccaac6faa8f664ef010eb16207938860d0867d91d2"}
Jul 17 21:59:05.954154 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:05.953950    2573 scope.go:117] "RemoveContainer" containerID="0047045605a8b1afaea21b3985e6c74549c3e962c9616e7388da10c2e56b538a"
Jul 17 21:59:05.976218 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:05.976194    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7"]
Jul 17 21:59:05.982156 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:05.982136    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-5hfs7"]
Jul 17 21:59:06.529895 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:06.529863    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13ffe57a-0f5d-47ec-984c-c4bc34c3a490" path="/var/lib/kubelet/pods/13ffe57a-0f5d-47ec-984c-c4bc34c3a490/volumes"
Jul 17 21:59:07.524920 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:07.524896    2573 scope.go:117] "RemoveContainer" containerID="fc1587150872f14b824cd50cc2d653733d63c591e0df3f229a178689781e9f0c"
Jul 17 21:59:07.965339 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:07.965310    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb" event={"ID":"ab04f272-9386-417d-9220-870a3ff71842","Type":"ContainerStarted","Data":"28d30b262053037df9392984c5bd82b633c0793ccf58041374226e537cc9e94d"}
Jul 17 21:59:07.983723 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:07.983679    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb" podStartSLOduration=18.983664807 podStartE2EDuration="18.983664807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:59:07.982415281 +0000 UTC m=+3047.946535891" watchObservedRunningTime="2026-07-17 21:59:07.983664807 +0000 UTC m=+3047.947785417"
Jul 17 21:59:08.970614 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:08.970583    2573 generic.go:356] "Generic (PLEG): container finished" podID="ab04f272-9386-417d-9220-870a3ff71842" containerID="28d30b262053037df9392984c5bd82b633c0793ccf58041374226e537cc9e94d" exitCode=6
Jul 17 21:59:08.971103 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:08.970656    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb" event={"ID":"ab04f272-9386-417d-9220-870a3ff71842","Type":"ContainerDied","Data":"28d30b262053037df9392984c5bd82b633c0793ccf58041374226e537cc9e94d"}
Jul 17 21:59:08.971103 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:08.970716    2573 scope.go:117] "RemoveContainer" containerID="fc1587150872f14b824cd50cc2d653733d63c591e0df3f229a178689781e9f0c"
Jul 17 21:59:08.971103 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:08.971009    2573 scope.go:117] "RemoveContainer" containerID="28d30b262053037df9392984c5bd82b633c0793ccf58041374226e537cc9e94d"
Jul 17 21:59:08.971275 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:59:08.971204    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb_odh-ai-gateway-infra(ab04f272-9386-417d-9220-870a3ff71842)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb" podUID="ab04f272-9386-417d-9220-870a3ff71842"
Jul 17 21:59:08.995997 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:08.995969    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb"]
Jul 17 21:59:10.103216 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:10.103191    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb"
Jul 17 21:59:10.220104 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:10.220076    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab04f272-9386-417d-9220-870a3ff71842-maas-api-service-ca\") pod \"ab04f272-9386-417d-9220-870a3ff71842\" (UID: \"ab04f272-9386-417d-9220-870a3ff71842\") "
Jul 17 21:59:10.220424 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:10.220402    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab04f272-9386-417d-9220-870a3ff71842-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ab04f272-9386-417d-9220-870a3ff71842" (UID: "ab04f272-9386-417d-9220-870a3ff71842"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:59:10.321227 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:10.321157    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab04f272-9386-417d-9220-870a3ff71842-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:59:10.981860 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:10.981833    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb"
Jul 17 21:59:10.981860 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:10.981845    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb" event={"ID":"ab04f272-9386-417d-9220-870a3ff71842","Type":"ContainerDied","Data":"c358aace7ff651c5972ff42d1612fd8b2b59333064e9c5b13943d615055c8f6e"}
Jul 17 21:59:10.982096 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:10.981907    2573 scope.go:117] "RemoveContainer" containerID="28d30b262053037df9392984c5bd82b633c0793ccf58041374226e537cc9e94d"
Jul 17 21:59:10.999116 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:10.999091    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb"]
Jul 17 21:59:11.004606 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:11.004585    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-2btkb"]
Jul 17 21:59:12.530161 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:12.530131    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab04f272-9386-417d-9220-870a3ff71842" path="/var/lib/kubelet/pods/ab04f272-9386-417d-9220-870a3ff71842/volumes"
Jul 17 21:59:16.868982 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:16.868937    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"]
Jul 17 21:59:16.869479 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:16.869211    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs" podUID="30cfba07-06c4-4cf3-a0ee-c95bc09e833e" containerName="istio-proxy" containerID="cri-o://2a5b4159d9e9ad2a68798e1990952b297fd06b748eacc5293c14593f3793d50e" gracePeriod=30
Jul 17 21:59:22.030365 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.030335    2573 generic.go:356] "Generic (PLEG): container finished" podID="30cfba07-06c4-4cf3-a0ee-c95bc09e833e" containerID="2a5b4159d9e9ad2a68798e1990952b297fd06b748eacc5293c14593f3793d50e" exitCode=0
Jul 17 21:59:22.030764 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.030406    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs" event={"ID":"30cfba07-06c4-4cf3-a0ee-c95bc09e833e","Type":"ContainerDied","Data":"2a5b4159d9e9ad2a68798e1990952b297fd06b748eacc5293c14593f3793d50e"}
Jul 17 21:59:22.030764 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.030470    2573 scope.go:117] "RemoveContainer" containerID="c9f1db2eb95152130f87ee2b410921ef5f6c06db703200add27292d407b1a2d7"
Jul 17 21:59:22.112180 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.112157    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:59:22.218605 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.218516    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-podinfo\") pod \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") "
Jul 17 21:59:22.218605 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.218552    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-workload-certs\") pod \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") "
Jul 17 21:59:22.218793 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.218611    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-workload-socket\") pod \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") "
Jul 17 21:59:22.218793 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.218712    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c4jp\" (UniqueName: \"kubernetes.io/projected/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-kube-api-access-7c4jp\") pod \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") "
Jul 17 21:59:22.218793 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.218744    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-data\") pod \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") "
Jul 17 21:59:22.218793 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.218770    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-token\") pod \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") "
Jul 17 21:59:22.219006 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.218794    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istiod-ca-cert\") pod \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") "
Jul 17 21:59:22.219006 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.218852    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-credential-socket\") pod \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") "
Jul 17 21:59:22.219006 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.218896    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-envoy\") pod \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\" (UID: \"30cfba07-06c4-4cf3-a0ee-c95bc09e833e\") "
Jul 17 21:59:22.219150 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.218889    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "30cfba07-06c4-4cf3-a0ee-c95bc09e833e" (UID: "30cfba07-06c4-4cf3-a0ee-c95bc09e833e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:59:22.219150 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.218900    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "30cfba07-06c4-4cf3-a0ee-c95bc09e833e" (UID: "30cfba07-06c4-4cf3-a0ee-c95bc09e833e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:59:22.219150 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.219100    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "30cfba07-06c4-4cf3-a0ee-c95bc09e833e" (UID: "30cfba07-06c4-4cf3-a0ee-c95bc09e833e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:59:22.219330 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.219205    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "30cfba07-06c4-4cf3-a0ee-c95bc09e833e" (UID: "30cfba07-06c4-4cf3-a0ee-c95bc09e833e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:59:22.219696 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.219674    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-data" (OuterVolumeSpecName: "istio-data") pod "30cfba07-06c4-4cf3-a0ee-c95bc09e833e" (UID: "30cfba07-06c4-4cf3-a0ee-c95bc09e833e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:59:22.221201 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.221158    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "30cfba07-06c4-4cf3-a0ee-c95bc09e833e" (UID: "30cfba07-06c4-4cf3-a0ee-c95bc09e833e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 21:59:22.221318 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.221269    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-token" (OuterVolumeSpecName: "istio-token") pod "30cfba07-06c4-4cf3-a0ee-c95bc09e833e" (UID: "30cfba07-06c4-4cf3-a0ee-c95bc09e833e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:59:22.221318 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.221273    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-kube-api-access-7c4jp" (OuterVolumeSpecName: "kube-api-access-7c4jp") pod "30cfba07-06c4-4cf3-a0ee-c95bc09e833e" (UID: "30cfba07-06c4-4cf3-a0ee-c95bc09e833e"). InnerVolumeSpecName "kube-api-access-7c4jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:59:22.221438 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.221411    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "30cfba07-06c4-4cf3-a0ee-c95bc09e833e" (UID: "30cfba07-06c4-4cf3-a0ee-c95bc09e833e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:59:22.319561 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.319530    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:59:22.319561 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.319555    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:59:22.319561 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.319564    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:59:22.319775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.319572    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:59:22.319775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.319581    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:59:22.319775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.319589    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:59:22.319775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.319597    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:59:22.319775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.319605    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c4jp\" (UniqueName: \"kubernetes.io/projected/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-kube-api-access-7c4jp\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:59:22.319775 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:22.319615    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30cfba07-06c4-4cf3-a0ee-c95bc09e833e-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 21:59:23.035881 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:23.035851    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"
Jul 17 21:59:23.036481 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:23.035852    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs" event={"ID":"30cfba07-06c4-4cf3-a0ee-c95bc09e833e","Type":"ContainerDied","Data":"935980dac507f4ca34714b1d667ea19b6e96eed4966b2b8d207264cee0c668e9"}
Jul 17 21:59:23.036481 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:23.035999    2573 scope.go:117] "RemoveContainer" containerID="2a5b4159d9e9ad2a68798e1990952b297fd06b748eacc5293c14593f3793d50e"
Jul 17 21:59:23.056470 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:23.056445    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"]
Jul 17 21:59:23.059828 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:23.059809    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dd0e35-openshift-default-5c744684f5-cj9bs"]
Jul 17 21:59:24.530848 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:24.530812    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30cfba07-06c4-4cf3-a0ee-c95bc09e833e" path="/var/lib/kubelet/pods/30cfba07-06c4-4cf3-a0ee-c95bc09e833e/volumes"
Jul 17 21:59:25.537926 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.537895    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"]
Jul 17 21:59:25.538301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538163    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89" containerName="revoke-keys"
Jul 17 21:59:25.538301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538175    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89" containerName="revoke-keys"
Jul 17 21:59:25.538301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538185    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13ffe57a-0f5d-47ec-984c-c4bc34c3a490" containerName="revoke-keys"
Jul 17 21:59:25.538301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538191    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="13ffe57a-0f5d-47ec-984c-c4bc34c3a490" containerName="revoke-keys"
Jul 17 21:59:25.538301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538199    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30cfba07-06c4-4cf3-a0ee-c95bc09e833e" containerName="istio-proxy"
Jul 17 21:59:25.538301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538204    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="30cfba07-06c4-4cf3-a0ee-c95bc09e833e" containerName="istio-proxy"
Jul 17 21:59:25.538301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538216    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab04f272-9386-417d-9220-870a3ff71842" containerName="revoke-keys"
Jul 17 21:59:25.538301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538221    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab04f272-9386-417d-9220-870a3ff71842" containerName="revoke-keys"
Jul 17 21:59:25.538301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538228    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab04f272-9386-417d-9220-870a3ff71842" containerName="revoke-keys"
Jul 17 21:59:25.538301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538233    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab04f272-9386-417d-9220-870a3ff71842" containerName="revoke-keys"
Jul 17 21:59:25.538301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538239    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30cfba07-06c4-4cf3-a0ee-c95bc09e833e" containerName="istio-proxy"
Jul 17 21:59:25.538301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538244    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="30cfba07-06c4-4cf3-a0ee-c95bc09e833e" containerName="istio-proxy"
Jul 17 21:59:25.538301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538249    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13ffe57a-0f5d-47ec-984c-c4bc34c3a490" containerName="revoke-keys"
Jul 17 21:59:25.538301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538254    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="13ffe57a-0f5d-47ec-984c-c4bc34c3a490" containerName="revoke-keys"
Jul 17 21:59:25.538301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538260    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13ffe57a-0f5d-47ec-984c-c4bc34c3a490" containerName="revoke-keys"
Jul 17 21:59:25.538301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538265    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="13ffe57a-0f5d-47ec-984c-c4bc34c3a490" containerName="revoke-keys"
Jul 17 21:59:25.538801 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538326    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="30cfba07-06c4-4cf3-a0ee-c95bc09e833e" containerName="istio-proxy"
Jul 17 21:59:25.538801 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538334    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="13ffe57a-0f5d-47ec-984c-c4bc34c3a490" containerName="revoke-keys"
Jul 17 21:59:25.538801 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538340    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="13ffe57a-0f5d-47ec-984c-c4bc34c3a490" containerName="revoke-keys"
Jul 17 21:59:25.538801 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538344    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab04f272-9386-417d-9220-870a3ff71842" containerName="revoke-keys"
Jul 17 21:59:25.538801 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538349    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab04f272-9386-417d-9220-870a3ff71842" containerName="revoke-keys"
Jul 17 21:59:25.538801 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538356    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab04f272-9386-417d-9220-870a3ff71842" containerName="revoke-keys"
Jul 17 21:59:25.538801 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538361    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="13ffe57a-0f5d-47ec-984c-c4bc34c3a490" containerName="revoke-keys"
Jul 17 21:59:25.538801 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538367    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89" containerName="revoke-keys"
Jul 17 21:59:25.538801 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.538373    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="30cfba07-06c4-4cf3-a0ee-c95bc09e833e" containerName="istio-proxy"
Jul 17 21:59:25.542735 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.542714    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.548364 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.548341    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-051ea5-openshift-default-dockercfg-5hzm5\""
Jul 17 21:59:25.564301 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.564254    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"]
Jul 17 21:59:25.644047 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.643996    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-istio-data\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.644253 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.644070    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-workload-socket\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.644253 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.644104    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nng9\" (UniqueName: \"kubernetes.io/projected/41e9349f-4a00-4427-bcc6-88f26802c273-kube-api-access-4nng9\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.644253 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.644205    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-credential-socket\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.644463 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.644256    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e9349f-4a00-4427-bcc6-88f26802c273-istio-podinfo\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.644463 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.644316    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e9349f-4a00-4427-bcc6-88f26802c273-istio-token\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.644463 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.644353    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-workload-certs\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.644463 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.644391    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-istio-envoy\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.644463 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.644415    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e9349f-4a00-4427-bcc6-88f26802c273-istiod-ca-cert\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.745113 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.745077    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e9349f-4a00-4427-bcc6-88f26802c273-istio-token\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.745113 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.745120    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-workload-certs\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.745406 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.745158    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-istio-envoy\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.745406 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.745175    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e9349f-4a00-4427-bcc6-88f26802c273-istiod-ca-cert\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.745406 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.745215    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-istio-data\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.745406 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.745241    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-workload-socket\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.745620 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.745472    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nng9\" (UniqueName: \"kubernetes.io/projected/41e9349f-4a00-4427-bcc6-88f26802c273-kube-api-access-4nng9\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.745620 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.745538    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-credential-socket\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.745620 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.745574    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e9349f-4a00-4427-bcc6-88f26802c273-istio-podinfo\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.745885 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.745706    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-workload-socket\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.745885 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.745740    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-workload-certs\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.745885 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.745849    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-istio-data\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.746055 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.745966    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-credential-socket\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.746152 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.746133    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e9349f-4a00-4427-bcc6-88f26802c273-istiod-ca-cert\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.747547 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.747519    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-istio-envoy\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.748034 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.748013    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e9349f-4a00-4427-bcc6-88f26802c273-istio-podinfo\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.753652 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.753623    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e9349f-4a00-4427-bcc6-88f26802c273-istio-token\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.753804 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.753783    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nng9\" (UniqueName: \"kubernetes.io/projected/41e9349f-4a00-4427-bcc6-88f26802c273-kube-api-access-4nng9\") pod \"e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") " pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.852123 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.852080    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:25.990459 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.990437    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"]
Jul 17 21:59:25.992263 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:59:25.992238    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e9349f_4a00_4427_bcc6_88f26802c273.slice/crio-628b5ed22f057299eaefe82e096f18745ec87f2a478239d21ee87feb8a83845a WatchSource:0}: Error finding container 628b5ed22f057299eaefe82e096f18745ec87f2a478239d21ee87feb8a83845a: Status 404 returned error can't find the container with id 628b5ed22f057299eaefe82e096f18745ec87f2a478239d21ee87feb8a83845a
Jul 17 21:59:25.994380 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.994344    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:59:25.994470 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.994430    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:59:25.994470 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:25.994460    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:59:26.052870 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:26.052841    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9" event={"ID":"41e9349f-4a00-4427-bcc6-88f26802c273","Type":"ContainerStarted","Data":"628b5ed22f057299eaefe82e096f18745ec87f2a478239d21ee87feb8a83845a"}
Jul 17 21:59:27.058053 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:27.058016    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9" event={"ID":"41e9349f-4a00-4427-bcc6-88f26802c273","Type":"ContainerStarted","Data":"dfe0c924cf0cad18b44866d294090a46d03183e7ada180c42be1de97261e31e2"}
Jul 17 21:59:27.078861 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:27.078809    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9" podStartSLOduration=2.078792254 podStartE2EDuration="2.078792254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:59:27.077203715 +0000 UTC m=+3067.041324337" watchObservedRunningTime="2026-07-17 21:59:27.078792254 +0000 UTC m=+3067.042912863"
Jul 17 21:59:27.853210 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:27.853174    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:27.857993 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:27.857968    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:28.062505 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:28.062476    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:28.063654 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:28.063636    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:40.112814 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:40.112782    2573 generic.go:356] "Generic (PLEG): container finished" podID="41e9349f-4a00-4427-bcc6-88f26802c273" containerID="dfe0c924cf0cad18b44866d294090a46d03183e7ada180c42be1de97261e31e2" exitCode=0
Jul 17 21:59:40.113241 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:40.112855    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9" event={"ID":"41e9349f-4a00-4427-bcc6-88f26802c273","Type":"ContainerDied","Data":"dfe0c924cf0cad18b44866d294090a46d03183e7ada180c42be1de97261e31e2"}
Jul 17 21:59:40.113241 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:40.113219    2573 scope.go:117] "RemoveContainer" containerID="dfe0c924cf0cad18b44866d294090a46d03183e7ada180c42be1de97261e31e2"
Jul 17 21:59:40.114315 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:40.114270    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:59:40.114387 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:40.114370    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:59:40.114438 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:40.114421    2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 17 21:59:41.118887 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:41.118850    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9" event={"ID":"41e9349f-4a00-4427-bcc6-88f26802c273","Type":"ContainerStarted","Data":"c310df8531813fc7af23eb294e59d3b0ce83de0e107ea79da4d243b7fd19cf9a"}
Jul 17 21:59:41.852875 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:41.852836    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:41.854178 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:41.854153    2573 patch_prober.go:28] interesting pod/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Jul 17 21:59:41.854326 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:41.854200    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9" podUID="41e9349f-4a00-4427-bcc6-88f26802c273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Jul 17 21:59:42.853342 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:42.853301    2573 patch_prober.go:28] interesting pod/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Jul 17 21:59:42.853851 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:42.853355    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9" podUID="41e9349f-4a00-4427-bcc6-88f26802c273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Jul 17 21:59:43.062808 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:43.062769    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:43.852544 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:43.852508    2573 patch_prober.go:28] interesting pod/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Jul 17 21:59:43.852726 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:43.852562    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9" podUID="41e9349f-4a00-4427-bcc6-88f26802c273" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Jul 17 21:59:44.860899 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:44.860865    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:44.862758 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:44.862734    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 21:59:45.624883 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:45.624849    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr"]
Jul 17 21:59:45.625164 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:45.625151    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab04f272-9386-417d-9220-870a3ff71842" containerName="revoke-keys"
Jul 17 21:59:45.625219 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:45.625166    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab04f272-9386-417d-9220-870a3ff71842" containerName="revoke-keys"
Jul 17 21:59:45.626764 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:45.626748    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr"
Jul 17 21:59:45.629506 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:45.629485    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 21:59:45.629741 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:45.629724    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:59:45.645150 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:45.645121    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr"]
Jul 17 21:59:45.719391 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:45.719362    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed251fc-0c05-4574-9c15-e699836c3f83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr\" (UID: \"aed251fc-0c05-4574-9c15-e699836c3f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr"
Jul 17 21:59:45.820216 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:45.820195    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed251fc-0c05-4574-9c15-e699836c3f83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr\" (UID: \"aed251fc-0c05-4574-9c15-e699836c3f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr"
Jul 17 21:59:45.820791 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:45.820772    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed251fc-0c05-4574-9c15-e699836c3f83-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr\" (UID: \"aed251fc-0c05-4574-9c15-e699836c3f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr"
Jul 17 21:59:45.936972 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:45.936919    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr"
Jul 17 21:59:46.059423 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:46.059399    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr"]
Jul 17 21:59:46.061890 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:59:46.061863    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed251fc_0c05_4574_9c15_e699836c3f83.slice/crio-9c19827030abf097d92245c6cdeded2c0dd12ab66bfdf2162ee216928a803951 WatchSource:0}: Error finding container 9c19827030abf097d92245c6cdeded2c0dd12ab66bfdf2162ee216928a803951: Status 404 returned error can't find the container with id 9c19827030abf097d92245c6cdeded2c0dd12ab66bfdf2162ee216928a803951
Jul 17 21:59:46.142373 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:46.142343    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr" event={"ID":"aed251fc-0c05-4574-9c15-e699836c3f83","Type":"ContainerStarted","Data":"9c19827030abf097d92245c6cdeded2c0dd12ab66bfdf2162ee216928a803951"}
Jul 17 21:59:47.148417 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:47.148383    2573 generic.go:356] "Generic (PLEG): container finished" podID="aed251fc-0c05-4574-9c15-e699836c3f83" containerID="4e323e1c13aa93a2b2a809c7b2a1a4ce0135e538a95b6d98b9b0734bae48c830" exitCode=6
Jul 17 21:59:47.148792 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:47.148441    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr" event={"ID":"aed251fc-0c05-4574-9c15-e699836c3f83","Type":"ContainerDied","Data":"4e323e1c13aa93a2b2a809c7b2a1a4ce0135e538a95b6d98b9b0734bae48c830"}
Jul 17 21:59:47.148792 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:47.148646    2573 scope.go:117] "RemoveContainer" containerID="4e323e1c13aa93a2b2a809c7b2a1a4ce0135e538a95b6d98b9b0734bae48c830"
Jul 17 21:59:48.154953 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:48.154917    2573 generic.go:356] "Generic (PLEG): container finished" podID="aed251fc-0c05-4574-9c15-e699836c3f83" containerID="47a0b5d6aeda37d7b71222aa0d583ceb3c1e59e376ec2678ae43b78e5882672d" exitCode=6
Jul 17 21:59:48.155381 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:48.154982    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr" event={"ID":"aed251fc-0c05-4574-9c15-e699836c3f83","Type":"ContainerDied","Data":"47a0b5d6aeda37d7b71222aa0d583ceb3c1e59e376ec2678ae43b78e5882672d"}
Jul 17 21:59:48.155381 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:48.155029    2573 scope.go:117] "RemoveContainer" containerID="4e323e1c13aa93a2b2a809c7b2a1a4ce0135e538a95b6d98b9b0734bae48c830"
Jul 17 21:59:48.155381 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:48.155179    2573 scope.go:117] "RemoveContainer" containerID="47a0b5d6aeda37d7b71222aa0d583ceb3c1e59e376ec2678ae43b78e5882672d"
Jul 17 21:59:48.155546 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:59:48.155415    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr_odh-ai-gateway-infra(aed251fc-0c05-4574-9c15-e699836c3f83)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr" podUID="aed251fc-0c05-4574-9c15-e699836c3f83"
Jul 17 21:59:49.161158 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:49.161133    2573 scope.go:117] "RemoveContainer" containerID="47a0b5d6aeda37d7b71222aa0d583ceb3c1e59e376ec2678ae43b78e5882672d"
Jul 17 21:59:49.161589 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:59:49.161310    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr_odh-ai-gateway-infra(aed251fc-0c05-4574-9c15-e699836c3f83)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr" podUID="aed251fc-0c05-4574-9c15-e699836c3f83"
Jul 17 21:59:49.776173 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:49.776145    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx"]
Jul 17 21:59:49.778515 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:49.778499    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx"
Jul 17 21:59:49.786645 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:49.786619    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx"]
Jul 17 21:59:49.849056 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:49.849031    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3cd75a-18d8-4feb-bd5a-696dcae50e26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx\" (UID: \"bc3cd75a-18d8-4feb-bd5a-696dcae50e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx"
Jul 17 21:59:49.950176 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:49.950152    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3cd75a-18d8-4feb-bd5a-696dcae50e26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx\" (UID: \"bc3cd75a-18d8-4feb-bd5a-696dcae50e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx"
Jul 17 21:59:49.950734 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:49.950717    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3cd75a-18d8-4feb-bd5a-696dcae50e26-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx\" (UID: \"bc3cd75a-18d8-4feb-bd5a-696dcae50e26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx"
Jul 17 21:59:50.089635 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:50.089604    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx"
Jul 17 21:59:50.214409 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:50.214381    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx"]
Jul 17 21:59:50.217329 ip-10-0-128-246 kubenswrapper[2573]: W0717 21:59:50.217300    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3cd75a_18d8_4feb_bd5a_696dcae50e26.slice/crio-691a90dcf38ade560a2fb9ee78c791adce98be5dc89370385d2d3c469760324d WatchSource:0}: Error finding container 691a90dcf38ade560a2fb9ee78c791adce98be5dc89370385d2d3c469760324d: Status 404 returned error can't find the container with id 691a90dcf38ade560a2fb9ee78c791adce98be5dc89370385d2d3c469760324d
Jul 17 21:59:51.172379 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:51.172348    2573 generic.go:356] "Generic (PLEG): container finished" podID="bc3cd75a-18d8-4feb-bd5a-696dcae50e26" containerID="5ef7250c17e12a9a7fae95455a13d5511227893b2b2cccbc0b8b28a7b1ce992a" exitCode=6
Jul 17 21:59:51.172556 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:51.172385    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx" event={"ID":"bc3cd75a-18d8-4feb-bd5a-696dcae50e26","Type":"ContainerDied","Data":"5ef7250c17e12a9a7fae95455a13d5511227893b2b2cccbc0b8b28a7b1ce992a"}
Jul 17 21:59:51.172556 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:51.172420    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx" event={"ID":"bc3cd75a-18d8-4feb-bd5a-696dcae50e26","Type":"ContainerStarted","Data":"691a90dcf38ade560a2fb9ee78c791adce98be5dc89370385d2d3c469760324d"}
Jul 17 21:59:51.172637 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:51.172622    2573 scope.go:117] "RemoveContainer" containerID="5ef7250c17e12a9a7fae95455a13d5511227893b2b2cccbc0b8b28a7b1ce992a"
Jul 17 21:59:52.178511 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:52.178481    2573 generic.go:356] "Generic (PLEG): container finished" podID="bc3cd75a-18d8-4feb-bd5a-696dcae50e26" containerID="76f482407c18869e710c5b2e4b6f46438b5be785cdf1b7ec1596b641d208c344" exitCode=6
Jul 17 21:59:52.178913 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:52.178572    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx" event={"ID":"bc3cd75a-18d8-4feb-bd5a-696dcae50e26","Type":"ContainerDied","Data":"76f482407c18869e710c5b2e4b6f46438b5be785cdf1b7ec1596b641d208c344"}
Jul 17 21:59:52.178913 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:52.178651    2573 scope.go:117] "RemoveContainer" containerID="5ef7250c17e12a9a7fae95455a13d5511227893b2b2cccbc0b8b28a7b1ce992a"
Jul 17 21:59:52.178913 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:52.178758    2573 scope.go:117] "RemoveContainer" containerID="76f482407c18869e710c5b2e4b6f46438b5be785cdf1b7ec1596b641d208c344"
Jul 17 21:59:52.179052 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:59:52.178971    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx_odh-ai-gateway-infra(bc3cd75a-18d8-4feb-bd5a-696dcae50e26)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx" podUID="bc3cd75a-18d8-4feb-bd5a-696dcae50e26"
Jul 17 21:59:53.184758 ip-10-0-128-246 kubenswrapper[2573]: I0717 21:59:53.184727    2573 scope.go:117] "RemoveContainer" containerID="76f482407c18869e710c5b2e4b6f46438b5be785cdf1b7ec1596b641d208c344"
Jul 17 21:59:53.185166 ip-10-0-128-246 kubenswrapper[2573]: E0717 21:59:53.184897    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx_odh-ai-gateway-infra(bc3cd75a-18d8-4feb-bd5a-696dcae50e26)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx" podUID="bc3cd75a-18d8-4feb-bd5a-696dcae50e26"
Jul 17 22:00:00.132666 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:00.132630    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ac333d-29738760-4jchp"]
Jul 17 22:00:00.136015 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:00.135994    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-051ea5-29738760-s4f5f"]
Jul 17 22:00:00.136156 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:00.136100    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ac333d-29738760-4jchp"
Jul 17 22:00:00.138112 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:00.138093    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738760-569qx"]
Jul 17 22:00:00.138227 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:00.138199    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-051ea5-29738760-s4f5f"
Jul 17 22:00:00.140387 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:00.140370    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-569qx"
Jul 17 22:00:00.144136 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:00.144116    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ac333d-29738760-4jchp"]
Jul 17 22:00:00.146116 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:00.146085    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-051ea5-29738760-s4f5f"]
Jul 17 22:00:00.147938 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:00.147921    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ac333d-29738760-4jchp"
Jul 17 22:00:00.148206 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:00.148182    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738760-569qx"]
Jul 17 22:00:00.158212 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:00.158194    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-051ea5-29738760-s4f5f"
Jul 17 22:00:00.189749 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:00.189727    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-569qx"
Jul 17 22:00:00.305154 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:00.305120    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ac333d-29738760-4jchp"]
Jul 17 22:00:00.310132 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:00:00.309891    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31fa2354_db49_41d0_8b4f_bb5f713faa84.slice/crio-02014a792aaf8290d2defadc9049bf0c27a3f015cb53b7745102eb3f93a265ca WatchSource:0}: Error finding container 02014a792aaf8290d2defadc9049bf0c27a3f015cb53b7745102eb3f93a265ca: Status 404 returned error can't find the container with id 02014a792aaf8290d2defadc9049bf0c27a3f015cb53b7745102eb3f93a265ca
Jul 17 22:00:00.327485 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:00.327437    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-051ea5-29738760-s4f5f"]
Jul 17 22:00:00.330780 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:00:00.330739    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3801dad_7ae9_4cf5_a24d_e60376ff1490.slice/crio-3944d69adf19beeefaa7f0f37ac0657f4902a7fd29132fd7a4d61f98d9d02500 WatchSource:0}: Error finding container 3944d69adf19beeefaa7f0f37ac0657f4902a7fd29132fd7a4d61f98d9d02500: Status 404 returned error can't find the container with id 3944d69adf19beeefaa7f0f37ac0657f4902a7fd29132fd7a4d61f98d9d02500
Jul 17 22:00:00.360540 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:00.360519    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738760-569qx"]
Jul 17 22:00:00.363363 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:00:00.363332    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38b12d6_9aed_4f15_b927_a25b17914f27.slice/crio-e8c67ee1f7f7b832cc43d1d38005d4ec1cdff9ecdc135d23655d29ab796d4f3f WatchSource:0}: Error finding container e8c67ee1f7f7b832cc43d1d38005d4ec1cdff9ecdc135d23655d29ab796d4f3f: Status 404 returned error can't find the container with id e8c67ee1f7f7b832cc43d1d38005d4ec1cdff9ecdc135d23655d29ab796d4f3f
Jul 17 22:00:01.223811 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:01.223775    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ac333d-29738760-4jchp" event={"ID":"31fa2354-db49-41d0-8b4f-bb5f713faa84","Type":"ContainerStarted","Data":"e38b762369701c3eade9894c9c48e47e1a7b14d38de0da6fd21e5ae6b0efe1e3"}
Jul 17 22:00:01.224205 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:01.223833    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ac333d-29738760-4jchp" event={"ID":"31fa2354-db49-41d0-8b4f-bb5f713faa84","Type":"ContainerStarted","Data":"02014a792aaf8290d2defadc9049bf0c27a3f015cb53b7745102eb3f93a265ca"}
Jul 17 22:00:01.225344 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:01.225315    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-569qx" event={"ID":"c38b12d6-9aed-4f15-b927-a25b17914f27","Type":"ContainerStarted","Data":"33b09565740599c7ea39024e8603d64e295696796d6477fbc0ea7c0199cf572d"}
Jul 17 22:00:01.225478 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:01.225360    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-569qx" event={"ID":"c38b12d6-9aed-4f15-b927-a25b17914f27","Type":"ContainerStarted","Data":"e8c67ee1f7f7b832cc43d1d38005d4ec1cdff9ecdc135d23655d29ab796d4f3f"}
Jul 17 22:00:01.226573 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:01.226552    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-051ea5-29738760-s4f5f" event={"ID":"f3801dad-7ae9-4cf5-a24d-e60376ff1490","Type":"ContainerStarted","Data":"c6b3ddb3d2c0974d25e3c0cd1aa2d526d7379a312d3428e8d8c12dbb77e0588b"}
Jul 17 22:00:01.226701 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:01.226596    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-051ea5-29738760-s4f5f" event={"ID":"f3801dad-7ae9-4cf5-a24d-e60376ff1490","Type":"ContainerStarted","Data":"3944d69adf19beeefaa7f0f37ac0657f4902a7fd29132fd7a4d61f98d9d02500"}
Jul 17 22:00:01.239936 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:01.239889    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ac333d-29738760-4jchp" podStartSLOduration=1.239874431 podStartE2EDuration="1.239874431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:00:01.239216229 +0000 UTC m=+3101.203336841" watchObservedRunningTime="2026-07-17 22:00:01.239874431 +0000 UTC m=+3101.203995040"
Jul 17 22:00:01.254194 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:01.254151    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-569qx" podStartSLOduration=1.254133248 podStartE2EDuration="1.254133248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:00:01.253706916 +0000 UTC m=+3101.217827526" watchObservedRunningTime="2026-07-17 22:00:01.254133248 +0000 UTC m=+3101.218253861"
Jul 17 22:00:01.269544 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:01.269504    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-051ea5-29738760-s4f5f" podStartSLOduration=1.269490918 podStartE2EDuration="1.269490918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:00:01.268888538 +0000 UTC m=+3101.233009149" watchObservedRunningTime="2026-07-17 22:00:01.269490918 +0000 UTC m=+3101.233611528"
Jul 17 22:00:01.524850 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:01.524766    2573 scope.go:117] "RemoveContainer" containerID="47a0b5d6aeda37d7b71222aa0d583ceb3c1e59e376ec2678ae43b78e5882672d"
Jul 17 22:00:02.232885 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:02.232801    2573 generic.go:356] "Generic (PLEG): container finished" podID="f3801dad-7ae9-4cf5-a24d-e60376ff1490" containerID="c6b3ddb3d2c0974d25e3c0cd1aa2d526d7379a312d3428e8d8c12dbb77e0588b" exitCode=0
Jul 17 22:00:02.233339 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:02.232882    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-051ea5-29738760-s4f5f" event={"ID":"f3801dad-7ae9-4cf5-a24d-e60376ff1490","Type":"ContainerDied","Data":"c6b3ddb3d2c0974d25e3c0cd1aa2d526d7379a312d3428e8d8c12dbb77e0588b"}
Jul 17 22:00:02.234728 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:02.234704    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr" event={"ID":"aed251fc-0c05-4574-9c15-e699836c3f83","Type":"ContainerStarted","Data":"8f733849b37e28c6fa24dd6e0ab8f87e7dc76d9ede1ee8d8486cd00661090989"}
Jul 17 22:00:02.236086 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:02.236055    2573 generic.go:356] "Generic (PLEG): container finished" podID="31fa2354-db49-41d0-8b4f-bb5f713faa84" containerID="e38b762369701c3eade9894c9c48e47e1a7b14d38de0da6fd21e5ae6b0efe1e3" exitCode=0
Jul 17 22:00:02.236187 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:02.236131    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ac333d-29738760-4jchp" event={"ID":"31fa2354-db49-41d0-8b4f-bb5f713faa84","Type":"ContainerDied","Data":"e38b762369701c3eade9894c9c48e47e1a7b14d38de0da6fd21e5ae6b0efe1e3"}
Jul 17 22:00:02.237605 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:02.237584    2573 generic.go:356] "Generic (PLEG): container finished" podID="c38b12d6-9aed-4f15-b927-a25b17914f27" containerID="33b09565740599c7ea39024e8603d64e295696796d6477fbc0ea7c0199cf572d" exitCode=0
Jul 17 22:00:02.237685 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:02.237623    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-569qx" event={"ID":"c38b12d6-9aed-4f15-b927-a25b17914f27","Type":"ContainerDied","Data":"33b09565740599c7ea39024e8603d64e295696796d6477fbc0ea7c0199cf572d"}
Jul 17 22:00:02.288022 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:02.287959    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr" podStartSLOduration=17.287946094 podStartE2EDuration="17.287946094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:00:02.28727151 +0000 UTC m=+3102.251392116" watchObservedRunningTime="2026-07-17 22:00:02.287946094 +0000 UTC m=+3102.252066773"
Jul 17 22:00:03.244276 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:03.244244    2573 generic.go:356] "Generic (PLEG): container finished" podID="aed251fc-0c05-4574-9c15-e699836c3f83" containerID="8f733849b37e28c6fa24dd6e0ab8f87e7dc76d9ede1ee8d8486cd00661090989" exitCode=6
Jul 17 22:00:03.244710 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:03.244328    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr" event={"ID":"aed251fc-0c05-4574-9c15-e699836c3f83","Type":"ContainerDied","Data":"8f733849b37e28c6fa24dd6e0ab8f87e7dc76d9ede1ee8d8486cd00661090989"}
Jul 17 22:00:03.244710 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:03.244401    2573 scope.go:117] "RemoveContainer" containerID="47a0b5d6aeda37d7b71222aa0d583ceb3c1e59e376ec2678ae43b78e5882672d"
Jul 17 22:00:03.245114 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:03.244816    2573 scope.go:117] "RemoveContainer" containerID="8f733849b37e28c6fa24dd6e0ab8f87e7dc76d9ede1ee8d8486cd00661090989"
Jul 17 22:00:03.245114 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:00:03.244998    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr_odh-ai-gateway-infra(aed251fc-0c05-4574-9c15-e699836c3f83)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr" podUID="aed251fc-0c05-4574-9c15-e699836c3f83"
Jul 17 22:00:03.297400 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:03.297363    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr"]
Jul 17 22:00:03.369619 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:03.369591    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-051ea5-29738760-s4f5f"
Jul 17 22:00:03.414551 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:03.414530    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-569qx"
Jul 17 22:00:03.443592 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:03.443571    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ac333d-29738760-4jchp"
Jul 17 22:00:04.251708 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:04.251677    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ac333d-29738760-4jchp"
Jul 17 22:00:04.251708 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:04.251695    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ac333d-29738760-4jchp" event={"ID":"31fa2354-db49-41d0-8b4f-bb5f713faa84","Type":"ContainerDied","Data":"02014a792aaf8290d2defadc9049bf0c27a3f015cb53b7745102eb3f93a265ca"}
Jul 17 22:00:04.252093 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:04.251771    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02014a792aaf8290d2defadc9049bf0c27a3f015cb53b7745102eb3f93a265ca"
Jul 17 22:00:04.253099 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:04.253079    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-569qx"
Jul 17 22:00:04.253218 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:04.253099    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-569qx" event={"ID":"c38b12d6-9aed-4f15-b927-a25b17914f27","Type":"ContainerDied","Data":"e8c67ee1f7f7b832cc43d1d38005d4ec1cdff9ecdc135d23655d29ab796d4f3f"}
Jul 17 22:00:04.253218 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:04.253135    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8c67ee1f7f7b832cc43d1d38005d4ec1cdff9ecdc135d23655d29ab796d4f3f"
Jul 17 22:00:04.254522 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:04.254504    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-051ea5-29738760-s4f5f"
Jul 17 22:00:04.254621 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:04.254503    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-051ea5-29738760-s4f5f" event={"ID":"f3801dad-7ae9-4cf5-a24d-e60376ff1490","Type":"ContainerDied","Data":"3944d69adf19beeefaa7f0f37ac0657f4902a7fd29132fd7a4d61f98d9d02500"}
Jul 17 22:00:04.254621 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:04.254613    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3944d69adf19beeefaa7f0f37ac0657f4902a7fd29132fd7a4d61f98d9d02500"
Jul 17 22:00:04.379382 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:04.379358    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr"
Jul 17 22:00:04.464124 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:04.464099    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed251fc-0c05-4574-9c15-e699836c3f83-maas-api-service-ca\") pod \"aed251fc-0c05-4574-9c15-e699836c3f83\" (UID: \"aed251fc-0c05-4574-9c15-e699836c3f83\") "
Jul 17 22:00:04.464491 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:04.464466    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed251fc-0c05-4574-9c15-e699836c3f83-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aed251fc-0c05-4574-9c15-e699836c3f83" (UID: "aed251fc-0c05-4574-9c15-e699836c3f83"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:00:04.565354 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:04.565327    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed251fc-0c05-4574-9c15-e699836c3f83-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:00:05.261501 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:05.261474    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr"
Jul 17 22:00:05.261501 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:05.261497    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr" event={"ID":"aed251fc-0c05-4574-9c15-e699836c3f83","Type":"ContainerDied","Data":"9c19827030abf097d92245c6cdeded2c0dd12ab66bfdf2162ee216928a803951"}
Jul 17 22:00:05.262036 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:05.261556    2573 scope.go:117] "RemoveContainer" containerID="8f733849b37e28c6fa24dd6e0ab8f87e7dc76d9ede1ee8d8486cd00661090989"
Jul 17 22:00:05.278125 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:05.278104    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr"]
Jul 17 22:00:05.281661 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:05.281639    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bqwgr"]
Jul 17 22:00:06.525178 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:06.525147    2573 scope.go:117] "RemoveContainer" containerID="76f482407c18869e710c5b2e4b6f46438b5be785cdf1b7ec1596b641d208c344"
Jul 17 22:00:06.530379 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:06.530356    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aed251fc-0c05-4574-9c15-e699836c3f83" path="/var/lib/kubelet/pods/aed251fc-0c05-4574-9c15-e699836c3f83/volumes"
Jul 17 22:00:07.272561 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:07.272531    2573 generic.go:356] "Generic (PLEG): container finished" podID="bc3cd75a-18d8-4feb-bd5a-696dcae50e26" containerID="63f4bfe173b3da810821744bf9d09b6c36fabfe3b680448c58b0c637d4b9737b" exitCode=6
Jul 17 22:00:07.272713 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:07.272606    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx" event={"ID":"bc3cd75a-18d8-4feb-bd5a-696dcae50e26","Type":"ContainerDied","Data":"63f4bfe173b3da810821744bf9d09b6c36fabfe3b680448c58b0c637d4b9737b"}
Jul 17 22:00:07.272713 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:07.272663    2573 scope.go:117] "RemoveContainer" containerID="76f482407c18869e710c5b2e4b6f46438b5be785cdf1b7ec1596b641d208c344"
Jul 17 22:00:07.272995 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:07.272979    2573 scope.go:117] "RemoveContainer" containerID="63f4bfe173b3da810821744bf9d09b6c36fabfe3b680448c58b0c637d4b9737b"
Jul 17 22:00:07.273213 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:00:07.273193    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx_odh-ai-gateway-infra(bc3cd75a-18d8-4feb-bd5a-696dcae50e26)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx" podUID="bc3cd75a-18d8-4feb-bd5a-696dcae50e26"
Jul 17 22:00:08.299541 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:08.299503    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx"]
Jul 17 22:00:08.428246 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:08.428218    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx"
Jul 17 22:00:08.495327 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:08.495277    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3cd75a-18d8-4feb-bd5a-696dcae50e26-maas-api-service-ca\") pod \"bc3cd75a-18d8-4feb-bd5a-696dcae50e26\" (UID: \"bc3cd75a-18d8-4feb-bd5a-696dcae50e26\") "
Jul 17 22:00:08.495631 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:08.495607    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3cd75a-18d8-4feb-bd5a-696dcae50e26-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc3cd75a-18d8-4feb-bd5a-696dcae50e26" (UID: "bc3cd75a-18d8-4feb-bd5a-696dcae50e26"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:00:08.596581 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:08.596555    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc3cd75a-18d8-4feb-bd5a-696dcae50e26-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:00:09.284537 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:09.284507    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx"
Jul 17 22:00:09.284710 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:09.284542    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx" event={"ID":"bc3cd75a-18d8-4feb-bd5a-696dcae50e26","Type":"ContainerDied","Data":"691a90dcf38ade560a2fb9ee78c791adce98be5dc89370385d2d3c469760324d"}
Jul 17 22:00:09.284710 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:09.284600    2573 scope.go:117] "RemoveContainer" containerID="63f4bfe173b3da810821744bf9d09b6c36fabfe3b680448c58b0c637d4b9737b"
Jul 17 22:00:09.299470 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:09.299449    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx"]
Jul 17 22:00:09.302932 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:09.302913    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-pmcwx"]
Jul 17 22:00:10.529955 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.529918    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc3cd75a-18d8-4feb-bd5a-696dcae50e26" path="/var/lib/kubelet/pods/bc3cd75a-18d8-4feb-bd5a-696dcae50e26/volumes"
Jul 17 22:00:10.958342 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958313    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55"]
Jul 17 22:00:10.958593 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958574    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aed251fc-0c05-4574-9c15-e699836c3f83" containerName="revoke-keys"
Jul 17 22:00:10.958633 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958595    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aed251fc-0c05-4574-9c15-e699836c3f83" containerName="revoke-keys"
Jul 17 22:00:10.958633 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958607    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31fa2354-db49-41d0-8b4f-bb5f713faa84" containerName="cleanup"
Jul 17 22:00:10.958633 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958613    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="31fa2354-db49-41d0-8b4f-bb5f713faa84" containerName="cleanup"
Jul 17 22:00:10.958633 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958622    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aed251fc-0c05-4574-9c15-e699836c3f83" containerName="revoke-keys"
Jul 17 22:00:10.958633 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958628    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aed251fc-0c05-4574-9c15-e699836c3f83" containerName="revoke-keys"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958635    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc3cd75a-18d8-4feb-bd5a-696dcae50e26" containerName="revoke-keys"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958641    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc3cd75a-18d8-4feb-bd5a-696dcae50e26" containerName="revoke-keys"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958650    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aed251fc-0c05-4574-9c15-e699836c3f83" containerName="revoke-keys"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958655    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aed251fc-0c05-4574-9c15-e699836c3f83" containerName="revoke-keys"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958663    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc3cd75a-18d8-4feb-bd5a-696dcae50e26" containerName="revoke-keys"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958669    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc3cd75a-18d8-4feb-bd5a-696dcae50e26" containerName="revoke-keys"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958675    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c38b12d6-9aed-4f15-b927-a25b17914f27" containerName="cleanup"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958681    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c38b12d6-9aed-4f15-b927-a25b17914f27" containerName="cleanup"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958690    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc3cd75a-18d8-4feb-bd5a-696dcae50e26" containerName="revoke-keys"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958695    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc3cd75a-18d8-4feb-bd5a-696dcae50e26" containerName="revoke-keys"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958701    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3801dad-7ae9-4cf5-a24d-e60376ff1490" containerName="cleanup"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958706    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3801dad-7ae9-4cf5-a24d-e60376ff1490" containerName="cleanup"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958748    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc3cd75a-18d8-4feb-bd5a-696dcae50e26" containerName="revoke-keys"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958757    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3801dad-7ae9-4cf5-a24d-e60376ff1490" containerName="cleanup"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958765    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aed251fc-0c05-4574-9c15-e699836c3f83" containerName="revoke-keys"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958770    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc3cd75a-18d8-4feb-bd5a-696dcae50e26" containerName="revoke-keys"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958775    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="31fa2354-db49-41d0-8b4f-bb5f713faa84" containerName="cleanup"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958782    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c38b12d6-9aed-4f15-b927-a25b17914f27" containerName="cleanup"
Jul 17 22:00:10.958797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.958787    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aed251fc-0c05-4574-9c15-e699836c3f83" containerName="revoke-keys"
Jul 17 22:00:10.963139 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.963120    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55"
Jul 17 22:00:10.965738 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.965718    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 22:00:10.965958 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.965945    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:00:10.969459 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:10.969438    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55"]
Jul 17 22:00:11.011770 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:11.011736    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8835231-810c-49bd-889e-e46b72e3f915-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-jbk55\" (UID: \"c8835231-810c-49bd-889e-e46b72e3f915\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55"
Jul 17 22:00:11.113010 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:11.112974    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8835231-810c-49bd-889e-e46b72e3f915-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-jbk55\" (UID: \"c8835231-810c-49bd-889e-e46b72e3f915\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55"
Jul 17 22:00:11.113602 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:11.113584    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8835231-810c-49bd-889e-e46b72e3f915-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-jbk55\" (UID: \"c8835231-810c-49bd-889e-e46b72e3f915\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55"
Jul 17 22:00:11.274151 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:11.274080    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55"
Jul 17 22:00:11.396611 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:11.396582    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55"]
Jul 17 22:00:11.398836 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:00:11.398810    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8835231_810c_49bd_889e_e46b72e3f915.slice/crio-8ed5fd51825e0e61350daf0a50fc23b9982e2945f46cb2714418cb83a0dc5063 WatchSource:0}: Error finding container 8ed5fd51825e0e61350daf0a50fc23b9982e2945f46cb2714418cb83a0dc5063: Status 404 returned error can't find the container with id 8ed5fd51825e0e61350daf0a50fc23b9982e2945f46cb2714418cb83a0dc5063
Jul 17 22:00:12.302400 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:12.302318    2573 generic.go:356] "Generic (PLEG): container finished" podID="c8835231-810c-49bd-889e-e46b72e3f915" containerID="987970dc74150721deeb13befcf8d3594cde34cf3f7f15eac0f5f8b28acb68b3" exitCode=6
Jul 17 22:00:12.302400 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:12.302371    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55" event={"ID":"c8835231-810c-49bd-889e-e46b72e3f915","Type":"ContainerDied","Data":"987970dc74150721deeb13befcf8d3594cde34cf3f7f15eac0f5f8b28acb68b3"}
Jul 17 22:00:12.302790 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:12.302411    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55" event={"ID":"c8835231-810c-49bd-889e-e46b72e3f915","Type":"ContainerStarted","Data":"8ed5fd51825e0e61350daf0a50fc23b9982e2945f46cb2714418cb83a0dc5063"}
Jul 17 22:00:12.302790 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:12.302608    2573 scope.go:117] "RemoveContainer" containerID="987970dc74150721deeb13befcf8d3594cde34cf3f7f15eac0f5f8b28acb68b3"
Jul 17 22:00:13.308892 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:13.308799    2573 generic.go:356] "Generic (PLEG): container finished" podID="c8835231-810c-49bd-889e-e46b72e3f915" containerID="ddd5a7e0734c193a2a651682203a0b029d7f2ed4e283440fb04deab330ef0580" exitCode=6
Jul 17 22:00:13.309309 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:13.308885    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55" event={"ID":"c8835231-810c-49bd-889e-e46b72e3f915","Type":"ContainerDied","Data":"ddd5a7e0734c193a2a651682203a0b029d7f2ed4e283440fb04deab330ef0580"}
Jul 17 22:00:13.309309 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:13.308940    2573 scope.go:117] "RemoveContainer" containerID="987970dc74150721deeb13befcf8d3594cde34cf3f7f15eac0f5f8b28acb68b3"
Jul 17 22:00:13.309309 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:13.309044    2573 scope.go:117] "RemoveContainer" containerID="ddd5a7e0734c193a2a651682203a0b029d7f2ed4e283440fb04deab330ef0580"
Jul 17 22:00:13.309309 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:00:13.309244    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-jbk55_odh-ai-gateway-infra(c8835231-810c-49bd-889e-e46b72e3f915)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55" podUID="c8835231-810c-49bd-889e-e46b72e3f915"
Jul 17 22:00:14.315010 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:14.314980    2573 scope.go:117] "RemoveContainer" containerID="ddd5a7e0734c193a2a651682203a0b029d7f2ed4e283440fb04deab330ef0580"
Jul 17 22:00:14.315436 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:00:14.315151    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-jbk55_odh-ai-gateway-infra(c8835231-810c-49bd-889e-e46b72e3f915)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55" podUID="c8835231-810c-49bd-889e-e46b72e3f915"
Jul 17 22:00:27.905606 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:27.905569    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm"]
Jul 17 22:00:27.905974 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:27.905892    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc3cd75a-18d8-4feb-bd5a-696dcae50e26" containerName="revoke-keys"
Jul 17 22:00:27.905974 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:27.905905    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aed251fc-0c05-4574-9c15-e699836c3f83" containerName="revoke-keys"
Jul 17 22:00:27.910026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:27.910010    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm"
Jul 17 22:00:27.917213 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:27.917162    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm"]
Jul 17 22:00:28.048073 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:28.048045    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddaba242-98ed-49e2-b44f-2c09173bd013-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm\" (UID: \"ddaba242-98ed-49e2-b44f-2c09173bd013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm"
Jul 17 22:00:28.149349 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:28.149326    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddaba242-98ed-49e2-b44f-2c09173bd013-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm\" (UID: \"ddaba242-98ed-49e2-b44f-2c09173bd013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm"
Jul 17 22:00:28.149893 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:28.149875    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddaba242-98ed-49e2-b44f-2c09173bd013-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm\" (UID: \"ddaba242-98ed-49e2-b44f-2c09173bd013\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm"
Jul 17 22:00:28.221022 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:28.220964    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm"
Jul 17 22:00:28.350336 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:28.350304    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm"]
Jul 17 22:00:28.350964 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:00:28.350942    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddaba242_98ed_49e2_b44f_2c09173bd013.slice/crio-d8dd828b235cc7157ebd676af43bf7d96113dfd64f4380e548fd797ce8d3fab0 WatchSource:0}: Error finding container d8dd828b235cc7157ebd676af43bf7d96113dfd64f4380e548fd797ce8d3fab0: Status 404 returned error can't find the container with id d8dd828b235cc7157ebd676af43bf7d96113dfd64f4380e548fd797ce8d3fab0
Jul 17 22:00:28.373936 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:28.373906    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm" event={"ID":"ddaba242-98ed-49e2-b44f-2c09173bd013","Type":"ContainerStarted","Data":"d8dd828b235cc7157ebd676af43bf7d96113dfd64f4380e548fd797ce8d3fab0"}
Jul 17 22:00:28.530629 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:28.530563    2573 scope.go:117] "RemoveContainer" containerID="ddd5a7e0734c193a2a651682203a0b029d7f2ed4e283440fb04deab330ef0580"
Jul 17 22:00:29.379177 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:29.379147    2573 generic.go:356] "Generic (PLEG): container finished" podID="ddaba242-98ed-49e2-b44f-2c09173bd013" containerID="c00938ff414c24dbed1d7341ef8bacb00fcccfeda4f12eba5e4968362d2b3393" exitCode=6
Jul 17 22:00:29.379552 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:29.379206    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm" event={"ID":"ddaba242-98ed-49e2-b44f-2c09173bd013","Type":"ContainerDied","Data":"c00938ff414c24dbed1d7341ef8bacb00fcccfeda4f12eba5e4968362d2b3393"}
Jul 17 22:00:29.379552 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:29.379443    2573 scope.go:117] "RemoveContainer" containerID="c00938ff414c24dbed1d7341ef8bacb00fcccfeda4f12eba5e4968362d2b3393"
Jul 17 22:00:29.381163 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:29.381144    2573 generic.go:356] "Generic (PLEG): container finished" podID="c8835231-810c-49bd-889e-e46b72e3f915" containerID="84c189c3027c2bd4987b28962496a14706424c79c5b9ec7b7050972e2b3e49ba" exitCode=6
Jul 17 22:00:29.381262 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:29.381201    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55" event={"ID":"c8835231-810c-49bd-889e-e46b72e3f915","Type":"ContainerDied","Data":"84c189c3027c2bd4987b28962496a14706424c79c5b9ec7b7050972e2b3e49ba"}
Jul 17 22:00:29.381262 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:29.381244    2573 scope.go:117] "RemoveContainer" containerID="ddd5a7e0734c193a2a651682203a0b029d7f2ed4e283440fb04deab330ef0580"
Jul 17 22:00:29.381536 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:29.381522    2573 scope.go:117] "RemoveContainer" containerID="84c189c3027c2bd4987b28962496a14706424c79c5b9ec7b7050972e2b3e49ba"
Jul 17 22:00:29.381722 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:00:29.381689    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-jbk55_odh-ai-gateway-infra(c8835231-810c-49bd-889e-e46b72e3f915)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55" podUID="c8835231-810c-49bd-889e-e46b72e3f915"
Jul 17 22:00:30.389203 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:30.389170    2573 generic.go:356] "Generic (PLEG): container finished" podID="ddaba242-98ed-49e2-b44f-2c09173bd013" containerID="a16d0470a0b914877bfe94ab33a2a4045ede431b8bce7845bf55b59bd94ce675" exitCode=6
Jul 17 22:00:30.389648 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:30.389208    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm" event={"ID":"ddaba242-98ed-49e2-b44f-2c09173bd013","Type":"ContainerDied","Data":"a16d0470a0b914877bfe94ab33a2a4045ede431b8bce7845bf55b59bd94ce675"}
Jul 17 22:00:30.389648 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:30.389262    2573 scope.go:117] "RemoveContainer" containerID="c00938ff414c24dbed1d7341ef8bacb00fcccfeda4f12eba5e4968362d2b3393"
Jul 17 22:00:30.389648 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:30.389472    2573 scope.go:117] "RemoveContainer" containerID="a16d0470a0b914877bfe94ab33a2a4045ede431b8bce7845bf55b59bd94ce675"
Jul 17 22:00:30.389807 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:00:30.389673    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm_odh-ai-gateway-infra(ddaba242-98ed-49e2-b44f-2c09173bd013)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm" podUID="ddaba242-98ed-49e2-b44f-2c09173bd013"
Jul 17 22:00:30.443549 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:30.443516    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55"]
Jul 17 22:00:30.571922 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:30.571899    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55"
Jul 17 22:00:30.668662 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:30.668593    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8835231-810c-49bd-889e-e46b72e3f915-maas-api-service-ca\") pod \"c8835231-810c-49bd-889e-e46b72e3f915\" (UID: \"c8835231-810c-49bd-889e-e46b72e3f915\") "
Jul 17 22:00:30.673996 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:30.669205    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8835231-810c-49bd-889e-e46b72e3f915-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c8835231-810c-49bd-889e-e46b72e3f915" (UID: "c8835231-810c-49bd-889e-e46b72e3f915"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:00:30.769397 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:30.769374    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8835231-810c-49bd-889e-e46b72e3f915-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:00:31.394273 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:31.394243    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55"
Jul 17 22:00:31.394779 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:31.394276    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55" event={"ID":"c8835231-810c-49bd-889e-e46b72e3f915","Type":"ContainerDied","Data":"8ed5fd51825e0e61350daf0a50fc23b9982e2945f46cb2714418cb83a0dc5063"}
Jul 17 22:00:31.394779 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:31.394396    2573 scope.go:117] "RemoveContainer" containerID="84c189c3027c2bd4987b28962496a14706424c79c5b9ec7b7050972e2b3e49ba"
Jul 17 22:00:31.396118 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:31.396102    2573 scope.go:117] "RemoveContainer" containerID="a16d0470a0b914877bfe94ab33a2a4045ede431b8bce7845bf55b59bd94ce675"
Jul 17 22:00:31.396277 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:00:31.396259    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm_odh-ai-gateway-infra(ddaba242-98ed-49e2-b44f-2c09173bd013)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm" podUID="ddaba242-98ed-49e2-b44f-2c09173bd013"
Jul 17 22:00:31.426373 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:31.426348    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55"]
Jul 17 22:00:31.428724 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:31.428698    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jbk55"]
Jul 17 22:00:32.529461 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:32.529430    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8835231-810c-49bd-889e-e46b72e3f915" path="/var/lib/kubelet/pods/c8835231-810c-49bd-889e-e46b72e3f915/volumes"
Jul 17 22:00:35.669082 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:35.669048    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q"]
Jul 17 22:00:35.669581 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:35.669335    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8835231-810c-49bd-889e-e46b72e3f915" containerName="revoke-keys"
Jul 17 22:00:35.669581 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:35.669347    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8835231-810c-49bd-889e-e46b72e3f915" containerName="revoke-keys"
Jul 17 22:00:35.669581 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:35.669362    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8835231-810c-49bd-889e-e46b72e3f915" containerName="revoke-keys"
Jul 17 22:00:35.669581 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:35.669368    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8835231-810c-49bd-889e-e46b72e3f915" containerName="revoke-keys"
Jul 17 22:00:35.669581 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:35.669416    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8835231-810c-49bd-889e-e46b72e3f915" containerName="revoke-keys"
Jul 17 22:00:35.669581 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:35.669425    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8835231-810c-49bd-889e-e46b72e3f915" containerName="revoke-keys"
Jul 17 22:00:35.669581 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:35.669431    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8835231-810c-49bd-889e-e46b72e3f915" containerName="revoke-keys"
Jul 17 22:00:35.673550 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:35.673534    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q"
Jul 17 22:00:35.678877 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:35.678853    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q"]
Jul 17 22:00:35.815152 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:35.815125    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24889d61-6b7f-4804-9975-ed369fb14af1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q\" (UID: \"24889d61-6b7f-4804-9975-ed369fb14af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q"
Jul 17 22:00:35.915760 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:35.915731    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24889d61-6b7f-4804-9975-ed369fb14af1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q\" (UID: \"24889d61-6b7f-4804-9975-ed369fb14af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q"
Jul 17 22:00:35.916353 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:35.916337    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24889d61-6b7f-4804-9975-ed369fb14af1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q\" (UID: \"24889d61-6b7f-4804-9975-ed369fb14af1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q"
Jul 17 22:00:35.984375 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:35.984316    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q"
Jul 17 22:00:36.113273 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:36.113251    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q"]
Jul 17 22:00:36.115572 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:00:36.115544    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24889d61_6b7f_4804_9975_ed369fb14af1.slice/crio-bd8bcdffd69a92746a5cc06e7e2db4bba15f65304b7be2fbadf20b0791783c49 WatchSource:0}: Error finding container bd8bcdffd69a92746a5cc06e7e2db4bba15f65304b7be2fbadf20b0791783c49: Status 404 returned error can't find the container with id bd8bcdffd69a92746a5cc06e7e2db4bba15f65304b7be2fbadf20b0791783c49
Jul 17 22:00:36.424152 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:36.424117    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q" event={"ID":"24889d61-6b7f-4804-9975-ed369fb14af1","Type":"ContainerStarted","Data":"bd8bcdffd69a92746a5cc06e7e2db4bba15f65304b7be2fbadf20b0791783c49"}
Jul 17 22:00:37.430231 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:37.430193    2573 generic.go:356] "Generic (PLEG): container finished" podID="24889d61-6b7f-4804-9975-ed369fb14af1" containerID="8ea877d2f32e9e1399d548bdd30f588aadbbe8c7d4d133238ecc68c3a9b1e758" exitCode=6
Jul 17 22:00:37.430651 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:37.430254    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q" event={"ID":"24889d61-6b7f-4804-9975-ed369fb14af1","Type":"ContainerDied","Data":"8ea877d2f32e9e1399d548bdd30f588aadbbe8c7d4d133238ecc68c3a9b1e758"}
Jul 17 22:00:37.430651 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:37.430484    2573 scope.go:117] "RemoveContainer" containerID="8ea877d2f32e9e1399d548bdd30f588aadbbe8c7d4d133238ecc68c3a9b1e758"
Jul 17 22:00:38.437022 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:38.436989    2573 generic.go:356] "Generic (PLEG): container finished" podID="24889d61-6b7f-4804-9975-ed369fb14af1" containerID="e7cbb0758b76519f070d3613a25d1049966058cf5cd3c4ff395c4be853285add" exitCode=6
Jul 17 22:00:38.437430 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:38.437083    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q" event={"ID":"24889d61-6b7f-4804-9975-ed369fb14af1","Type":"ContainerDied","Data":"e7cbb0758b76519f070d3613a25d1049966058cf5cd3c4ff395c4be853285add"}
Jul 17 22:00:38.437430 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:38.437139    2573 scope.go:117] "RemoveContainer" containerID="8ea877d2f32e9e1399d548bdd30f588aadbbe8c7d4d133238ecc68c3a9b1e758"
Jul 17 22:00:38.437430 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:38.437248    2573 scope.go:117] "RemoveContainer" containerID="e7cbb0758b76519f070d3613a25d1049966058cf5cd3c4ff395c4be853285add"
Jul 17 22:00:38.437566 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:00:38.437477    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q_odh-ai-gateway-infra(24889d61-6b7f-4804-9975-ed369fb14af1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q" podUID="24889d61-6b7f-4804-9975-ed369fb14af1"
Jul 17 22:00:39.442509 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:39.442482    2573 scope.go:117] "RemoveContainer" containerID="e7cbb0758b76519f070d3613a25d1049966058cf5cd3c4ff395c4be853285add"
Jul 17 22:00:39.442900 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:00:39.442672    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q_odh-ai-gateway-infra(24889d61-6b7f-4804-9975-ed369fb14af1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q" podUID="24889d61-6b7f-4804-9975-ed369fb14af1"
Jul 17 22:00:39.821203 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:39.821118    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz"]
Jul 17 22:00:39.821521 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:39.821504    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8835231-810c-49bd-889e-e46b72e3f915" containerName="revoke-keys"
Jul 17 22:00:39.821618 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:39.821524    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8835231-810c-49bd-889e-e46b72e3f915" containerName="revoke-keys"
Jul 17 22:00:39.825697 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:39.825674    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz"
Jul 17 22:00:39.831716 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:39.831561    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz"]
Jul 17 22:00:39.847578 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:39.847553    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3665735c-f895-4d5d-a2b5-e2a82e6398e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz\" (UID: \"3665735c-f895-4d5d-a2b5-e2a82e6398e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz"
Jul 17 22:00:39.948569 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:39.948541    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3665735c-f895-4d5d-a2b5-e2a82e6398e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz\" (UID: \"3665735c-f895-4d5d-a2b5-e2a82e6398e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz"
Jul 17 22:00:39.949129 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:39.949109    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3665735c-f895-4d5d-a2b5-e2a82e6398e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz\" (UID: \"3665735c-f895-4d5d-a2b5-e2a82e6398e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz"
Jul 17 22:00:40.137141 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:40.137111    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz"
Jul 17 22:00:40.266154 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:40.266128    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz"]
Jul 17 22:00:40.268375 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:00:40.268345    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3665735c_f895_4d5d_a2b5_e2a82e6398e8.slice/crio-ff745f599f9f36e032f94729bcbce2b88209453e15ac01277814122afd42ebdb WatchSource:0}: Error finding container ff745f599f9f36e032f94729bcbce2b88209453e15ac01277814122afd42ebdb: Status 404 returned error can't find the container with id ff745f599f9f36e032f94729bcbce2b88209453e15ac01277814122afd42ebdb
Jul 17 22:00:40.447592 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:40.447561    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz" event={"ID":"3665735c-f895-4d5d-a2b5-e2a82e6398e8","Type":"ContainerStarted","Data":"ff745f599f9f36e032f94729bcbce2b88209453e15ac01277814122afd42ebdb"}
Jul 17 22:00:41.453629 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:41.453596    2573 generic.go:356] "Generic (PLEG): container finished" podID="3665735c-f895-4d5d-a2b5-e2a82e6398e8" containerID="f37c364fd24bcaef2d1c1e39a6bdf70c3c1bd0b428aaf0b6ab52da52a6c7e6d3" exitCode=6
Jul 17 22:00:41.453994 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:41.453644    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz" event={"ID":"3665735c-f895-4d5d-a2b5-e2a82e6398e8","Type":"ContainerDied","Data":"f37c364fd24bcaef2d1c1e39a6bdf70c3c1bd0b428aaf0b6ab52da52a6c7e6d3"}
Jul 17 22:00:41.453994 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:41.453845    2573 scope.go:117] "RemoveContainer" containerID="f37c364fd24bcaef2d1c1e39a6bdf70c3c1bd0b428aaf0b6ab52da52a6c7e6d3"
Jul 17 22:00:42.460061 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:42.460029    2573 generic.go:356] "Generic (PLEG): container finished" podID="3665735c-f895-4d5d-a2b5-e2a82e6398e8" containerID="26fe3f2a90cfb18370ddd7fdeaf3c539cf6283dfd3890c6b8bf8e1a87c9fd04d" exitCode=6
Jul 17 22:00:42.460533 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:42.460103    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz" event={"ID":"3665735c-f895-4d5d-a2b5-e2a82e6398e8","Type":"ContainerDied","Data":"26fe3f2a90cfb18370ddd7fdeaf3c539cf6283dfd3890c6b8bf8e1a87c9fd04d"}
Jul 17 22:00:42.460533 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:42.460180    2573 scope.go:117] "RemoveContainer" containerID="f37c364fd24bcaef2d1c1e39a6bdf70c3c1bd0b428aaf0b6ab52da52a6c7e6d3"
Jul 17 22:00:42.460533 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:42.460374    2573 scope.go:117] "RemoveContainer" containerID="26fe3f2a90cfb18370ddd7fdeaf3c539cf6283dfd3890c6b8bf8e1a87c9fd04d"
Jul 17 22:00:42.460700 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:00:42.460561    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz_odh-ai-gateway-infra(3665735c-f895-4d5d-a2b5-e2a82e6398e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz" podUID="3665735c-f895-4d5d-a2b5-e2a82e6398e8"
Jul 17 22:00:43.466796 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:43.466771    2573 scope.go:117] "RemoveContainer" containerID="26fe3f2a90cfb18370ddd7fdeaf3c539cf6283dfd3890c6b8bf8e1a87c9fd04d"
Jul 17 22:00:43.467247 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:00:43.466969    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz_odh-ai-gateway-infra(3665735c-f895-4d5d-a2b5-e2a82e6398e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz" podUID="3665735c-f895-4d5d-a2b5-e2a82e6398e8"
Jul 17 22:00:45.525143 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:45.525114    2573 scope.go:117] "RemoveContainer" containerID="a16d0470a0b914877bfe94ab33a2a4045ede431b8bce7845bf55b59bd94ce675"
Jul 17 22:00:46.481562 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:46.481473    2573 generic.go:356] "Generic (PLEG): container finished" podID="ddaba242-98ed-49e2-b44f-2c09173bd013" containerID="a59a855750bba3a49e3f812251904ccc989dca07c80fe838348a137b839aa43a" exitCode=6
Jul 17 22:00:46.481732 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:46.481551    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm" event={"ID":"ddaba242-98ed-49e2-b44f-2c09173bd013","Type":"ContainerDied","Data":"a59a855750bba3a49e3f812251904ccc989dca07c80fe838348a137b839aa43a"}
Jul 17 22:00:46.481732 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:46.481629    2573 scope.go:117] "RemoveContainer" containerID="a16d0470a0b914877bfe94ab33a2a4045ede431b8bce7845bf55b59bd94ce675"
Jul 17 22:00:46.481969 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:46.481952    2573 scope.go:117] "RemoveContainer" containerID="a59a855750bba3a49e3f812251904ccc989dca07c80fe838348a137b839aa43a"
Jul 17 22:00:46.482155 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:00:46.482136    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm_odh-ai-gateway-infra(ddaba242-98ed-49e2-b44f-2c09173bd013)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm" podUID="ddaba242-98ed-49e2-b44f-2c09173bd013"
Jul 17 22:00:47.506342 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:47.506308    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm"]
Jul 17 22:00:47.640275 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:47.640248    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm"
Jul 17 22:00:47.706787 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:47.706764    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddaba242-98ed-49e2-b44f-2c09173bd013-maas-api-service-ca\") pod \"ddaba242-98ed-49e2-b44f-2c09173bd013\" (UID: \"ddaba242-98ed-49e2-b44f-2c09173bd013\") "
Jul 17 22:00:47.707106 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:47.707082    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddaba242-98ed-49e2-b44f-2c09173bd013-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ddaba242-98ed-49e2-b44f-2c09173bd013" (UID: "ddaba242-98ed-49e2-b44f-2c09173bd013"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:00:47.807899 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:47.807836    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddaba242-98ed-49e2-b44f-2c09173bd013-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:00:48.493370 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:48.493329    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm" event={"ID":"ddaba242-98ed-49e2-b44f-2c09173bd013","Type":"ContainerDied","Data":"d8dd828b235cc7157ebd676af43bf7d96113dfd64f4380e548fd797ce8d3fab0"}
Jul 17 22:00:48.493546 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:48.493416    2573 scope.go:117] "RemoveContainer" containerID="a59a855750bba3a49e3f812251904ccc989dca07c80fe838348a137b839aa43a"
Jul 17 22:00:48.493546 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:48.493345    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm"
Jul 17 22:00:48.515437 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:48.515407    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm"]
Jul 17 22:00:48.517619 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:48.517595    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dq2sm"]
Jul 17 22:00:48.530922 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:48.530898    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddaba242-98ed-49e2-b44f-2c09173bd013" path="/var/lib/kubelet/pods/ddaba242-98ed-49e2-b44f-2c09173bd013/volumes"
Jul 17 22:00:51.525128 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:51.525101    2573 scope.go:117] "RemoveContainer" containerID="e7cbb0758b76519f070d3613a25d1049966058cf5cd3c4ff395c4be853285add"
Jul 17 22:00:52.513679 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:52.513598    2573 generic.go:356] "Generic (PLEG): container finished" podID="24889d61-6b7f-4804-9975-ed369fb14af1" containerID="c33100f6c90c129ce400921ff2ad7359785d01a8d50b422d477e94f305b5e672" exitCode=6
Jul 17 22:00:52.513825 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:52.513672    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q" event={"ID":"24889d61-6b7f-4804-9975-ed369fb14af1","Type":"ContainerDied","Data":"c33100f6c90c129ce400921ff2ad7359785d01a8d50b422d477e94f305b5e672"}
Jul 17 22:00:52.513825 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:52.513734    2573 scope.go:117] "RemoveContainer" containerID="e7cbb0758b76519f070d3613a25d1049966058cf5cd3c4ff395c4be853285add"
Jul 17 22:00:52.514066 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:52.514045    2573 scope.go:117] "RemoveContainer" containerID="c33100f6c90c129ce400921ff2ad7359785d01a8d50b422d477e94f305b5e672"
Jul 17 22:00:52.514247 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:00:52.514227    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q_odh-ai-gateway-infra(24889d61-6b7f-4804-9975-ed369fb14af1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q" podUID="24889d61-6b7f-4804-9975-ed369fb14af1"
Jul 17 22:00:53.539023 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:53.538988    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q"]
Jul 17 22:00:53.667975 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:53.667953    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q"
Jul 17 22:00:53.752881 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:53.752854    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24889d61-6b7f-4804-9975-ed369fb14af1-maas-api-service-ca\") pod \"24889d61-6b7f-4804-9975-ed369fb14af1\" (UID: \"24889d61-6b7f-4804-9975-ed369fb14af1\") "
Jul 17 22:00:53.753179 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:53.753159    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24889d61-6b7f-4804-9975-ed369fb14af1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "24889d61-6b7f-4804-9975-ed369fb14af1" (UID: "24889d61-6b7f-4804-9975-ed369fb14af1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:00:53.853882 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:53.853862    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24889d61-6b7f-4804-9975-ed369fb14af1-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:00:54.524660 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:54.524627    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q"
Jul 17 22:00:54.529020 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:54.528993    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q" event={"ID":"24889d61-6b7f-4804-9975-ed369fb14af1","Type":"ContainerDied","Data":"bd8bcdffd69a92746a5cc06e7e2db4bba15f65304b7be2fbadf20b0791783c49"}
Jul 17 22:00:54.529165 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:54.529038    2573 scope.go:117] "RemoveContainer" containerID="c33100f6c90c129ce400921ff2ad7359785d01a8d50b422d477e94f305b5e672"
Jul 17 22:00:54.551181 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:54.551160    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q"]
Jul 17 22:00:54.559105 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:54.559080    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mn52q"]
Jul 17 22:00:55.524852 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:55.524817    2573 scope.go:117] "RemoveContainer" containerID="26fe3f2a90cfb18370ddd7fdeaf3c539cf6283dfd3890c6b8bf8e1a87c9fd04d"
Jul 17 22:00:56.529691 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:56.529618    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24889d61-6b7f-4804-9975-ed369fb14af1" path="/var/lib/kubelet/pods/24889d61-6b7f-4804-9975-ed369fb14af1/volumes"
Jul 17 22:00:56.536997 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:56.536968    2573 generic.go:356] "Generic (PLEG): container finished" podID="3665735c-f895-4d5d-a2b5-e2a82e6398e8" containerID="25d629e175535b3d36e02034b9bfd7a62a673f019ebfbcd89fa3678eb7b34224" exitCode=6
Jul 17 22:00:56.537151 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:56.537035    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz" event={"ID":"3665735c-f895-4d5d-a2b5-e2a82e6398e8","Type":"ContainerDied","Data":"25d629e175535b3d36e02034b9bfd7a62a673f019ebfbcd89fa3678eb7b34224"}
Jul 17 22:00:56.537151 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:56.537106    2573 scope.go:117] "RemoveContainer" containerID="26fe3f2a90cfb18370ddd7fdeaf3c539cf6283dfd3890c6b8bf8e1a87c9fd04d"
Jul 17 22:00:56.537448 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:56.537435    2573 scope.go:117] "RemoveContainer" containerID="25d629e175535b3d36e02034b9bfd7a62a673f019ebfbcd89fa3678eb7b34224"
Jul 17 22:00:56.537637 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:00:56.537616    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz_odh-ai-gateway-infra(3665735c-f895-4d5d-a2b5-e2a82e6398e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz" podUID="3665735c-f895-4d5d-a2b5-e2a82e6398e8"
Jul 17 22:00:57.561898 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:57.561863    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz"]
Jul 17 22:00:57.690758 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:57.690734    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz"
Jul 17 22:00:57.785762 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:57.785731    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3665735c-f895-4d5d-a2b5-e2a82e6398e8-maas-api-service-ca\") pod \"3665735c-f895-4d5d-a2b5-e2a82e6398e8\" (UID: \"3665735c-f895-4d5d-a2b5-e2a82e6398e8\") "
Jul 17 22:00:57.786076 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:57.786058    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3665735c-f895-4d5d-a2b5-e2a82e6398e8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3665735c-f895-4d5d-a2b5-e2a82e6398e8" (UID: "3665735c-f895-4d5d-a2b5-e2a82e6398e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:00:57.886475 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:57.886441    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3665735c-f895-4d5d-a2b5-e2a82e6398e8-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:00:58.549114 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:58.549089    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz"
Jul 17 22:00:58.549114 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:58.549105    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz" event={"ID":"3665735c-f895-4d5d-a2b5-e2a82e6398e8","Type":"ContainerDied","Data":"ff745f599f9f36e032f94729bcbce2b88209453e15ac01277814122afd42ebdb"}
Jul 17 22:00:58.549340 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:58.549156    2573 scope.go:117] "RemoveContainer" containerID="25d629e175535b3d36e02034b9bfd7a62a673f019ebfbcd89fa3678eb7b34224"
Jul 17 22:00:58.567531 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:58.567505    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz"]
Jul 17 22:00:58.578387 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:00:58.574553    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-n2xgz"]
Jul 17 22:01:00.530034 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:00.530006    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3665735c-f895-4d5d-a2b5-e2a82e6398e8" path="/var/lib/kubelet/pods/3665735c-f895-4d5d-a2b5-e2a82e6398e8/volumes"
Jul 17 22:01:05.055022 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.054987    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kfpwg"]
Jul 17 22:01:05.055415 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055244    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddaba242-98ed-49e2-b44f-2c09173bd013" containerName="revoke-keys"
Jul 17 22:01:05.055415 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055254    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddaba242-98ed-49e2-b44f-2c09173bd013" containerName="revoke-keys"
Jul 17 22:01:05.055415 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055263    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24889d61-6b7f-4804-9975-ed369fb14af1" containerName="revoke-keys"
Jul 17 22:01:05.055415 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055270    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="24889d61-6b7f-4804-9975-ed369fb14af1" containerName="revoke-keys"
Jul 17 22:01:05.055415 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055277    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3665735c-f895-4d5d-a2b5-e2a82e6398e8" containerName="revoke-keys"
Jul 17 22:01:05.055415 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055295    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3665735c-f895-4d5d-a2b5-e2a82e6398e8" containerName="revoke-keys"
Jul 17 22:01:05.055415 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055310    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3665735c-f895-4d5d-a2b5-e2a82e6398e8" containerName="revoke-keys"
Jul 17 22:01:05.055415 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055319    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3665735c-f895-4d5d-a2b5-e2a82e6398e8" containerName="revoke-keys"
Jul 17 22:01:05.055415 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055332    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddaba242-98ed-49e2-b44f-2c09173bd013" containerName="revoke-keys"
Jul 17 22:01:05.055415 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055337    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddaba242-98ed-49e2-b44f-2c09173bd013" containerName="revoke-keys"
Jul 17 22:01:05.055415 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055344    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24889d61-6b7f-4804-9975-ed369fb14af1" containerName="revoke-keys"
Jul 17 22:01:05.055415 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055349    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="24889d61-6b7f-4804-9975-ed369fb14af1" containerName="revoke-keys"
Jul 17 22:01:05.055415 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055356    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3665735c-f895-4d5d-a2b5-e2a82e6398e8" containerName="revoke-keys"
Jul 17 22:01:05.055415 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055361    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3665735c-f895-4d5d-a2b5-e2a82e6398e8" containerName="revoke-keys"
Jul 17 22:01:05.055415 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055370    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24889d61-6b7f-4804-9975-ed369fb14af1" containerName="revoke-keys"
Jul 17 22:01:05.055415 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055376    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="24889d61-6b7f-4804-9975-ed369fb14af1" containerName="revoke-keys"
Jul 17 22:01:05.055415 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055421    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3665735c-f895-4d5d-a2b5-e2a82e6398e8" containerName="revoke-keys"
Jul 17 22:01:05.055924 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055430    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddaba242-98ed-49e2-b44f-2c09173bd013" containerName="revoke-keys"
Jul 17 22:01:05.055924 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055437    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="24889d61-6b7f-4804-9975-ed369fb14af1" containerName="revoke-keys"
Jul 17 22:01:05.055924 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055442    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddaba242-98ed-49e2-b44f-2c09173bd013" containerName="revoke-keys"
Jul 17 22:01:05.055924 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055448    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="24889d61-6b7f-4804-9975-ed369fb14af1" containerName="revoke-keys"
Jul 17 22:01:05.055924 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055453    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="24889d61-6b7f-4804-9975-ed369fb14af1" containerName="revoke-keys"
Jul 17 22:01:05.055924 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.055473    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3665735c-f895-4d5d-a2b5-e2a82e6398e8" containerName="revoke-keys"
Jul 17 22:01:05.058115 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.058099    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kfpwg"
Jul 17 22:01:05.060482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.060463    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 22:01:05.060592 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.060549    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-nw6jm\""
Jul 17 22:01:05.061454 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.061442    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 22:01:05.066162 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.066143    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kfpwg"]
Jul 17 22:01:05.138741 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.138706    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcj7g\" (UniqueName: \"kubernetes.io/projected/8f221057-b697-4981-b686-f96890e43f39-kube-api-access-dcj7g\") pod \"kuadrant-operator-catalog-kfpwg\" (UID: \"8f221057-b697-4981-b686-f96890e43f39\") " pod="kuadrant-system/kuadrant-operator-catalog-kfpwg"
Jul 17 22:01:05.239748 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.239714    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcj7g\" (UniqueName: \"kubernetes.io/projected/8f221057-b697-4981-b686-f96890e43f39-kube-api-access-dcj7g\") pod \"kuadrant-operator-catalog-kfpwg\" (UID: \"8f221057-b697-4981-b686-f96890e43f39\") " pod="kuadrant-system/kuadrant-operator-catalog-kfpwg"
Jul 17 22:01:05.247531 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.247503    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcj7g\" (UniqueName: \"kubernetes.io/projected/8f221057-b697-4981-b686-f96890e43f39-kube-api-access-dcj7g\") pod \"kuadrant-operator-catalog-kfpwg\" (UID: \"8f221057-b697-4981-b686-f96890e43f39\") " pod="kuadrant-system/kuadrant-operator-catalog-kfpwg"
Jul 17 22:01:05.367944 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.367913    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kfpwg"
Jul 17 22:01:05.492351 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.492237    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kfpwg"]
Jul 17 22:01:05.494881 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:01:05.494856    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f221057_b697_4981_b686_f96890e43f39.slice/crio-b51bef11c4754624e9e8007994aa85fa580354cfdba4ce589115d4f66f51cd0d WatchSource:0}: Error finding container b51bef11c4754624e9e8007994aa85fa580354cfdba4ce589115d4f66f51cd0d: Status 404 returned error can't find the container with id b51bef11c4754624e9e8007994aa85fa580354cfdba4ce589115d4f66f51cd0d
Jul 17 22:01:05.496300 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.496265    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 22:01:05.584173 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:05.584141    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kfpwg" event={"ID":"8f221057-b697-4981-b686-f96890e43f39","Type":"ContainerStarted","Data":"b51bef11c4754624e9e8007994aa85fa580354cfdba4ce589115d4f66f51cd0d"}
Jul 17 22:01:08.597866 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:08.597821    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kfpwg" event={"ID":"8f221057-b697-4981-b686-f96890e43f39","Type":"ContainerStarted","Data":"e808783c571e02f773efb7a5454236edaeeb52ba64c57056f3694df85bae45e8"}
Jul 17 22:01:08.614796 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:08.614731    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-kfpwg" podStartSLOduration=1.556511792 podStartE2EDuration="3.614705621s" totalImagesPullingTime="2.058193829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:01:08.614662393 +0000 UTC m=+3168.578783003" watchObservedRunningTime="2026-07-17 22:01:08.614705621 +0000 UTC m=+3168.578826232"
Jul 17 22:01:11.015867 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:11.015829    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7"]
Jul 17 22:01:11.016267 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:11.016110    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddaba242-98ed-49e2-b44f-2c09173bd013" containerName="revoke-keys"
Jul 17 22:01:11.016267 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:11.016120    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddaba242-98ed-49e2-b44f-2c09173bd013" containerName="revoke-keys"
Jul 17 22:01:11.016267 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:11.016169    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddaba242-98ed-49e2-b44f-2c09173bd013" containerName="revoke-keys"
Jul 17 22:01:11.016267 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:11.016178    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3665735c-f895-4d5d-a2b5-e2a82e6398e8" containerName="revoke-keys"
Jul 17 22:01:11.019166 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:11.019150    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7"
Jul 17 22:01:11.021593 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:11.021573    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:01:11.021694 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:11.021603    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 22:01:11.026971 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:11.026947    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7"]
Jul 17 22:01:11.084733 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:11.084704    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76413919-f253-462a-bebf-65c9c1fec911-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-mmhg7\" (UID: \"76413919-f253-462a-bebf-65c9c1fec911\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7"
Jul 17 22:01:11.185942 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:11.185905    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76413919-f253-462a-bebf-65c9c1fec911-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-mmhg7\" (UID: \"76413919-f253-462a-bebf-65c9c1fec911\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7"
Jul 17 22:01:11.186584 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:11.186563    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76413919-f253-462a-bebf-65c9c1fec911-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-mmhg7\" (UID: \"76413919-f253-462a-bebf-65c9c1fec911\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7"
Jul 17 22:01:11.330369 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:11.330294    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7"
Jul 17 22:01:11.452533 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:11.452506    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7"]
Jul 17 22:01:11.454492 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:01:11.454466    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76413919_f253_462a_bebf_65c9c1fec911.slice/crio-8f4848172259880732d4ea997221e05ce3829c9d0cbe7da03516975f531d2f38 WatchSource:0}: Error finding container 8f4848172259880732d4ea997221e05ce3829c9d0cbe7da03516975f531d2f38: Status 404 returned error can't find the container with id 8f4848172259880732d4ea997221e05ce3829c9d0cbe7da03516975f531d2f38
Jul 17 22:01:11.611493 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:11.611465    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7" event={"ID":"76413919-f253-462a-bebf-65c9c1fec911","Type":"ContainerStarted","Data":"8f4848172259880732d4ea997221e05ce3829c9d0cbe7da03516975f531d2f38"}
Jul 17 22:01:12.617244 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:12.617211    2573 generic.go:356] "Generic (PLEG): container finished" podID="76413919-f253-462a-bebf-65c9c1fec911" containerID="0962beae9270bd0db839be5e53ee67d67523472b7c1c31b0d7c9b604540cd061" exitCode=6
Jul 17 22:01:12.617244 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:12.617249    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7" event={"ID":"76413919-f253-462a-bebf-65c9c1fec911","Type":"ContainerDied","Data":"0962beae9270bd0db839be5e53ee67d67523472b7c1c31b0d7c9b604540cd061"}
Jul 17 22:01:12.617682 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:12.617461    2573 scope.go:117] "RemoveContainer" containerID="0962beae9270bd0db839be5e53ee67d67523472b7c1c31b0d7c9b604540cd061"
Jul 17 22:01:13.623647 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:13.623612    2573 generic.go:356] "Generic (PLEG): container finished" podID="76413919-f253-462a-bebf-65c9c1fec911" containerID="5fdbffa6f70b747440e6b9484c65442dabbb0a2e54e84d8b5048d1fb036abcad" exitCode=6
Jul 17 22:01:13.624058 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:13.623683    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7" event={"ID":"76413919-f253-462a-bebf-65c9c1fec911","Type":"ContainerDied","Data":"5fdbffa6f70b747440e6b9484c65442dabbb0a2e54e84d8b5048d1fb036abcad"}
Jul 17 22:01:13.624058 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:13.623745    2573 scope.go:117] "RemoveContainer" containerID="0962beae9270bd0db839be5e53ee67d67523472b7c1c31b0d7c9b604540cd061"
Jul 17 22:01:13.624058 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:13.623894    2573 scope.go:117] "RemoveContainer" containerID="5fdbffa6f70b747440e6b9484c65442dabbb0a2e54e84d8b5048d1fb036abcad"
Jul 17 22:01:13.624183 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:01:13.624064    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-mmhg7_odh-ai-gateway-infra(76413919-f253-462a-bebf-65c9c1fec911)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7" podUID="76413919-f253-462a-bebf-65c9c1fec911"
Jul 17 22:01:14.630642 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:14.630610    2573 scope.go:117] "RemoveContainer" containerID="5fdbffa6f70b747440e6b9484c65442dabbb0a2e54e84d8b5048d1fb036abcad"
Jul 17 22:01:14.630976 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:01:14.630769    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-mmhg7_odh-ai-gateway-infra(76413919-f253-462a-bebf-65c9c1fec911)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7" podUID="76413919-f253-462a-bebf-65c9c1fec911"
Jul 17 22:01:15.368994 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:15.368954    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-kfpwg"
Jul 17 22:01:15.369196 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:15.369026    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-kfpwg"
Jul 17 22:01:15.391086 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:15.391063    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-kfpwg"
Jul 17 22:01:15.656455 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:15.656376    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-kfpwg"
Jul 17 22:01:15.687621 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:15.687591    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kfpwg"]
Jul 17 22:01:17.644176 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:17.644125    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-kfpwg" podUID="8f221057-b697-4981-b686-f96890e43f39" containerName="registry-server" containerID="cri-o://e808783c571e02f773efb7a5454236edaeeb52ba64c57056f3694df85bae45e8" gracePeriod=2
Jul 17 22:01:17.887214 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:17.887194    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kfpwg"
Jul 17 22:01:18.045709 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:18.045611    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcj7g\" (UniqueName: \"kubernetes.io/projected/8f221057-b697-4981-b686-f96890e43f39-kube-api-access-dcj7g\") pod \"8f221057-b697-4981-b686-f96890e43f39\" (UID: \"8f221057-b697-4981-b686-f96890e43f39\") "
Jul 17 22:01:18.047816 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:18.047786    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f221057-b697-4981-b686-f96890e43f39-kube-api-access-dcj7g" (OuterVolumeSpecName: "kube-api-access-dcj7g") pod "8f221057-b697-4981-b686-f96890e43f39" (UID: "8f221057-b697-4981-b686-f96890e43f39"). InnerVolumeSpecName "kube-api-access-dcj7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:01:18.146110 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:18.146085    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcj7g\" (UniqueName: \"kubernetes.io/projected/8f221057-b697-4981-b686-f96890e43f39-kube-api-access-dcj7g\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:01:18.649346 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:18.649317    2573 generic.go:356] "Generic (PLEG): container finished" podID="8f221057-b697-4981-b686-f96890e43f39" containerID="e808783c571e02f773efb7a5454236edaeeb52ba64c57056f3694df85bae45e8" exitCode=0
Jul 17 22:01:18.649788 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:18.649362    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kfpwg"
Jul 17 22:01:18.649788 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:18.649363    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kfpwg" event={"ID":"8f221057-b697-4981-b686-f96890e43f39","Type":"ContainerDied","Data":"e808783c571e02f773efb7a5454236edaeeb52ba64c57056f3694df85bae45e8"}
Jul 17 22:01:18.649788 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:18.649454    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kfpwg" event={"ID":"8f221057-b697-4981-b686-f96890e43f39","Type":"ContainerDied","Data":"b51bef11c4754624e9e8007994aa85fa580354cfdba4ce589115d4f66f51cd0d"}
Jul 17 22:01:18.649788 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:18.649479    2573 scope.go:117] "RemoveContainer" containerID="e808783c571e02f773efb7a5454236edaeeb52ba64c57056f3694df85bae45e8"
Jul 17 22:01:18.658254 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:18.658235    2573 scope.go:117] "RemoveContainer" containerID="e808783c571e02f773efb7a5454236edaeeb52ba64c57056f3694df85bae45e8"
Jul 17 22:01:18.658507 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:01:18.658491    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e808783c571e02f773efb7a5454236edaeeb52ba64c57056f3694df85bae45e8\": container with ID starting with e808783c571e02f773efb7a5454236edaeeb52ba64c57056f3694df85bae45e8 not found: ID does not exist" containerID="e808783c571e02f773efb7a5454236edaeeb52ba64c57056f3694df85bae45e8"
Jul 17 22:01:18.658562 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:18.658514    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e808783c571e02f773efb7a5454236edaeeb52ba64c57056f3694df85bae45e8"} err="failed to get container status \"e808783c571e02f773efb7a5454236edaeeb52ba64c57056f3694df85bae45e8\": rpc error: code = NotFound desc = could not find container \"e808783c571e02f773efb7a5454236edaeeb52ba64c57056f3694df85bae45e8\": container with ID starting with e808783c571e02f773efb7a5454236edaeeb52ba64c57056f3694df85bae45e8 not found: ID does not exist"
Jul 17 22:01:18.664178 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:18.664158    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kfpwg"]
Jul 17 22:01:18.668043 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:18.668022    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kfpwg"]
Jul 17 22:01:20.530466 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:20.530432    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f221057-b697-4981-b686-f96890e43f39" path="/var/lib/kubelet/pods/8f221057-b697-4981-b686-f96890e43f39/volumes"
Jul 17 22:01:25.710673 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:25.710643    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7"]
Jul 17 22:01:25.711037 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:25.710946    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f221057-b697-4981-b686-f96890e43f39" containerName="registry-server"
Jul 17 22:01:25.711037 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:25.710957    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f221057-b697-4981-b686-f96890e43f39" containerName="registry-server"
Jul 17 22:01:25.711037 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:25.711004    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f221057-b697-4981-b686-f96890e43f39" containerName="registry-server"
Jul 17 22:01:25.715398 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:25.715381    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7"
Jul 17 22:01:25.720545 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:25.720519    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7"]
Jul 17 22:01:25.804190 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:25.804155    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a124b86e-89f7-4554-aef4-dda076584c4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7\" (UID: \"a124b86e-89f7-4554-aef4-dda076584c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7"
Jul 17 22:01:25.904757 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:25.904722    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a124b86e-89f7-4554-aef4-dda076584c4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7\" (UID: \"a124b86e-89f7-4554-aef4-dda076584c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7"
Jul 17 22:01:25.905326 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:25.905309    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a124b86e-89f7-4554-aef4-dda076584c4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7\" (UID: \"a124b86e-89f7-4554-aef4-dda076584c4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7"
Jul 17 22:01:26.026021 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:26.025957    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7"
Jul 17 22:01:26.150764 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:26.150737    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7"]
Jul 17 22:01:26.152579 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:01:26.152551    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda124b86e_89f7_4554_aef4_dda076584c4b.slice/crio-a9cb7df748a437fd1034e829a729537f4d13be7f900dbc83f440497b6a2d284b WatchSource:0}: Error finding container a9cb7df748a437fd1034e829a729537f4d13be7f900dbc83f440497b6a2d284b: Status 404 returned error can't find the container with id a9cb7df748a437fd1034e829a729537f4d13be7f900dbc83f440497b6a2d284b
Jul 17 22:01:26.686546 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:26.686464    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7" event={"ID":"a124b86e-89f7-4554-aef4-dda076584c4b","Type":"ContainerStarted","Data":"1c8d08f0c320a8b7cef972484e9c115f6938545eecbc919a0649238e2207d574"}
Jul 17 22:01:26.686546 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:26.686517    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7" event={"ID":"a124b86e-89f7-4554-aef4-dda076584c4b","Type":"ContainerStarted","Data":"a9cb7df748a437fd1034e829a729537f4d13be7f900dbc83f440497b6a2d284b"}
Jul 17 22:01:26.701069 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:26.701028    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7" podStartSLOduration=1.7010138879999999 podStartE2EDuration="1.701013888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:01:26.700030306 +0000 UTC m=+3186.664150916" watchObservedRunningTime="2026-07-17 22:01:26.701013888 +0000 UTC m=+3186.665134497"
Jul 17 22:01:27.525198 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:27.525171    2573 scope.go:117] "RemoveContainer" containerID="5fdbffa6f70b747440e6b9484c65442dabbb0a2e54e84d8b5048d1fb036abcad"
Jul 17 22:01:27.692487 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:27.692447    2573 generic.go:356] "Generic (PLEG): container finished" podID="a124b86e-89f7-4554-aef4-dda076584c4b" containerID="1c8d08f0c320a8b7cef972484e9c115f6938545eecbc919a0649238e2207d574" exitCode=6
Jul 17 22:01:27.692776 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:27.692523    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7" event={"ID":"a124b86e-89f7-4554-aef4-dda076584c4b","Type":"ContainerDied","Data":"1c8d08f0c320a8b7cef972484e9c115f6938545eecbc919a0649238e2207d574"}
Jul 17 22:01:27.692776 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:27.692717    2573 scope.go:117] "RemoveContainer" containerID="1c8d08f0c320a8b7cef972484e9c115f6938545eecbc919a0649238e2207d574"
Jul 17 22:01:27.955028 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:27.955005    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl"]
Jul 17 22:01:27.957994 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:27.957977    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl"
Jul 17 22:01:27.965382 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:27.965361    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl"]
Jul 17 22:01:28.123192 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:28.123167    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d84c5df-94c4-4433-87cd-790dfcfe2878-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl\" (UID: \"9d84c5df-94c4-4433-87cd-790dfcfe2878\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl"
Jul 17 22:01:28.223537 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:28.223506    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d84c5df-94c4-4433-87cd-790dfcfe2878-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl\" (UID: \"9d84c5df-94c4-4433-87cd-790dfcfe2878\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl"
Jul 17 22:01:28.224109 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:28.224091    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d84c5df-94c4-4433-87cd-790dfcfe2878-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl\" (UID: \"9d84c5df-94c4-4433-87cd-790dfcfe2878\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl"
Jul 17 22:01:28.269576 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:28.269551    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl"
Jul 17 22:01:28.398136 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:28.393311    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl"]
Jul 17 22:01:28.399144 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:01:28.399115    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d84c5df_94c4_4433_87cd_790dfcfe2878.slice/crio-8aac977d5294d9bd014d6e020a15fee9a5cbbaa094286ff2001ea8dbc003d588 WatchSource:0}: Error finding container 8aac977d5294d9bd014d6e020a15fee9a5cbbaa094286ff2001ea8dbc003d588: Status 404 returned error can't find the container with id 8aac977d5294d9bd014d6e020a15fee9a5cbbaa094286ff2001ea8dbc003d588
Jul 17 22:01:28.698617 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:28.698573    2573 generic.go:356] "Generic (PLEG): container finished" podID="a124b86e-89f7-4554-aef4-dda076584c4b" containerID="7107e29320309f953d753b2252b659eaf068e9fbc40bed28c5edc90796cb9fc1" exitCode=6
Jul 17 22:01:28.699064 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:28.698653    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7" event={"ID":"a124b86e-89f7-4554-aef4-dda076584c4b","Type":"ContainerDied","Data":"7107e29320309f953d753b2252b659eaf068e9fbc40bed28c5edc90796cb9fc1"}
Jul 17 22:01:28.699064 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:28.698727    2573 scope.go:117] "RemoveContainer" containerID="1c8d08f0c320a8b7cef972484e9c115f6938545eecbc919a0649238e2207d574"
Jul 17 22:01:28.699064 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:28.698974    2573 scope.go:117] "RemoveContainer" containerID="7107e29320309f953d753b2252b659eaf068e9fbc40bed28c5edc90796cb9fc1"
Jul 17 22:01:28.699233 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:01:28.699184    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7_odh-ai-gateway-infra(a124b86e-89f7-4554-aef4-dda076584c4b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7" podUID="a124b86e-89f7-4554-aef4-dda076584c4b"
Jul 17 22:01:28.700009 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:28.699988    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl" event={"ID":"9d84c5df-94c4-4433-87cd-790dfcfe2878","Type":"ContainerStarted","Data":"8aac977d5294d9bd014d6e020a15fee9a5cbbaa094286ff2001ea8dbc003d588"}
Jul 17 22:01:28.701530 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:28.701503    2573 generic.go:356] "Generic (PLEG): container finished" podID="76413919-f253-462a-bebf-65c9c1fec911" containerID="f2258bfa15c2944525b1ccd49da1debba8fea5b75ea8e48fc074329aaf7d2882" exitCode=6
Jul 17 22:01:28.701608 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:28.701540    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7" event={"ID":"76413919-f253-462a-bebf-65c9c1fec911","Type":"ContainerDied","Data":"f2258bfa15c2944525b1ccd49da1debba8fea5b75ea8e48fc074329aaf7d2882"}
Jul 17 22:01:28.701809 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:28.701796    2573 scope.go:117] "RemoveContainer" containerID="f2258bfa15c2944525b1ccd49da1debba8fea5b75ea8e48fc074329aaf7d2882"
Jul 17 22:01:28.701964 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:01:28.701948    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-mmhg7_odh-ai-gateway-infra(76413919-f253-462a-bebf-65c9c1fec911)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7" podUID="76413919-f253-462a-bebf-65c9c1fec911"
Jul 17 22:01:28.710950 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:28.710934    2573 scope.go:117] "RemoveContainer" containerID="5fdbffa6f70b747440e6b9484c65442dabbb0a2e54e84d8b5048d1fb036abcad"
Jul 17 22:01:29.708115 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:29.708090    2573 scope.go:117] "RemoveContainer" containerID="7107e29320309f953d753b2252b659eaf068e9fbc40bed28c5edc90796cb9fc1"
Jul 17 22:01:29.708532 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:01:29.708332    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7_odh-ai-gateway-infra(a124b86e-89f7-4554-aef4-dda076584c4b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7" podUID="a124b86e-89f7-4554-aef4-dda076584c4b"
Jul 17 22:01:29.709374 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:29.709347    2573 generic.go:356] "Generic (PLEG): container finished" podID="9d84c5df-94c4-4433-87cd-790dfcfe2878" containerID="428351e2c7485452566194815c915cfe298658be53f89d0d46875bb102df2178" exitCode=6
Jul 17 22:01:29.709476 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:29.709418    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl" event={"ID":"9d84c5df-94c4-4433-87cd-790dfcfe2878","Type":"ContainerDied","Data":"428351e2c7485452566194815c915cfe298658be53f89d0d46875bb102df2178"}
Jul 17 22:01:29.709589 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:29.709576    2573 scope.go:117] "RemoveContainer" containerID="428351e2c7485452566194815c915cfe298658be53f89d0d46875bb102df2178"
Jul 17 22:01:29.741479 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:29.741451    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7"]
Jul 17 22:01:29.863489 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:29.863462    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj"]
Jul 17 22:01:29.868147 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:29.868128    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj"
Jul 17 22:01:29.868822 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:29.868805    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7"
Jul 17 22:01:29.874525 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:29.874487    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj"]
Jul 17 22:01:30.037891 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.037803    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76413919-f253-462a-bebf-65c9c1fec911-maas-api-service-ca\") pod \"76413919-f253-462a-bebf-65c9c1fec911\" (UID: \"76413919-f253-462a-bebf-65c9c1fec911\") "
Jul 17 22:01:30.038056 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.038024    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f668be2-c58f-4c81-83b0-85c237b6bc02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj\" (UID: \"3f668be2-c58f-4c81-83b0-85c237b6bc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj"
Jul 17 22:01:30.038223 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.038197    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76413919-f253-462a-bebf-65c9c1fec911-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "76413919-f253-462a-bebf-65c9c1fec911" (UID: "76413919-f253-462a-bebf-65c9c1fec911"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:01:30.139554 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.139527    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f668be2-c58f-4c81-83b0-85c237b6bc02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj\" (UID: \"3f668be2-c58f-4c81-83b0-85c237b6bc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj"
Jul 17 22:01:30.139723 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.139606    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76413919-f253-462a-bebf-65c9c1fec911-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:01:30.140148 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.140128    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f668be2-c58f-4c81-83b0-85c237b6bc02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj\" (UID: \"3f668be2-c58f-4c81-83b0-85c237b6bc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj"
Jul 17 22:01:30.180135 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.180105    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj"
Jul 17 22:01:30.305861 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.305665    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj"]
Jul 17 22:01:30.308519 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:01:30.308494    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f668be2_c58f_4c81_83b0_85c237b6bc02.slice/crio-3f968e96612cd183df70e1dd7d444f0df52e0662f3096bb2a9c88837f965d65b WatchSource:0}: Error finding container 3f968e96612cd183df70e1dd7d444f0df52e0662f3096bb2a9c88837f965d65b: Status 404 returned error can't find the container with id 3f968e96612cd183df70e1dd7d444f0df52e0662f3096bb2a9c88837f965d65b
Jul 17 22:01:30.715957 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.715930    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7"
Jul 17 22:01:30.716429 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.715954    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7" event={"ID":"76413919-f253-462a-bebf-65c9c1fec911","Type":"ContainerDied","Data":"8f4848172259880732d4ea997221e05ce3829c9d0cbe7da03516975f531d2f38"}
Jul 17 22:01:30.716429 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.716014    2573 scope.go:117] "RemoveContainer" containerID="f2258bfa15c2944525b1ccd49da1debba8fea5b75ea8e48fc074329aaf7d2882"
Jul 17 22:01:30.717567 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.717544    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj" event={"ID":"3f668be2-c58f-4c81-83b0-85c237b6bc02","Type":"ContainerStarted","Data":"8c368c5a6941e6996e3d73656f074f1f44e36f11e65a3dc7f299ac1cc48b0ca9"}
Jul 17 22:01:30.717705 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.717590    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj" event={"ID":"3f668be2-c58f-4c81-83b0-85c237b6bc02","Type":"ContainerStarted","Data":"3f968e96612cd183df70e1dd7d444f0df52e0662f3096bb2a9c88837f965d65b"}
Jul 17 22:01:30.719719 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.719695    2573 generic.go:356] "Generic (PLEG): container finished" podID="9d84c5df-94c4-4433-87cd-790dfcfe2878" containerID="10990d66e39b12a10f721542d41e036a31582a97a82a0934d836c353e534ec9b" exitCode=6
Jul 17 22:01:30.719843 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.719738    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl" event={"ID":"9d84c5df-94c4-4433-87cd-790dfcfe2878","Type":"ContainerDied","Data":"10990d66e39b12a10f721542d41e036a31582a97a82a0934d836c353e534ec9b"}
Jul 17 22:01:30.719926 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.719906    2573 scope.go:117] "RemoveContainer" containerID="10990d66e39b12a10f721542d41e036a31582a97a82a0934d836c353e534ec9b"
Jul 17 22:01:30.720120 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:01:30.720104    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl_odh-ai-gateway-infra(9d84c5df-94c4-4433-87cd-790dfcfe2878)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl" podUID="9d84c5df-94c4-4433-87cd-790dfcfe2878"
Jul 17 22:01:30.730973 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.730956    2573 scope.go:117] "RemoveContainer" containerID="428351e2c7485452566194815c915cfe298658be53f89d0d46875bb102df2178"
Jul 17 22:01:30.751420 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.751383    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj" podStartSLOduration=1.75137059 podStartE2EDuration="1.75137059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:01:30.75081431 +0000 UTC m=+3190.714934921" watchObservedRunningTime="2026-07-17 22:01:30.75137059 +0000 UTC m=+3190.715491219"
Jul 17 22:01:30.766242 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.766214    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7"]
Jul 17 22:01:30.772447 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:30.772424    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-mmhg7"]
Jul 17 22:01:31.725116 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:31.725079    2573 generic.go:356] "Generic (PLEG): container finished" podID="3f668be2-c58f-4c81-83b0-85c237b6bc02" containerID="8c368c5a6941e6996e3d73656f074f1f44e36f11e65a3dc7f299ac1cc48b0ca9" exitCode=6
Jul 17 22:01:31.725574 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:31.725163    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj" event={"ID":"3f668be2-c58f-4c81-83b0-85c237b6bc02","Type":"ContainerDied","Data":"8c368c5a6941e6996e3d73656f074f1f44e36f11e65a3dc7f299ac1cc48b0ca9"}
Jul 17 22:01:31.725574 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:31.725312    2573 scope.go:117] "RemoveContainer" containerID="8c368c5a6941e6996e3d73656f074f1f44e36f11e65a3dc7f299ac1cc48b0ca9"
Jul 17 22:01:31.726951 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:31.726932    2573 scope.go:117] "RemoveContainer" containerID="10990d66e39b12a10f721542d41e036a31582a97a82a0934d836c353e534ec9b"
Jul 17 22:01:31.727118 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:01:31.727100    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl_odh-ai-gateway-infra(9d84c5df-94c4-4433-87cd-790dfcfe2878)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl" podUID="9d84c5df-94c4-4433-87cd-790dfcfe2878"
Jul 17 22:01:32.529804 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:32.529776    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76413919-f253-462a-bebf-65c9c1fec911" path="/var/lib/kubelet/pods/76413919-f253-462a-bebf-65c9c1fec911/volumes"
Jul 17 22:01:32.733249 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:32.733219    2573 generic.go:356] "Generic (PLEG): container finished" podID="3f668be2-c58f-4c81-83b0-85c237b6bc02" containerID="2ba7451927dfb29f794762bf401d4d2ab34822dbf175980620a31cde09e2b642" exitCode=6
Jul 17 22:01:32.733717 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:32.733315    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj" event={"ID":"3f668be2-c58f-4c81-83b0-85c237b6bc02","Type":"ContainerDied","Data":"2ba7451927dfb29f794762bf401d4d2ab34822dbf175980620a31cde09e2b642"}
Jul 17 22:01:32.733717 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:32.733371    2573 scope.go:117] "RemoveContainer" containerID="8c368c5a6941e6996e3d73656f074f1f44e36f11e65a3dc7f299ac1cc48b0ca9"
Jul 17 22:01:32.733717 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:32.733567    2573 scope.go:117] "RemoveContainer" containerID="2ba7451927dfb29f794762bf401d4d2ab34822dbf175980620a31cde09e2b642"
Jul 17 22:01:32.733907 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:01:32.733776    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj_odh-ai-gateway-infra(3f668be2-c58f-4c81-83b0-85c237b6bc02)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj" podUID="3f668be2-c58f-4c81-83b0-85c237b6bc02"
Jul 17 22:01:33.739507 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:33.739481    2573 scope.go:117] "RemoveContainer" containerID="2ba7451927dfb29f794762bf401d4d2ab34822dbf175980620a31cde09e2b642"
Jul 17 22:01:33.739884 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:01:33.739649    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj_odh-ai-gateway-infra(3f668be2-c58f-4c81-83b0-85c237b6bc02)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj" podUID="3f668be2-c58f-4c81-83b0-85c237b6bc02"
Jul 17 22:01:41.525094 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:41.525063    2573 scope.go:117] "RemoveContainer" containerID="7107e29320309f953d753b2252b659eaf068e9fbc40bed28c5edc90796cb9fc1"
Jul 17 22:01:41.773883 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:41.773855    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7" event={"ID":"a124b86e-89f7-4554-aef4-dda076584c4b","Type":"ContainerStarted","Data":"20805e076f8c722fef4f034d205b6daba911273b9ed2dcd8c7881ecdead5ce4b"}
Jul 17 22:01:42.779793 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:42.779757    2573 generic.go:356] "Generic (PLEG): container finished" podID="a124b86e-89f7-4554-aef4-dda076584c4b" containerID="20805e076f8c722fef4f034d205b6daba911273b9ed2dcd8c7881ecdead5ce4b" exitCode=6
Jul 17 22:01:42.780190 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:42.779826    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7" event={"ID":"a124b86e-89f7-4554-aef4-dda076584c4b","Type":"ContainerDied","Data":"20805e076f8c722fef4f034d205b6daba911273b9ed2dcd8c7881ecdead5ce4b"}
Jul 17 22:01:42.780190 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:42.779888    2573 scope.go:117] "RemoveContainer" containerID="7107e29320309f953d753b2252b659eaf068e9fbc40bed28c5edc90796cb9fc1"
Jul 17 22:01:42.780190 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:42.780122    2573 scope.go:117] "RemoveContainer" containerID="20805e076f8c722fef4f034d205b6daba911273b9ed2dcd8c7881ecdead5ce4b"
Jul 17 22:01:42.780367 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:01:42.780348    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7_odh-ai-gateway-infra(a124b86e-89f7-4554-aef4-dda076584c4b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7" podUID="a124b86e-89f7-4554-aef4-dda076584c4b"
Jul 17 22:01:42.801199 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:42.801176    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7"]
Jul 17 22:01:43.942376 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:43.942350    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7"
Jul 17 22:01:44.054567 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:44.054491    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a124b86e-89f7-4554-aef4-dda076584c4b-maas-api-service-ca\") pod \"a124b86e-89f7-4554-aef4-dda076584c4b\" (UID: \"a124b86e-89f7-4554-aef4-dda076584c4b\") "
Jul 17 22:01:44.054852 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:44.054831    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a124b86e-89f7-4554-aef4-dda076584c4b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a124b86e-89f7-4554-aef4-dda076584c4b" (UID: "a124b86e-89f7-4554-aef4-dda076584c4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:01:44.155764 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:44.155742    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a124b86e-89f7-4554-aef4-dda076584c4b-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:01:44.791583 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:44.791513    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7"
Jul 17 22:01:44.791764 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:44.791511    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7" event={"ID":"a124b86e-89f7-4554-aef4-dda076584c4b","Type":"ContainerDied","Data":"a9cb7df748a437fd1034e829a729537f4d13be7f900dbc83f440497b6a2d284b"}
Jul 17 22:01:44.791764 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:44.791659    2573 scope.go:117] "RemoveContainer" containerID="20805e076f8c722fef4f034d205b6daba911273b9ed2dcd8c7881ecdead5ce4b"
Jul 17 22:01:44.807124 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:44.807102    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7"]
Jul 17 22:01:44.817194 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:44.817173    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-nhsj7"]
Jul 17 22:01:46.524568 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:46.524540    2573 scope.go:117] "RemoveContainer" containerID="2ba7451927dfb29f794762bf401d4d2ab34822dbf175980620a31cde09e2b642"
Jul 17 22:01:46.524988 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:46.524645    2573 scope.go:117] "RemoveContainer" containerID="10990d66e39b12a10f721542d41e036a31582a97a82a0934d836c353e534ec9b"
Jul 17 22:01:46.529767 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:46.529744    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a124b86e-89f7-4554-aef4-dda076584c4b" path="/var/lib/kubelet/pods/a124b86e-89f7-4554-aef4-dda076584c4b/volumes"
Jul 17 22:01:47.808396 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:47.808363    2573 generic.go:356] "Generic (PLEG): container finished" podID="3f668be2-c58f-4c81-83b0-85c237b6bc02" containerID="26c2440b1a73189bda77fd4abb58aca1eac24ba8ecd526d34ceab18898a4c6a9" exitCode=6
Jul 17 22:01:47.808814 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:47.808440    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj" event={"ID":"3f668be2-c58f-4c81-83b0-85c237b6bc02","Type":"ContainerDied","Data":"26c2440b1a73189bda77fd4abb58aca1eac24ba8ecd526d34ceab18898a4c6a9"}
Jul 17 22:01:47.808814 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:47.808496    2573 scope.go:117] "RemoveContainer" containerID="2ba7451927dfb29f794762bf401d4d2ab34822dbf175980620a31cde09e2b642"
Jul 17 22:01:47.808814 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:47.808771    2573 scope.go:117] "RemoveContainer" containerID="26c2440b1a73189bda77fd4abb58aca1eac24ba8ecd526d34ceab18898a4c6a9"
Jul 17 22:01:47.808994 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:01:47.808975    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj_odh-ai-gateway-infra(3f668be2-c58f-4c81-83b0-85c237b6bc02)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj" podUID="3f668be2-c58f-4c81-83b0-85c237b6bc02"
Jul 17 22:01:47.810367 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:47.810349    2573 generic.go:356] "Generic (PLEG): container finished" podID="9d84c5df-94c4-4433-87cd-790dfcfe2878" containerID="04a143b1723ea9764a72f0cf1ae9430cc436006e6e3f0574ef339ccfdc0b6892" exitCode=6
Jul 17 22:01:47.810437 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:47.810419    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl" event={"ID":"9d84c5df-94c4-4433-87cd-790dfcfe2878","Type":"ContainerDied","Data":"04a143b1723ea9764a72f0cf1ae9430cc436006e6e3f0574ef339ccfdc0b6892"}
Jul 17 22:01:47.810694 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:47.810681    2573 scope.go:117] "RemoveContainer" containerID="04a143b1723ea9764a72f0cf1ae9430cc436006e6e3f0574ef339ccfdc0b6892"
Jul 17 22:01:47.810832 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:01:47.810817    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl_odh-ai-gateway-infra(9d84c5df-94c4-4433-87cd-790dfcfe2878)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl" podUID="9d84c5df-94c4-4433-87cd-790dfcfe2878"
Jul 17 22:01:47.818725 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:47.818708    2573 scope.go:117] "RemoveContainer" containerID="10990d66e39b12a10f721542d41e036a31582a97a82a0934d836c353e534ec9b"
Jul 17 22:01:48.835557 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:48.835525    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj"]
Jul 17 22:01:48.848389 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:48.848364    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl"]
Jul 17 22:01:48.972592 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:48.972523    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj"
Jul 17 22:01:48.984958 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:48.984938    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl"
Jul 17 22:01:49.095131 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:49.095058    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d84c5df-94c4-4433-87cd-790dfcfe2878-maas-api-service-ca\") pod \"9d84c5df-94c4-4433-87cd-790dfcfe2878\" (UID: \"9d84c5df-94c4-4433-87cd-790dfcfe2878\") "
Jul 17 22:01:49.095131 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:49.095099    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f668be2-c58f-4c81-83b0-85c237b6bc02-maas-api-service-ca\") pod \"3f668be2-c58f-4c81-83b0-85c237b6bc02\" (UID: \"3f668be2-c58f-4c81-83b0-85c237b6bc02\") "
Jul 17 22:01:49.095459 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:49.095440    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d84c5df-94c4-4433-87cd-790dfcfe2878-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9d84c5df-94c4-4433-87cd-790dfcfe2878" (UID: "9d84c5df-94c4-4433-87cd-790dfcfe2878"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:01:49.095512 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:49.095493    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f668be2-c58f-4c81-83b0-85c237b6bc02-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f668be2-c58f-4c81-83b0-85c237b6bc02" (UID: "3f668be2-c58f-4c81-83b0-85c237b6bc02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:01:49.195781 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:49.195755    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d84c5df-94c4-4433-87cd-790dfcfe2878-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:01:49.195781 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:49.195778    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f668be2-c58f-4c81-83b0-85c237b6bc02-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:01:49.823457 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:49.823431    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl"
Jul 17 22:01:49.823639 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:49.823480    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl" event={"ID":"9d84c5df-94c4-4433-87cd-790dfcfe2878","Type":"ContainerDied","Data":"8aac977d5294d9bd014d6e020a15fee9a5cbbaa094286ff2001ea8dbc003d588"}
Jul 17 22:01:49.823639 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:49.823558    2573 scope.go:117] "RemoveContainer" containerID="04a143b1723ea9764a72f0cf1ae9430cc436006e6e3f0574ef339ccfdc0b6892"
Jul 17 22:01:49.824697 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:49.824678    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj" event={"ID":"3f668be2-c58f-4c81-83b0-85c237b6bc02","Type":"ContainerDied","Data":"3f968e96612cd183df70e1dd7d444f0df52e0662f3096bb2a9c88837f965d65b"}
Jul 17 22:01:49.824776 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:49.824708    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj"
Jul 17 22:01:49.833569 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:49.833554    2573 scope.go:117] "RemoveContainer" containerID="26c2440b1a73189bda77fd4abb58aca1eac24ba8ecd526d34ceab18898a4c6a9"
Jul 17 22:01:49.851173 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:49.851151    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl"]
Jul 17 22:01:49.861349 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:49.861328    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-kzkgl"]
Jul 17 22:01:49.874414 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:49.874397    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj"]
Jul 17 22:01:49.881870 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:49.881851    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-8njgj"]
Jul 17 22:01:50.529428 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:50.529396    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f668be2-c58f-4c81-83b0-85c237b6bc02" path="/var/lib/kubelet/pods/3f668be2-c58f-4c81-83b0-85c237b6bc02/volumes"
Jul 17 22:01:50.529732 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:01:50.529719    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d84c5df-94c4-4433-87cd-790dfcfe2878" path="/var/lib/kubelet/pods/9d84c5df-94c4-4433-87cd-790dfcfe2878/volumes"
Jul 17 22:02:01.054543 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054502    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6"]
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054789    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d84c5df-94c4-4433-87cd-790dfcfe2878" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054801    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d84c5df-94c4-4433-87cd-790dfcfe2878" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054813    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f668be2-c58f-4c81-83b0-85c237b6bc02" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054819    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f668be2-c58f-4c81-83b0-85c237b6bc02" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054827    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a124b86e-89f7-4554-aef4-dda076584c4b" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054832    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a124b86e-89f7-4554-aef4-dda076584c4b" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054838    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76413919-f253-462a-bebf-65c9c1fec911" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054843    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="76413919-f253-462a-bebf-65c9c1fec911" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054849    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f668be2-c58f-4c81-83b0-85c237b6bc02" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054854    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f668be2-c58f-4c81-83b0-85c237b6bc02" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054867    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76413919-f253-462a-bebf-65c9c1fec911" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054872    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="76413919-f253-462a-bebf-65c9c1fec911" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054879    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a124b86e-89f7-4554-aef4-dda076584c4b" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054883    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a124b86e-89f7-4554-aef4-dda076584c4b" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054890    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d84c5df-94c4-4433-87cd-790dfcfe2878" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054895    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d84c5df-94c4-4433-87cd-790dfcfe2878" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054937    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="76413919-f253-462a-bebf-65c9c1fec911" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054944    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a124b86e-89f7-4554-aef4-dda076584c4b" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054951    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a124b86e-89f7-4554-aef4-dda076584c4b" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054956    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="76413919-f253-462a-bebf-65c9c1fec911" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054962    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d84c5df-94c4-4433-87cd-790dfcfe2878" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054967    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a124b86e-89f7-4554-aef4-dda076584c4b" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054972    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="76413919-f253-462a-bebf-65c9c1fec911" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054977    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d84c5df-94c4-4433-87cd-790dfcfe2878" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054982    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f668be2-c58f-4c81-83b0-85c237b6bc02" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054986    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d84c5df-94c4-4433-87cd-790dfcfe2878" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054992    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f668be2-c58f-4c81-83b0-85c237b6bc02" containerName="revoke-keys"
Jul 17 22:02:01.055026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.054998    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f668be2-c58f-4c81-83b0-85c237b6bc02" containerName="revoke-keys"
Jul 17 22:02:01.058784 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.058768    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6"
Jul 17 22:02:01.061046 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.061026    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:02:01.061046 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.061029    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 22:02:01.064173 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.064148    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6"]
Jul 17 22:02:01.077678 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.077646    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4320171f-2e59-445b-bbf9-bb9e79b3d243-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-q9fk6\" (UID: \"4320171f-2e59-445b-bbf9-bb9e79b3d243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6"
Jul 17 22:02:01.178473 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.178447    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4320171f-2e59-445b-bbf9-bb9e79b3d243-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-q9fk6\" (UID: \"4320171f-2e59-445b-bbf9-bb9e79b3d243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6"
Jul 17 22:02:01.179090 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.179070    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4320171f-2e59-445b-bbf9-bb9e79b3d243-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-q9fk6\" (UID: \"4320171f-2e59-445b-bbf9-bb9e79b3d243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6"
Jul 17 22:02:01.370598 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.370565    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6"
Jul 17 22:02:01.493305 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.493267    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6"]
Jul 17 22:02:01.495296 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:02:01.495261    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4320171f_2e59_445b_bbf9_bb9e79b3d243.slice/crio-17f92409ffe210e1f644df97cd0613091d34018691ef1c365043f8d55159db12 WatchSource:0}: Error finding container 17f92409ffe210e1f644df97cd0613091d34018691ef1c365043f8d55159db12: Status 404 returned error can't find the container with id 17f92409ffe210e1f644df97cd0613091d34018691ef1c365043f8d55159db12
Jul 17 22:02:01.876529 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.876486    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6" event={"ID":"4320171f-2e59-445b-bbf9-bb9e79b3d243","Type":"ContainerStarted","Data":"375383a27cf0b2e855c8f484cf37da5a739727541e10a6ccb0a95df6f5ed7b23"}
Jul 17 22:02:01.876719 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.876569    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6" event={"ID":"4320171f-2e59-445b-bbf9-bb9e79b3d243","Type":"ContainerStarted","Data":"17f92409ffe210e1f644df97cd0613091d34018691ef1c365043f8d55159db12"}
Jul 17 22:02:01.892850 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:01.892796    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6" podStartSLOduration=0.892781753 podStartE2EDuration="892.781753ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:02:01.891041259 +0000 UTC m=+3221.855161870" watchObservedRunningTime="2026-07-17 22:02:01.892781753 +0000 UTC m=+3221.856902363"
Jul 17 22:02:02.882271 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:02.882230    2573 generic.go:356] "Generic (PLEG): container finished" podID="4320171f-2e59-445b-bbf9-bb9e79b3d243" containerID="375383a27cf0b2e855c8f484cf37da5a739727541e10a6ccb0a95df6f5ed7b23" exitCode=6
Jul 17 22:02:02.882692 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:02.882310    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6" event={"ID":"4320171f-2e59-445b-bbf9-bb9e79b3d243","Type":"ContainerDied","Data":"375383a27cf0b2e855c8f484cf37da5a739727541e10a6ccb0a95df6f5ed7b23"}
Jul 17 22:02:02.882692 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:02.882469    2573 scope.go:117] "RemoveContainer" containerID="375383a27cf0b2e855c8f484cf37da5a739727541e10a6ccb0a95df6f5ed7b23"
Jul 17 22:02:03.888430 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:03.888397    2573 generic.go:356] "Generic (PLEG): container finished" podID="4320171f-2e59-445b-bbf9-bb9e79b3d243" containerID="3cb9766437847a6e2f03919f33dd3aab54264e2b38ab9dcdd58dfdad7ae63b8f" exitCode=6
Jul 17 22:02:03.888857 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:03.888480    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6" event={"ID":"4320171f-2e59-445b-bbf9-bb9e79b3d243","Type":"ContainerDied","Data":"3cb9766437847a6e2f03919f33dd3aab54264e2b38ab9dcdd58dfdad7ae63b8f"}
Jul 17 22:02:03.888857 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:03.888545    2573 scope.go:117] "RemoveContainer" containerID="375383a27cf0b2e855c8f484cf37da5a739727541e10a6ccb0a95df6f5ed7b23"
Jul 17 22:02:03.888857 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:03.888691    2573 scope.go:117] "RemoveContainer" containerID="3cb9766437847a6e2f03919f33dd3aab54264e2b38ab9dcdd58dfdad7ae63b8f"
Jul 17 22:02:03.889007 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:02:03.888885    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-q9fk6_odh-ai-gateway-infra(4320171f-2e59-445b-bbf9-bb9e79b3d243)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6" podUID="4320171f-2e59-445b-bbf9-bb9e79b3d243"
Jul 17 22:02:04.894179 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:04.894149    2573 scope.go:117] "RemoveContainer" containerID="3cb9766437847a6e2f03919f33dd3aab54264e2b38ab9dcdd58dfdad7ae63b8f"
Jul 17 22:02:04.894581 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:02:04.894353    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-q9fk6_odh-ai-gateway-infra(4320171f-2e59-445b-bbf9-bb9e79b3d243)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6" podUID="4320171f-2e59-445b-bbf9-bb9e79b3d243"
Jul 17 22:02:15.756125 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:15.756089    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq"]
Jul 17 22:02:15.756511 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:15.756413    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d84c5df-94c4-4433-87cd-790dfcfe2878" containerName="revoke-keys"
Jul 17 22:02:15.756511 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:15.756426    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d84c5df-94c4-4433-87cd-790dfcfe2878" containerName="revoke-keys"
Jul 17 22:02:15.756511 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:15.756437    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a124b86e-89f7-4554-aef4-dda076584c4b" containerName="revoke-keys"
Jul 17 22:02:15.756511 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:15.756442    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a124b86e-89f7-4554-aef4-dda076584c4b" containerName="revoke-keys"
Jul 17 22:02:15.756511 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:15.756449    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76413919-f253-462a-bebf-65c9c1fec911" containerName="revoke-keys"
Jul 17 22:02:15.756511 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:15.756453    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="76413919-f253-462a-bebf-65c9c1fec911" containerName="revoke-keys"
Jul 17 22:02:15.756511 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:15.756460    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f668be2-c58f-4c81-83b0-85c237b6bc02" containerName="revoke-keys"
Jul 17 22:02:15.756511 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:15.756466    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f668be2-c58f-4c81-83b0-85c237b6bc02" containerName="revoke-keys"
Jul 17 22:02:15.760437 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:15.760419    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq"
Jul 17 22:02:15.767605 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:15.767254    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq"]
Jul 17 22:02:15.791002 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:15.790978    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6261f747-b9e9-443f-93ed-6add3a021008-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq\" (UID: \"6261f747-b9e9-443f-93ed-6add3a021008\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq"
Jul 17 22:02:15.891793 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:15.891772    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6261f747-b9e9-443f-93ed-6add3a021008-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq\" (UID: \"6261f747-b9e9-443f-93ed-6add3a021008\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq"
Jul 17 22:02:15.892392 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:15.892374    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6261f747-b9e9-443f-93ed-6add3a021008-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq\" (UID: \"6261f747-b9e9-443f-93ed-6add3a021008\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq"
Jul 17 22:02:16.072721 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:16.072633    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq"
Jul 17 22:02:16.199059 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:16.199033    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq"]
Jul 17 22:02:16.201306 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:02:16.201265    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6261f747_b9e9_443f_93ed_6add3a021008.slice/crio-005bc826c61013b3e978cf64fceebab775dbbb4ca70e06a52a88a3844b55a31e WatchSource:0}: Error finding container 005bc826c61013b3e978cf64fceebab775dbbb4ca70e06a52a88a3844b55a31e: Status 404 returned error can't find the container with id 005bc826c61013b3e978cf64fceebab775dbbb4ca70e06a52a88a3844b55a31e
Jul 17 22:02:16.946373 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:16.946338    2573 generic.go:356] "Generic (PLEG): container finished" podID="6261f747-b9e9-443f-93ed-6add3a021008" containerID="6ff79fd818fe13298491556b943789fb2c4bf1f94acc4c037cecc6bbd8e81af0" exitCode=6
Jul 17 22:02:16.946721 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:16.946397    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq" event={"ID":"6261f747-b9e9-443f-93ed-6add3a021008","Type":"ContainerDied","Data":"6ff79fd818fe13298491556b943789fb2c4bf1f94acc4c037cecc6bbd8e81af0"}
Jul 17 22:02:16.946721 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:16.946460    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq" event={"ID":"6261f747-b9e9-443f-93ed-6add3a021008","Type":"ContainerStarted","Data":"005bc826c61013b3e978cf64fceebab775dbbb4ca70e06a52a88a3844b55a31e"}
Jul 17 22:02:16.946721 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:16.946555    2573 scope.go:117] "RemoveContainer" containerID="6ff79fd818fe13298491556b943789fb2c4bf1f94acc4c037cecc6bbd8e81af0"
Jul 17 22:02:17.951749 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:17.951712    2573 generic.go:356] "Generic (PLEG): container finished" podID="6261f747-b9e9-443f-93ed-6add3a021008" containerID="ebb45e01f0325f727a2755898d0aad7eb148984667169fdda10b04be91bb121e" exitCode=6
Jul 17 22:02:17.952182 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:17.951792    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq" event={"ID":"6261f747-b9e9-443f-93ed-6add3a021008","Type":"ContainerDied","Data":"ebb45e01f0325f727a2755898d0aad7eb148984667169fdda10b04be91bb121e"}
Jul 17 22:02:17.952182 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:17.951850    2573 scope.go:117] "RemoveContainer" containerID="6ff79fd818fe13298491556b943789fb2c4bf1f94acc4c037cecc6bbd8e81af0"
Jul 17 22:02:17.952182 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:17.952030    2573 scope.go:117] "RemoveContainer" containerID="ebb45e01f0325f727a2755898d0aad7eb148984667169fdda10b04be91bb121e"
Jul 17 22:02:17.952318 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:02:17.952224    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq_odh-ai-gateway-infra(6261f747-b9e9-443f-93ed-6add3a021008)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq" podUID="6261f747-b9e9-443f-93ed-6add3a021008"
Jul 17 22:02:18.958623 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:18.958594    2573 scope.go:117] "RemoveContainer" containerID="ebb45e01f0325f727a2755898d0aad7eb148984667169fdda10b04be91bb121e"
Jul 17 22:02:18.959006 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:02:18.958776    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq_odh-ai-gateway-infra(6261f747-b9e9-443f-93ed-6add3a021008)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq" podUID="6261f747-b9e9-443f-93ed-6add3a021008"
Jul 17 22:02:19.525175 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:19.525148    2573 scope.go:117] "RemoveContainer" containerID="3cb9766437847a6e2f03919f33dd3aab54264e2b38ab9dcdd58dfdad7ae63b8f"
Jul 17 22:02:19.952140 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:19.952113    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j"]
Jul 17 22:02:19.955274 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:19.955257    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j"
Jul 17 22:02:19.964175 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:19.964152    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j"]
Jul 17 22:02:19.966072 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:19.966051    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6" event={"ID":"4320171f-2e59-445b-bbf9-bb9e79b3d243","Type":"ContainerStarted","Data":"1bb685a17dfd20da753c6b6985bc36a3f9d759c27e641e952fb98292a22069b8"}
Jul 17 22:02:20.029539 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:20.029507    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59b8c9e-842c-45cc-9927-bdd8b31da7a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j\" (UID: \"e59b8c9e-842c-45cc-9927-bdd8b31da7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j"
Jul 17 22:02:20.130545 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:20.130518    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59b8c9e-842c-45cc-9927-bdd8b31da7a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j\" (UID: \"e59b8c9e-842c-45cc-9927-bdd8b31da7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j"
Jul 17 22:02:20.131152 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:20.131131    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59b8c9e-842c-45cc-9927-bdd8b31da7a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j\" (UID: \"e59b8c9e-842c-45cc-9927-bdd8b31da7a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j"
Jul 17 22:02:20.266077 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:20.266009    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j"
Jul 17 22:02:20.388063 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:20.388041    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j"]
Jul 17 22:02:20.390576 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:02:20.390548    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59b8c9e_842c_45cc_9927_bdd8b31da7a6.slice/crio-6a0f75229f744b398e0e89717c9d86298bfabe70aecebc8ecda9a48124a3bc1d WatchSource:0}: Error finding container 6a0f75229f744b398e0e89717c9d86298bfabe70aecebc8ecda9a48124a3bc1d: Status 404 returned error can't find the container with id 6a0f75229f744b398e0e89717c9d86298bfabe70aecebc8ecda9a48124a3bc1d
Jul 17 22:02:20.972002 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:20.971968    2573 generic.go:356] "Generic (PLEG): container finished" podID="4320171f-2e59-445b-bbf9-bb9e79b3d243" containerID="1bb685a17dfd20da753c6b6985bc36a3f9d759c27e641e952fb98292a22069b8" exitCode=6
Jul 17 22:02:20.972328 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:20.972039    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6" event={"ID":"4320171f-2e59-445b-bbf9-bb9e79b3d243","Type":"ContainerDied","Data":"1bb685a17dfd20da753c6b6985bc36a3f9d759c27e641e952fb98292a22069b8"}
Jul 17 22:02:20.972328 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:20.972112    2573 scope.go:117] "RemoveContainer" containerID="3cb9766437847a6e2f03919f33dd3aab54264e2b38ab9dcdd58dfdad7ae63b8f"
Jul 17 22:02:20.972450 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:20.972391    2573 scope.go:117] "RemoveContainer" containerID="1bb685a17dfd20da753c6b6985bc36a3f9d759c27e641e952fb98292a22069b8"
Jul 17 22:02:20.972639 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:02:20.972612    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-q9fk6_odh-ai-gateway-infra(4320171f-2e59-445b-bbf9-bb9e79b3d243)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6" podUID="4320171f-2e59-445b-bbf9-bb9e79b3d243"
Jul 17 22:02:20.973645 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:20.973622    2573 generic.go:356] "Generic (PLEG): container finished" podID="e59b8c9e-842c-45cc-9927-bdd8b31da7a6" containerID="ef0874b4a0a2dcacc40ae42729fb5e67b9786536afda58040c07aa3388f1ba89" exitCode=6
Jul 17 22:02:20.973720 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:20.973654    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j" event={"ID":"e59b8c9e-842c-45cc-9927-bdd8b31da7a6","Type":"ContainerDied","Data":"ef0874b4a0a2dcacc40ae42729fb5e67b9786536afda58040c07aa3388f1ba89"}
Jul 17 22:02:20.973720 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:20.973691    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j" event={"ID":"e59b8c9e-842c-45cc-9927-bdd8b31da7a6","Type":"ContainerStarted","Data":"6a0f75229f744b398e0e89717c9d86298bfabe70aecebc8ecda9a48124a3bc1d"}
Jul 17 22:02:20.973841 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:20.973829    2573 scope.go:117] "RemoveContainer" containerID="ef0874b4a0a2dcacc40ae42729fb5e67b9786536afda58040c07aa3388f1ba89"
Jul 17 22:02:20.998863 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:20.998841    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6"]
Jul 17 22:02:21.981521 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:21.981490    2573 generic.go:356] "Generic (PLEG): container finished" podID="e59b8c9e-842c-45cc-9927-bdd8b31da7a6" containerID="0aab564370f6bdc7ee2cad609f67938325efd993977f5836521c4fdb57eab8d4" exitCode=6
Jul 17 22:02:21.981886 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:21.981566    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j" event={"ID":"e59b8c9e-842c-45cc-9927-bdd8b31da7a6","Type":"ContainerDied","Data":"0aab564370f6bdc7ee2cad609f67938325efd993977f5836521c4fdb57eab8d4"}
Jul 17 22:02:21.981886 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:21.981625    2573 scope.go:117] "RemoveContainer" containerID="ef0874b4a0a2dcacc40ae42729fb5e67b9786536afda58040c07aa3388f1ba89"
Jul 17 22:02:21.981886 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:21.981729    2573 scope.go:117] "RemoveContainer" containerID="0aab564370f6bdc7ee2cad609f67938325efd993977f5836521c4fdb57eab8d4"
Jul 17 22:02:21.982019 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:02:21.981894    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j_odh-ai-gateway-infra(e59b8c9e-842c-45cc-9927-bdd8b31da7a6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j" podUID="e59b8c9e-842c-45cc-9927-bdd8b31da7a6"
Jul 17 22:02:22.115155 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:22.115129    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6"
Jul 17 22:02:22.247405 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:22.247333    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4320171f-2e59-445b-bbf9-bb9e79b3d243-maas-api-service-ca\") pod \"4320171f-2e59-445b-bbf9-bb9e79b3d243\" (UID: \"4320171f-2e59-445b-bbf9-bb9e79b3d243\") "
Jul 17 22:02:22.247680 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:22.247660    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4320171f-2e59-445b-bbf9-bb9e79b3d243-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4320171f-2e59-445b-bbf9-bb9e79b3d243" (UID: "4320171f-2e59-445b-bbf9-bb9e79b3d243"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:02:22.348077 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:22.348049    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4320171f-2e59-445b-bbf9-bb9e79b3d243-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:02:22.986702 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:22.986670    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6"
Jul 17 22:02:22.987140 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:22.986673    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6" event={"ID":"4320171f-2e59-445b-bbf9-bb9e79b3d243","Type":"ContainerDied","Data":"17f92409ffe210e1f644df97cd0613091d34018691ef1c365043f8d55159db12"}
Jul 17 22:02:22.987140 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:22.986806    2573 scope.go:117] "RemoveContainer" containerID="1bb685a17dfd20da753c6b6985bc36a3f9d759c27e641e952fb98292a22069b8"
Jul 17 22:02:22.988661 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:22.988638    2573 scope.go:117] "RemoveContainer" containerID="0aab564370f6bdc7ee2cad609f67938325efd993977f5836521c4fdb57eab8d4"
Jul 17 22:02:22.988873 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:02:22.988845    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j_odh-ai-gateway-infra(e59b8c9e-842c-45cc-9927-bdd8b31da7a6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j" podUID="e59b8c9e-842c-45cc-9927-bdd8b31da7a6"
Jul 17 22:02:23.015614 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:23.015592    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6"]
Jul 17 22:02:23.019855 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:23.019835    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-q9fk6"]
Jul 17 22:02:24.529364 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:24.529329    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4320171f-2e59-445b-bbf9-bb9e79b3d243" path="/var/lib/kubelet/pods/4320171f-2e59-445b-bbf9-bb9e79b3d243/volumes"
Jul 17 22:02:28.005392 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:28.005357    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb"]
Jul 17 22:02:28.005738 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:28.005654    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4320171f-2e59-445b-bbf9-bb9e79b3d243" containerName="revoke-keys"
Jul 17 22:02:28.005738 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:28.005668    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4320171f-2e59-445b-bbf9-bb9e79b3d243" containerName="revoke-keys"
Jul 17 22:02:28.005738 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:28.005683    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4320171f-2e59-445b-bbf9-bb9e79b3d243" containerName="revoke-keys"
Jul 17 22:02:28.005738 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:28.005689    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4320171f-2e59-445b-bbf9-bb9e79b3d243" containerName="revoke-keys"
Jul 17 22:02:28.005738 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:28.005707    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4320171f-2e59-445b-bbf9-bb9e79b3d243" containerName="revoke-keys"
Jul 17 22:02:28.005738 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:28.005712    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4320171f-2e59-445b-bbf9-bb9e79b3d243" containerName="revoke-keys"
Jul 17 22:02:28.005945 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:28.005763    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4320171f-2e59-445b-bbf9-bb9e79b3d243" containerName="revoke-keys"
Jul 17 22:02:28.005945 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:28.005775    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4320171f-2e59-445b-bbf9-bb9e79b3d243" containerName="revoke-keys"
Jul 17 22:02:28.010088 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:28.010062    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb"
Jul 17 22:02:28.015301 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:28.015258    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb"]
Jul 17 22:02:28.090528 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:28.090500    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a60b0b1-8379-4284-925f-03468b564cb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb\" (UID: \"0a60b0b1-8379-4284-925f-03468b564cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb"
Jul 17 22:02:28.191130 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:28.191108    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a60b0b1-8379-4284-925f-03468b564cb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb\" (UID: \"0a60b0b1-8379-4284-925f-03468b564cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb"
Jul 17 22:02:28.191705 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:28.191688    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a60b0b1-8379-4284-925f-03468b564cb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb\" (UID: \"0a60b0b1-8379-4284-925f-03468b564cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb"
Jul 17 22:02:28.321396 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:28.321332    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb"
Jul 17 22:02:28.443700 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:28.443676    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb"]
Jul 17 22:02:28.445692 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:02:28.445665    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a60b0b1_8379_4284_925f_03468b564cb8.slice/crio-989e85dee25b7edc65e0a11828e616cb15c05f16e3317801c1ca16423f1e308b WatchSource:0}: Error finding container 989e85dee25b7edc65e0a11828e616cb15c05f16e3317801c1ca16423f1e308b: Status 404 returned error can't find the container with id 989e85dee25b7edc65e0a11828e616cb15c05f16e3317801c1ca16423f1e308b
Jul 17 22:02:29.020238 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:29.020145    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb" event={"ID":"0a60b0b1-8379-4284-925f-03468b564cb8","Type":"ContainerStarted","Data":"ae7a914f6b8ce399a41a7dab2c250e189fb5f8a4c615512fa7d493b2d1c01c3d"}
Jul 17 22:02:29.020238 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:29.020200    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb" event={"ID":"0a60b0b1-8379-4284-925f-03468b564cb8","Type":"ContainerStarted","Data":"989e85dee25b7edc65e0a11828e616cb15c05f16e3317801c1ca16423f1e308b"}
Jul 17 22:02:29.035701 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:29.035658    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb" podStartSLOduration=2.035643004 podStartE2EDuration="2.035643004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:02:29.034005571 +0000 UTC m=+3248.998126182" watchObservedRunningTime="2026-07-17 22:02:29.035643004 +0000 UTC m=+3248.999763613"
Jul 17 22:02:30.027027 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:30.026991    2573 generic.go:356] "Generic (PLEG): container finished" podID="0a60b0b1-8379-4284-925f-03468b564cb8" containerID="ae7a914f6b8ce399a41a7dab2c250e189fb5f8a4c615512fa7d493b2d1c01c3d" exitCode=6
Jul 17 22:02:30.027536 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:30.027073    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb" event={"ID":"0a60b0b1-8379-4284-925f-03468b564cb8","Type":"ContainerDied","Data":"ae7a914f6b8ce399a41a7dab2c250e189fb5f8a4c615512fa7d493b2d1c01c3d"}
Jul 17 22:02:30.027536 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:30.027257    2573 scope.go:117] "RemoveContainer" containerID="ae7a914f6b8ce399a41a7dab2c250e189fb5f8a4c615512fa7d493b2d1c01c3d"
Jul 17 22:02:30.526903 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:30.526868    2573 scope.go:117] "RemoveContainer" containerID="ebb45e01f0325f727a2755898d0aad7eb148984667169fdda10b04be91bb121e"
Jul 17 22:02:31.033913 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:31.033815    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq" event={"ID":"6261f747-b9e9-443f-93ed-6add3a021008","Type":"ContainerStarted","Data":"be5a481954f8ea48ab99b4a10b2cbf0dd969e32d1ba170cda62950302ea2c9d3"}
Jul 17 22:02:31.035563 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:31.035540    2573 generic.go:356] "Generic (PLEG): container finished" podID="0a60b0b1-8379-4284-925f-03468b564cb8" containerID="362974e2a751d8ace2242943321921b3c99d7c1a516ef8b8d311199f5d7c5151" exitCode=6
Jul 17 22:02:31.035690 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:31.035616    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb" event={"ID":"0a60b0b1-8379-4284-925f-03468b564cb8","Type":"ContainerDied","Data":"362974e2a751d8ace2242943321921b3c99d7c1a516ef8b8d311199f5d7c5151"}
Jul 17 22:02:31.035690 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:31.035669    2573 scope.go:117] "RemoveContainer" containerID="ae7a914f6b8ce399a41a7dab2c250e189fb5f8a4c615512fa7d493b2d1c01c3d"
Jul 17 22:02:31.035765 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:31.035732    2573 scope.go:117] "RemoveContainer" containerID="362974e2a751d8ace2242943321921b3c99d7c1a516ef8b8d311199f5d7c5151"
Jul 17 22:02:31.035945 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:02:31.035926    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb_odh-ai-gateway-infra(0a60b0b1-8379-4284-925f-03468b564cb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb" podUID="0a60b0b1-8379-4284-925f-03468b564cb8"
Jul 17 22:02:31.052537 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:31.052501    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq" podStartSLOduration=16.052488583 podStartE2EDuration="16.052488583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:02:31.051474939 +0000 UTC m=+3251.015595549" watchObservedRunningTime="2026-07-17 22:02:31.052488583 +0000 UTC m=+3251.016609194"
Jul 17 22:02:32.042181 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:32.042148    2573 generic.go:356] "Generic (PLEG): container finished" podID="6261f747-b9e9-443f-93ed-6add3a021008" containerID="be5a481954f8ea48ab99b4a10b2cbf0dd969e32d1ba170cda62950302ea2c9d3" exitCode=6
Jul 17 22:02:32.042691 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:32.042226    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq" event={"ID":"6261f747-b9e9-443f-93ed-6add3a021008","Type":"ContainerDied","Data":"be5a481954f8ea48ab99b4a10b2cbf0dd969e32d1ba170cda62950302ea2c9d3"}
Jul 17 22:02:32.042691 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:32.042304    2573 scope.go:117] "RemoveContainer" containerID="ebb45e01f0325f727a2755898d0aad7eb148984667169fdda10b04be91bb121e"
Jul 17 22:02:32.042691 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:32.042605    2573 scope.go:117] "RemoveContainer" containerID="be5a481954f8ea48ab99b4a10b2cbf0dd969e32d1ba170cda62950302ea2c9d3"
Jul 17 22:02:32.042851 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:02:32.042827    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq_odh-ai-gateway-infra(6261f747-b9e9-443f-93ed-6add3a021008)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq" podUID="6261f747-b9e9-443f-93ed-6add3a021008"
Jul 17 22:02:32.044071 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:32.044056    2573 scope.go:117] "RemoveContainer" containerID="362974e2a751d8ace2242943321921b3c99d7c1a516ef8b8d311199f5d7c5151"
Jul 17 22:02:32.044202 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:02:32.044187    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb_odh-ai-gateway-infra(0a60b0b1-8379-4284-925f-03468b564cb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb" podUID="0a60b0b1-8379-4284-925f-03468b564cb8"
Jul 17 22:02:32.067683 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:32.067655    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq"]
Jul 17 22:02:33.182671 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:33.182650    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq"
Jul 17 22:02:33.232137 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:33.232110    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6261f747-b9e9-443f-93ed-6add3a021008-maas-api-service-ca\") pod \"6261f747-b9e9-443f-93ed-6add3a021008\" (UID: \"6261f747-b9e9-443f-93ed-6add3a021008\") "
Jul 17 22:02:33.232500 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:33.232476    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6261f747-b9e9-443f-93ed-6add3a021008-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6261f747-b9e9-443f-93ed-6add3a021008" (UID: "6261f747-b9e9-443f-93ed-6add3a021008"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:02:33.333505 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:33.333431    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6261f747-b9e9-443f-93ed-6add3a021008-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:02:34.055391 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:34.055360    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq"
Jul 17 22:02:34.055588 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:34.055404    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq" event={"ID":"6261f747-b9e9-443f-93ed-6add3a021008","Type":"ContainerDied","Data":"005bc826c61013b3e978cf64fceebab775dbbb4ca70e06a52a88a3844b55a31e"}
Jul 17 22:02:34.055588 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:34.055471    2573 scope.go:117] "RemoveContainer" containerID="be5a481954f8ea48ab99b4a10b2cbf0dd969e32d1ba170cda62950302ea2c9d3"
Jul 17 22:02:34.078726 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:34.078696    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq"]
Jul 17 22:02:34.082701 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:34.082676    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-95mhq"]
Jul 17 22:02:34.529521 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:34.529490    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6261f747-b9e9-443f-93ed-6add3a021008" path="/var/lib/kubelet/pods/6261f747-b9e9-443f-93ed-6add3a021008/volumes"
Jul 17 22:02:37.525569 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:37.525458    2573 scope.go:117] "RemoveContainer" containerID="0aab564370f6bdc7ee2cad609f67938325efd993977f5836521c4fdb57eab8d4"
Jul 17 22:02:38.074885 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:38.074853    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j" event={"ID":"e59b8c9e-842c-45cc-9927-bdd8b31da7a6","Type":"ContainerStarted","Data":"055770b4c62663d3598ff79f6b90b39df6e93db1dc7df464c8e64b40ecd2ae72"}
Jul 17 22:02:38.091215 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:38.091167    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j" podStartSLOduration=19.09115278 podStartE2EDuration="19.09115278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:02:38.089757464 +0000 UTC m=+3258.053878074" watchObservedRunningTime="2026-07-17 22:02:38.09115278 +0000 UTC m=+3258.055273391"
Jul 17 22:02:39.081272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:39.081240    2573 generic.go:356] "Generic (PLEG): container finished" podID="e59b8c9e-842c-45cc-9927-bdd8b31da7a6" containerID="055770b4c62663d3598ff79f6b90b39df6e93db1dc7df464c8e64b40ecd2ae72" exitCode=6
Jul 17 22:02:39.081694 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:39.081309    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j" event={"ID":"e59b8c9e-842c-45cc-9927-bdd8b31da7a6","Type":"ContainerDied","Data":"055770b4c62663d3598ff79f6b90b39df6e93db1dc7df464c8e64b40ecd2ae72"}
Jul 17 22:02:39.081694 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:39.081362    2573 scope.go:117] "RemoveContainer" containerID="0aab564370f6bdc7ee2cad609f67938325efd993977f5836521c4fdb57eab8d4"
Jul 17 22:02:39.081694 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:39.081620    2573 scope.go:117] "RemoveContainer" containerID="055770b4c62663d3598ff79f6b90b39df6e93db1dc7df464c8e64b40ecd2ae72"
Jul 17 22:02:39.081859 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:02:39.081837    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j_odh-ai-gateway-infra(e59b8c9e-842c-45cc-9927-bdd8b31da7a6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j" podUID="e59b8c9e-842c-45cc-9927-bdd8b31da7a6"
Jul 17 22:02:39.104845 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:39.104820    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j"]
Jul 17 22:02:40.209417 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:40.209394    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j"
Jul 17 22:02:40.286489 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:40.286463    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59b8c9e-842c-45cc-9927-bdd8b31da7a6-maas-api-service-ca\") pod \"e59b8c9e-842c-45cc-9927-bdd8b31da7a6\" (UID: \"e59b8c9e-842c-45cc-9927-bdd8b31da7a6\") "
Jul 17 22:02:40.286817 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:40.286798    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59b8c9e-842c-45cc-9927-bdd8b31da7a6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e59b8c9e-842c-45cc-9927-bdd8b31da7a6" (UID: "e59b8c9e-842c-45cc-9927-bdd8b31da7a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:02:40.387325 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:40.387268    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e59b8c9e-842c-45cc-9927-bdd8b31da7a6-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:02:41.094942 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:41.094910    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j"
Jul 17 22:02:41.095114 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:41.094943    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j" event={"ID":"e59b8c9e-842c-45cc-9927-bdd8b31da7a6","Type":"ContainerDied","Data":"6a0f75229f744b398e0e89717c9d86298bfabe70aecebc8ecda9a48124a3bc1d"}
Jul 17 22:02:41.095114 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:41.095005    2573 scope.go:117] "RemoveContainer" containerID="055770b4c62663d3598ff79f6b90b39df6e93db1dc7df464c8e64b40ecd2ae72"
Jul 17 22:02:41.109339 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:41.109315    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j"]
Jul 17 22:02:41.116938 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:41.116917    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-5s79j"]
Jul 17 22:02:42.529026 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:42.528994    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e59b8c9e-842c-45cc-9927-bdd8b31da7a6" path="/var/lib/kubelet/pods/e59b8c9e-842c-45cc-9927-bdd8b31da7a6/volumes"
Jul 17 22:02:47.525232 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:47.525205    2573 scope.go:117] "RemoveContainer" containerID="362974e2a751d8ace2242943321921b3c99d7c1a516ef8b8d311199f5d7c5151"
Jul 17 22:02:48.128672 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:48.128643    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb" event={"ID":"0a60b0b1-8379-4284-925f-03468b564cb8","Type":"ContainerStarted","Data":"f93314416a0ef16aef3da46549ea937d0198e9fa82153854606f1fd7b0db1a3a"}
Jul 17 22:02:49.134111 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:49.134083    2573 generic.go:356] "Generic (PLEG): container finished" podID="0a60b0b1-8379-4284-925f-03468b564cb8" containerID="f93314416a0ef16aef3da46549ea937d0198e9fa82153854606f1fd7b0db1a3a" exitCode=6
Jul 17 22:02:49.134111 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:49.134116    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb" event={"ID":"0a60b0b1-8379-4284-925f-03468b564cb8","Type":"ContainerDied","Data":"f93314416a0ef16aef3da46549ea937d0198e9fa82153854606f1fd7b0db1a3a"}
Jul 17 22:02:49.134589 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:49.134161    2573 scope.go:117] "RemoveContainer" containerID="362974e2a751d8ace2242943321921b3c99d7c1a516ef8b8d311199f5d7c5151"
Jul 17 22:02:49.134589 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:49.134421    2573 scope.go:117] "RemoveContainer" containerID="f93314416a0ef16aef3da46549ea937d0198e9fa82153854606f1fd7b0db1a3a"
Jul 17 22:02:49.134680 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:02:49.134644    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb_odh-ai-gateway-infra(0a60b0b1-8379-4284-925f-03468b564cb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb" podUID="0a60b0b1-8379-4284-925f-03468b564cb8"
Jul 17 22:02:49.154035 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:49.154006    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb"]
Jul 17 22:02:50.268988 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:50.268964    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb"
Jul 17 22:02:50.362523 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:50.362489    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a60b0b1-8379-4284-925f-03468b564cb8-maas-api-service-ca\") pod \"0a60b0b1-8379-4284-925f-03468b564cb8\" (UID: \"0a60b0b1-8379-4284-925f-03468b564cb8\") "
Jul 17 22:02:50.362865 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:50.362839    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a60b0b1-8379-4284-925f-03468b564cb8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a60b0b1-8379-4284-925f-03468b564cb8" (UID: "0a60b0b1-8379-4284-925f-03468b564cb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:02:50.463645 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:50.463570    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a60b0b1-8379-4284-925f-03468b564cb8-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:02:51.145619 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:51.145584    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb"
Jul 17 22:02:51.145797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:51.145618    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb" event={"ID":"0a60b0b1-8379-4284-925f-03468b564cb8","Type":"ContainerDied","Data":"989e85dee25b7edc65e0a11828e616cb15c05f16e3317801c1ca16423f1e308b"}
Jul 17 22:02:51.145797 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:51.145681    2573 scope.go:117] "RemoveContainer" containerID="f93314416a0ef16aef3da46549ea937d0198e9fa82153854606f1fd7b0db1a3a"
Jul 17 22:02:51.161320 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:51.161276    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb"]
Jul 17 22:02:51.167113 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:51.167089    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dmfzb"]
Jul 17 22:02:52.529817 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:02:52.529781    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a60b0b1-8379-4284-925f-03468b564cb8" path="/var/lib/kubelet/pods/0a60b0b1-8379-4284-925f-03468b564cb8/volumes"
Jul 17 22:03:01.099137 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099100    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg"]
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099389    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a60b0b1-8379-4284-925f-03468b564cb8" containerName="revoke-keys"
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099400    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a60b0b1-8379-4284-925f-03468b564cb8" containerName="revoke-keys"
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099408    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e59b8c9e-842c-45cc-9927-bdd8b31da7a6" containerName="revoke-keys"
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099414    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e59b8c9e-842c-45cc-9927-bdd8b31da7a6" containerName="revoke-keys"
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099422    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6261f747-b9e9-443f-93ed-6add3a021008" containerName="revoke-keys"
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099427    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6261f747-b9e9-443f-93ed-6add3a021008" containerName="revoke-keys"
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099433    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6261f747-b9e9-443f-93ed-6add3a021008" containerName="revoke-keys"
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099438    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6261f747-b9e9-443f-93ed-6add3a021008" containerName="revoke-keys"
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099444    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a60b0b1-8379-4284-925f-03468b564cb8" containerName="revoke-keys"
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099449    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a60b0b1-8379-4284-925f-03468b564cb8" containerName="revoke-keys"
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099459    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e59b8c9e-842c-45cc-9927-bdd8b31da7a6" containerName="revoke-keys"
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099464    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e59b8c9e-842c-45cc-9927-bdd8b31da7a6" containerName="revoke-keys"
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099470    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6261f747-b9e9-443f-93ed-6add3a021008" containerName="revoke-keys"
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099474    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6261f747-b9e9-443f-93ed-6add3a021008" containerName="revoke-keys"
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099481    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e59b8c9e-842c-45cc-9927-bdd8b31da7a6" containerName="revoke-keys"
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099486    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e59b8c9e-842c-45cc-9927-bdd8b31da7a6" containerName="revoke-keys"
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099544    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e59b8c9e-842c-45cc-9927-bdd8b31da7a6" containerName="revoke-keys"
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099551    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a60b0b1-8379-4284-925f-03468b564cb8" containerName="revoke-keys"
Jul 17 22:03:01.099553 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099557    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a60b0b1-8379-4284-925f-03468b564cb8" containerName="revoke-keys"
Jul 17 22:03:01.100126 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099566    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6261f747-b9e9-443f-93ed-6add3a021008" containerName="revoke-keys"
Jul 17 22:03:01.100126 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099571    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6261f747-b9e9-443f-93ed-6add3a021008" containerName="revoke-keys"
Jul 17 22:03:01.100126 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099578    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4320171f-2e59-445b-bbf9-bb9e79b3d243" containerName="revoke-keys"
Jul 17 22:03:01.100126 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.099584    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e59b8c9e-842c-45cc-9927-bdd8b31da7a6" containerName="revoke-keys"
Jul 17 22:03:01.103665 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.103646    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg"
Jul 17 22:03:01.105808 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.105791    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:03:01.105924 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.105791    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 22:03:01.109302 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.109263    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg"]
Jul 17 22:03:01.242604 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.242574    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b5537d-2b77-409c-b459-fbc2b8cc9ca7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-dbfxg\" (UID: \"02b5537d-2b77-409c-b459-fbc2b8cc9ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg"
Jul 17 22:03:01.342932 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.342907    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b5537d-2b77-409c-b459-fbc2b8cc9ca7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-dbfxg\" (UID: \"02b5537d-2b77-409c-b459-fbc2b8cc9ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg"
Jul 17 22:03:01.343683 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.343662    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b5537d-2b77-409c-b459-fbc2b8cc9ca7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-dbfxg\" (UID: \"02b5537d-2b77-409c-b459-fbc2b8cc9ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg"
Jul 17 22:03:01.415200 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.415137    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg"
Jul 17 22:03:01.537999 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:01.537977    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg"]
Jul 17 22:03:01.540386 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:03:01.540361    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b5537d_2b77_409c_b459_fbc2b8cc9ca7.slice/crio-221c5d555543c5bf019eeeb9a8354ee819070f055a08f50eadd8444408e824ff WatchSource:0}: Error finding container 221c5d555543c5bf019eeeb9a8354ee819070f055a08f50eadd8444408e824ff: Status 404 returned error can't find the container with id 221c5d555543c5bf019eeeb9a8354ee819070f055a08f50eadd8444408e824ff
Jul 17 22:03:02.195381 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:02.195339    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg" event={"ID":"02b5537d-2b77-409c-b459-fbc2b8cc9ca7","Type":"ContainerStarted","Data":"2abda06bb753c40102c556ec6387b70ba089a396cb292bff5a4aeef192073842"}
Jul 17 22:03:02.195753 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:02.195400    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg" event={"ID":"02b5537d-2b77-409c-b459-fbc2b8cc9ca7","Type":"ContainerStarted","Data":"221c5d555543c5bf019eeeb9a8354ee819070f055a08f50eadd8444408e824ff"}
Jul 17 22:03:02.210428 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:02.210383    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg" podStartSLOduration=1.210368681 podStartE2EDuration="1.210368681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:03:02.209063697 +0000 UTC m=+3282.173184308" watchObservedRunningTime="2026-07-17 22:03:02.210368681 +0000 UTC m=+3282.174489290"
Jul 17 22:03:03.200503 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:03.200468    2573 generic.go:356] "Generic (PLEG): container finished" podID="02b5537d-2b77-409c-b459-fbc2b8cc9ca7" containerID="2abda06bb753c40102c556ec6387b70ba089a396cb292bff5a4aeef192073842" exitCode=6
Jul 17 22:03:03.200944 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:03.200559    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg" event={"ID":"02b5537d-2b77-409c-b459-fbc2b8cc9ca7","Type":"ContainerDied","Data":"2abda06bb753c40102c556ec6387b70ba089a396cb292bff5a4aeef192073842"}
Jul 17 22:03:03.200944 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:03.200750    2573 scope.go:117] "RemoveContainer" containerID="2abda06bb753c40102c556ec6387b70ba089a396cb292bff5a4aeef192073842"
Jul 17 22:03:04.211578 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:04.211547    2573 generic.go:356] "Generic (PLEG): container finished" podID="02b5537d-2b77-409c-b459-fbc2b8cc9ca7" containerID="4bfac91a044d6afdc868639b2a9cd5841fbb35e941eb8b357fe2116c43930326" exitCode=6
Jul 17 22:03:04.212057 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:04.211636    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg" event={"ID":"02b5537d-2b77-409c-b459-fbc2b8cc9ca7","Type":"ContainerDied","Data":"4bfac91a044d6afdc868639b2a9cd5841fbb35e941eb8b357fe2116c43930326"}
Jul 17 22:03:04.212057 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:04.211734    2573 scope.go:117] "RemoveContainer" containerID="2abda06bb753c40102c556ec6387b70ba089a396cb292bff5a4aeef192073842"
Jul 17 22:03:04.212057 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:04.211812    2573 scope.go:117] "RemoveContainer" containerID="4bfac91a044d6afdc868639b2a9cd5841fbb35e941eb8b357fe2116c43930326"
Jul 17 22:03:04.212057 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:03:04.211980    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-dbfxg_odh-ai-gateway-infra(02b5537d-2b77-409c-b459-fbc2b8cc9ca7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg" podUID="02b5537d-2b77-409c-b459-fbc2b8cc9ca7"
Jul 17 22:03:05.218066 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:05.218032    2573 scope.go:117] "RemoveContainer" containerID="4bfac91a044d6afdc868639b2a9cd5841fbb35e941eb8b357fe2116c43930326"
Jul 17 22:03:05.218479 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:03:05.218223    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-dbfxg_odh-ai-gateway-infra(02b5537d-2b77-409c-b459-fbc2b8cc9ca7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg" podUID="02b5537d-2b77-409c-b459-fbc2b8cc9ca7"
Jul 17 22:03:05.800547 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:05.800512    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw"]
Jul 17 22:03:05.800823 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:05.800810    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a60b0b1-8379-4284-925f-03468b564cb8" containerName="revoke-keys"
Jul 17 22:03:05.800873 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:05.800825    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a60b0b1-8379-4284-925f-03468b564cb8" containerName="revoke-keys"
Jul 17 22:03:05.800908 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:05.800890    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e59b8c9e-842c-45cc-9927-bdd8b31da7a6" containerName="revoke-keys"
Jul 17 22:03:05.800908 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:05.800896    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a60b0b1-8379-4284-925f-03468b564cb8" containerName="revoke-keys"
Jul 17 22:03:05.800908 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:05.800904    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6261f747-b9e9-443f-93ed-6add3a021008" containerName="revoke-keys"
Jul 17 22:03:05.804957 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:05.804941    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw"
Jul 17 22:03:05.810460 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:05.810419    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw"]
Jul 17 22:03:05.982200 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:05.982173    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b1e0034-dbab-4300-b374-2ea283b0f18d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw\" (UID: \"1b1e0034-dbab-4300-b374-2ea283b0f18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw"
Jul 17 22:03:06.082530 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:06.082457    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b1e0034-dbab-4300-b374-2ea283b0f18d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw\" (UID: \"1b1e0034-dbab-4300-b374-2ea283b0f18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw"
Jul 17 22:03:06.083086 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:06.083063    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b1e0034-dbab-4300-b374-2ea283b0f18d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw\" (UID: \"1b1e0034-dbab-4300-b374-2ea283b0f18d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw"
Jul 17 22:03:06.116811 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:06.116790    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw"
Jul 17 22:03:06.267639 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:06.267615    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw"]
Jul 17 22:03:06.269840 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:03:06.269813    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1e0034_dbab_4300_b374_2ea283b0f18d.slice/crio-0eb2893a50e33cf834d3607f3ae16feca551d46802251213fecb196ea38a1bbf WatchSource:0}: Error finding container 0eb2893a50e33cf834d3607f3ae16feca551d46802251213fecb196ea38a1bbf: Status 404 returned error can't find the container with id 0eb2893a50e33cf834d3607f3ae16feca551d46802251213fecb196ea38a1bbf
Jul 17 22:03:07.228419 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:07.228337    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b1e0034-dbab-4300-b374-2ea283b0f18d" containerID="8e490b3b82617dcc3a632548c36871f2741347b67c715d45f1673e22e787bf2f" exitCode=6
Jul 17 22:03:07.228562 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:07.228431    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw" event={"ID":"1b1e0034-dbab-4300-b374-2ea283b0f18d","Type":"ContainerDied","Data":"8e490b3b82617dcc3a632548c36871f2741347b67c715d45f1673e22e787bf2f"}
Jul 17 22:03:07.228562 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:07.228496    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw" event={"ID":"1b1e0034-dbab-4300-b374-2ea283b0f18d","Type":"ContainerStarted","Data":"0eb2893a50e33cf834d3607f3ae16feca551d46802251213fecb196ea38a1bbf"}
Jul 17 22:03:07.228641 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:07.228603    2573 scope.go:117] "RemoveContainer" containerID="8e490b3b82617dcc3a632548c36871f2741347b67c715d45f1673e22e787bf2f"
Jul 17 22:03:08.234164 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:08.234127    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b1e0034-dbab-4300-b374-2ea283b0f18d" containerID="420d9dc92d1bb5a5546302cae3ccf1d8698bee5166955051016efc7597d19bbb" exitCode=6
Jul 17 22:03:08.234668 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:08.234201    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw" event={"ID":"1b1e0034-dbab-4300-b374-2ea283b0f18d","Type":"ContainerDied","Data":"420d9dc92d1bb5a5546302cae3ccf1d8698bee5166955051016efc7597d19bbb"}
Jul 17 22:03:08.234668 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:08.234247    2573 scope.go:117] "RemoveContainer" containerID="8e490b3b82617dcc3a632548c36871f2741347b67c715d45f1673e22e787bf2f"
Jul 17 22:03:08.234668 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:08.234460    2573 scope.go:117] "RemoveContainer" containerID="420d9dc92d1bb5a5546302cae3ccf1d8698bee5166955051016efc7597d19bbb"
Jul 17 22:03:08.234852 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:03:08.234681    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw_odh-ai-gateway-infra(1b1e0034-dbab-4300-b374-2ea283b0f18d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw" podUID="1b1e0034-dbab-4300-b374-2ea283b0f18d"
Jul 17 22:03:09.240295 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:09.240267    2573 scope.go:117] "RemoveContainer" containerID="420d9dc92d1bb5a5546302cae3ccf1d8698bee5166955051016efc7597d19bbb"
Jul 17 22:03:09.240676 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:03:09.240446    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw_odh-ai-gateway-infra(1b1e0034-dbab-4300-b374-2ea283b0f18d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw" podUID="1b1e0034-dbab-4300-b374-2ea283b0f18d"
Jul 17 22:03:19.524327 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:19.524272    2573 scope.go:117] "RemoveContainer" containerID="4bfac91a044d6afdc868639b2a9cd5841fbb35e941eb8b357fe2116c43930326"
Jul 17 22:03:19.995385 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:19.995358    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c"]
Jul 17 22:03:19.998613 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:19.998589    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c"
Jul 17 22:03:20.006150 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:20.006128    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c"]
Jul 17 22:03:20.091057 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:20.090987    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e65e9f-d64a-448c-9ff0-01972d676213-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c\" (UID: \"45e65e9f-d64a-448c-9ff0-01972d676213\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c"
Jul 17 22:03:20.191743 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:20.191713    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e65e9f-d64a-448c-9ff0-01972d676213-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c\" (UID: \"45e65e9f-d64a-448c-9ff0-01972d676213\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c"
Jul 17 22:03:20.192273 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:20.192254    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e65e9f-d64a-448c-9ff0-01972d676213-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c\" (UID: \"45e65e9f-d64a-448c-9ff0-01972d676213\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c"
Jul 17 22:03:20.289912 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:20.289886    2573 generic.go:356] "Generic (PLEG): container finished" podID="02b5537d-2b77-409c-b459-fbc2b8cc9ca7" containerID="ee8fee512246321e55d1c2d0220c7abb6aad5e35b5330d452c57c69146cf1d7d" exitCode=6
Jul 17 22:03:20.290029 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:20.289935    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg" event={"ID":"02b5537d-2b77-409c-b459-fbc2b8cc9ca7","Type":"ContainerDied","Data":"ee8fee512246321e55d1c2d0220c7abb6aad5e35b5330d452c57c69146cf1d7d"}
Jul 17 22:03:20.290029 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:20.289997    2573 scope.go:117] "RemoveContainer" containerID="4bfac91a044d6afdc868639b2a9cd5841fbb35e941eb8b357fe2116c43930326"
Jul 17 22:03:20.290279 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:20.290260    2573 scope.go:117] "RemoveContainer" containerID="ee8fee512246321e55d1c2d0220c7abb6aad5e35b5330d452c57c69146cf1d7d"
Jul 17 22:03:20.290537 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:03:20.290517    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-dbfxg_odh-ai-gateway-infra(02b5537d-2b77-409c-b459-fbc2b8cc9ca7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg" podUID="02b5537d-2b77-409c-b459-fbc2b8cc9ca7"
Jul 17 22:03:20.309425 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:20.309405    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c"
Jul 17 22:03:20.435113 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:20.435088    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c"]
Jul 17 22:03:20.436940 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:03:20.436913    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e65e9f_d64a_448c_9ff0_01972d676213.slice/crio-55b0b115c39fe91a13199f803400cf4ac40dd0c673cbf445537a225a95819a16 WatchSource:0}: Error finding container 55b0b115c39fe91a13199f803400cf4ac40dd0c673cbf445537a225a95819a16: Status 404 returned error can't find the container with id 55b0b115c39fe91a13199f803400cf4ac40dd0c673cbf445537a225a95819a16
Jul 17 22:03:20.526271 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:20.526252    2573 scope.go:117] "RemoveContainer" containerID="420d9dc92d1bb5a5546302cae3ccf1d8698bee5166955051016efc7597d19bbb"
Jul 17 22:03:21.295666 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:21.295636    2573 generic.go:356] "Generic (PLEG): container finished" podID="45e65e9f-d64a-448c-9ff0-01972d676213" containerID="c1887ecfd5873d348490b979a63d9fe0496ee0d15792407f201fc6b7cf580841" exitCode=6
Jul 17 22:03:21.295786 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:21.295710    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c" event={"ID":"45e65e9f-d64a-448c-9ff0-01972d676213","Type":"ContainerDied","Data":"c1887ecfd5873d348490b979a63d9fe0496ee0d15792407f201fc6b7cf580841"}
Jul 17 22:03:21.295855 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:21.295795    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c" event={"ID":"45e65e9f-d64a-448c-9ff0-01972d676213","Type":"ContainerStarted","Data":"55b0b115c39fe91a13199f803400cf4ac40dd0c673cbf445537a225a95819a16"}
Jul 17 22:03:21.295913 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:21.295893    2573 scope.go:117] "RemoveContainer" containerID="c1887ecfd5873d348490b979a63d9fe0496ee0d15792407f201fc6b7cf580841"
Jul 17 22:03:21.299003 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:21.298978    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b1e0034-dbab-4300-b374-2ea283b0f18d" containerID="ef09dc04390a24e9acd74721fbc329381fdc9a8e3a09bb74a85d9df5824bb05d" exitCode=6
Jul 17 22:03:21.299128 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:21.299032    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw" event={"ID":"1b1e0034-dbab-4300-b374-2ea283b0f18d","Type":"ContainerDied","Data":"ef09dc04390a24e9acd74721fbc329381fdc9a8e3a09bb74a85d9df5824bb05d"}
Jul 17 22:03:21.299128 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:21.299070    2573 scope.go:117] "RemoveContainer" containerID="420d9dc92d1bb5a5546302cae3ccf1d8698bee5166955051016efc7597d19bbb"
Jul 17 22:03:21.299407 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:21.299356    2573 scope.go:117] "RemoveContainer" containerID="ef09dc04390a24e9acd74721fbc329381fdc9a8e3a09bb74a85d9df5824bb05d"
Jul 17 22:03:21.299554 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:03:21.299527    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw_odh-ai-gateway-infra(1b1e0034-dbab-4300-b374-2ea283b0f18d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw" podUID="1b1e0034-dbab-4300-b374-2ea283b0f18d"
Jul 17 22:03:21.316461 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:21.316438    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg"]
Jul 17 22:03:21.438982 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:21.438959    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg"
Jul 17 22:03:21.501320 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:21.501247    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b5537d-2b77-409c-b459-fbc2b8cc9ca7-maas-api-service-ca\") pod \"02b5537d-2b77-409c-b459-fbc2b8cc9ca7\" (UID: \"02b5537d-2b77-409c-b459-fbc2b8cc9ca7\") "
Jul 17 22:03:21.501574 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:21.501554    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02b5537d-2b77-409c-b459-fbc2b8cc9ca7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02b5537d-2b77-409c-b459-fbc2b8cc9ca7" (UID: "02b5537d-2b77-409c-b459-fbc2b8cc9ca7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:03:21.601997 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:21.601971    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02b5537d-2b77-409c-b459-fbc2b8cc9ca7-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:03:22.304635 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:22.304552    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg"
Jul 17 22:03:22.304635 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:22.304588    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg" event={"ID":"02b5537d-2b77-409c-b459-fbc2b8cc9ca7","Type":"ContainerDied","Data":"221c5d555543c5bf019eeeb9a8354ee819070f055a08f50eadd8444408e824ff"}
Jul 17 22:03:22.304850 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:22.304646    2573 scope.go:117] "RemoveContainer" containerID="ee8fee512246321e55d1c2d0220c7abb6aad5e35b5330d452c57c69146cf1d7d"
Jul 17 22:03:22.308788 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:22.308759    2573 generic.go:356] "Generic (PLEG): container finished" podID="45e65e9f-d64a-448c-9ff0-01972d676213" containerID="3a633c57065034250397037aa339455365255e46a1692d1834b9b8274746c657" exitCode=6
Jul 17 22:03:22.308916 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:22.308798    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c" event={"ID":"45e65e9f-d64a-448c-9ff0-01972d676213","Type":"ContainerDied","Data":"3a633c57065034250397037aa339455365255e46a1692d1834b9b8274746c657"}
Jul 17 22:03:22.309007 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:22.308993    2573 scope.go:117] "RemoveContainer" containerID="3a633c57065034250397037aa339455365255e46a1692d1834b9b8274746c657"
Jul 17 22:03:22.309155 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:03:22.309139    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c_odh-ai-gateway-infra(45e65e9f-d64a-448c-9ff0-01972d676213)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c" podUID="45e65e9f-d64a-448c-9ff0-01972d676213"
Jul 17 22:03:22.313992 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:22.313878    2573 scope.go:117] "RemoveContainer" containerID="c1887ecfd5873d348490b979a63d9fe0496ee0d15792407f201fc6b7cf580841"
Jul 17 22:03:22.338691 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:22.338669    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg"]
Jul 17 22:03:22.342691 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:22.342672    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw"]
Jul 17 22:03:22.345083 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:22.345060    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-dbfxg"]
Jul 17 22:03:22.476776 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:22.476753    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw"
Jul 17 22:03:22.508992 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:22.508970    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b1e0034-dbab-4300-b374-2ea283b0f18d-maas-api-service-ca\") pod \"1b1e0034-dbab-4300-b374-2ea283b0f18d\" (UID: \"1b1e0034-dbab-4300-b374-2ea283b0f18d\") "
Jul 17 22:03:22.509269 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:22.509250    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1e0034-dbab-4300-b374-2ea283b0f18d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b1e0034-dbab-4300-b374-2ea283b0f18d" (UID: "1b1e0034-dbab-4300-b374-2ea283b0f18d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:03:22.530137 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:22.530112    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02b5537d-2b77-409c-b459-fbc2b8cc9ca7" path="/var/lib/kubelet/pods/02b5537d-2b77-409c-b459-fbc2b8cc9ca7/volumes"
Jul 17 22:03:22.609583 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:22.609550    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b1e0034-dbab-4300-b374-2ea283b0f18d-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:03:23.314705 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:23.314671    2573 scope.go:117] "RemoveContainer" containerID="3a633c57065034250397037aa339455365255e46a1692d1834b9b8274746c657"
Jul 17 22:03:23.314889 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:03:23.314867    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c_odh-ai-gateway-infra(45e65e9f-d64a-448c-9ff0-01972d676213)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c" podUID="45e65e9f-d64a-448c-9ff0-01972d676213"
Jul 17 22:03:23.316584 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:23.316558    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw" event={"ID":"1b1e0034-dbab-4300-b374-2ea283b0f18d","Type":"ContainerDied","Data":"0eb2893a50e33cf834d3607f3ae16feca551d46802251213fecb196ea38a1bbf"}
Jul 17 22:03:23.316713 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:23.316611    2573 scope.go:117] "RemoveContainer" containerID="ef09dc04390a24e9acd74721fbc329381fdc9a8e3a09bb74a85d9df5824bb05d"
Jul 17 22:03:23.316713 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:23.316568    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw"
Jul 17 22:03:23.339193 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:23.339164    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw"]
Jul 17 22:03:23.342404 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:23.342385    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-t9fjw"]
Jul 17 22:03:24.536605 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:24.536564    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b1e0034-dbab-4300-b374-2ea283b0f18d" path="/var/lib/kubelet/pods/1b1e0034-dbab-4300-b374-2ea283b0f18d/volumes"
Jul 17 22:03:28.054539 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.054506    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz"]
Jul 17 22:03:28.054915 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.054781    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b1e0034-dbab-4300-b374-2ea283b0f18d" containerName="revoke-keys"
Jul 17 22:03:28.054915 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.054793    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b1e0034-dbab-4300-b374-2ea283b0f18d" containerName="revoke-keys"
Jul 17 22:03:28.054915 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.054807    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b1e0034-dbab-4300-b374-2ea283b0f18d" containerName="revoke-keys"
Jul 17 22:03:28.054915 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.054812    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b1e0034-dbab-4300-b374-2ea283b0f18d" containerName="revoke-keys"
Jul 17 22:03:28.054915 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.054823    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02b5537d-2b77-409c-b459-fbc2b8cc9ca7" containerName="revoke-keys"
Jul 17 22:03:28.054915 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.054829    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="02b5537d-2b77-409c-b459-fbc2b8cc9ca7" containerName="revoke-keys"
Jul 17 22:03:28.054915 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.054841    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02b5537d-2b77-409c-b459-fbc2b8cc9ca7" containerName="revoke-keys"
Jul 17 22:03:28.054915 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.054847    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="02b5537d-2b77-409c-b459-fbc2b8cc9ca7" containerName="revoke-keys"
Jul 17 22:03:28.054915 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.054855    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02b5537d-2b77-409c-b459-fbc2b8cc9ca7" containerName="revoke-keys"
Jul 17 22:03:28.054915 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.054860    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="02b5537d-2b77-409c-b459-fbc2b8cc9ca7" containerName="revoke-keys"
Jul 17 22:03:28.054915 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.054908    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="02b5537d-2b77-409c-b459-fbc2b8cc9ca7" containerName="revoke-keys"
Jul 17 22:03:28.054915 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.054915    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b1e0034-dbab-4300-b374-2ea283b0f18d" containerName="revoke-keys"
Jul 17 22:03:28.054915 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.054922    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b1e0034-dbab-4300-b374-2ea283b0f18d" containerName="revoke-keys"
Jul 17 22:03:28.055435 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.054927    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="02b5537d-2b77-409c-b459-fbc2b8cc9ca7" containerName="revoke-keys"
Jul 17 22:03:28.055435 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.054933    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b1e0034-dbab-4300-b374-2ea283b0f18d" containerName="revoke-keys"
Jul 17 22:03:28.055435 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.054940    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="02b5537d-2b77-409c-b459-fbc2b8cc9ca7" containerName="revoke-keys"
Jul 17 22:03:28.058851 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.058834    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz"
Jul 17 22:03:28.064828 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.064806    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz"]
Jul 17 22:03:28.148843 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.148815    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd26dcd-25f8-43a9-87ed-48f2d41ccf29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz\" (UID: \"dcd26dcd-25f8-43a9-87ed-48f2d41ccf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz"
Jul 17 22:03:28.249207 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.249179    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd26dcd-25f8-43a9-87ed-48f2d41ccf29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz\" (UID: \"dcd26dcd-25f8-43a9-87ed-48f2d41ccf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz"
Jul 17 22:03:28.249770 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.249750    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd26dcd-25f8-43a9-87ed-48f2d41ccf29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz\" (UID: \"dcd26dcd-25f8-43a9-87ed-48f2d41ccf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz"
Jul 17 22:03:28.369614 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.369591    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz"
Jul 17 22:03:28.496632 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:28.496488    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz"]
Jul 17 22:03:28.498851 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:03:28.498811    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd26dcd_25f8_43a9_87ed_48f2d41ccf29.slice/crio-bf4a06d427047379c52a5618e26ab1f278437c4f333d2344d3242e9039680a48 WatchSource:0}: Error finding container bf4a06d427047379c52a5618e26ab1f278437c4f333d2344d3242e9039680a48: Status 404 returned error can't find the container with id bf4a06d427047379c52a5618e26ab1f278437c4f333d2344d3242e9039680a48
Jul 17 22:03:29.345020 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:29.344988    2573 generic.go:356] "Generic (PLEG): container finished" podID="dcd26dcd-25f8-43a9-87ed-48f2d41ccf29" containerID="62b59476478d11c6c744825d144307b2c6bc2d618f9da29e06836b444f0c2cdd" exitCode=6
Jul 17 22:03:29.345442 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:29.345060    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz" event={"ID":"dcd26dcd-25f8-43a9-87ed-48f2d41ccf29","Type":"ContainerDied","Data":"62b59476478d11c6c744825d144307b2c6bc2d618f9da29e06836b444f0c2cdd"}
Jul 17 22:03:29.345442 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:29.345097    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz" event={"ID":"dcd26dcd-25f8-43a9-87ed-48f2d41ccf29","Type":"ContainerStarted","Data":"bf4a06d427047379c52a5618e26ab1f278437c4f333d2344d3242e9039680a48"}
Jul 17 22:03:29.345442 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:29.345253    2573 scope.go:117] "RemoveContainer" containerID="62b59476478d11c6c744825d144307b2c6bc2d618f9da29e06836b444f0c2cdd"
Jul 17 22:03:30.351528 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:30.351495    2573 generic.go:356] "Generic (PLEG): container finished" podID="dcd26dcd-25f8-43a9-87ed-48f2d41ccf29" containerID="b335eadfc7eb78fa9ec265ca7965c4bca2f468d41faed4c63f9084ab7b664bb9" exitCode=6
Jul 17 22:03:30.351920 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:30.351581    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz" event={"ID":"dcd26dcd-25f8-43a9-87ed-48f2d41ccf29","Type":"ContainerDied","Data":"b335eadfc7eb78fa9ec265ca7965c4bca2f468d41faed4c63f9084ab7b664bb9"}
Jul 17 22:03:30.351920 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:30.351666    2573 scope.go:117] "RemoveContainer" containerID="62b59476478d11c6c744825d144307b2c6bc2d618f9da29e06836b444f0c2cdd"
Jul 17 22:03:30.351920 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:30.351774    2573 scope.go:117] "RemoveContainer" containerID="b335eadfc7eb78fa9ec265ca7965c4bca2f468d41faed4c63f9084ab7b664bb9"
Jul 17 22:03:30.352065 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:03:30.352010    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz_odh-ai-gateway-infra(dcd26dcd-25f8-43a9-87ed-48f2d41ccf29)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz" podUID="dcd26dcd-25f8-43a9-87ed-48f2d41ccf29"
Jul 17 22:03:31.357610 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:31.357585    2573 scope.go:117] "RemoveContainer" containerID="b335eadfc7eb78fa9ec265ca7965c4bca2f468d41faed4c63f9084ab7b664bb9"
Jul 17 22:03:31.357951 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:03:31.357765    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz_odh-ai-gateway-infra(dcd26dcd-25f8-43a9-87ed-48f2d41ccf29)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz" podUID="dcd26dcd-25f8-43a9-87ed-48f2d41ccf29"
Jul 17 22:03:35.524560 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:35.524534    2573 scope.go:117] "RemoveContainer" containerID="3a633c57065034250397037aa339455365255e46a1692d1834b9b8274746c657"
Jul 17 22:03:36.380203 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:36.380169    2573 generic.go:356] "Generic (PLEG): container finished" podID="45e65e9f-d64a-448c-9ff0-01972d676213" containerID="3e3576992cc42eea5b9f122e2c7e7d390f4d461fba4f9d3956412e324dbb48c6" exitCode=6
Jul 17 22:03:36.380360 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:36.380237    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c" event={"ID":"45e65e9f-d64a-448c-9ff0-01972d676213","Type":"ContainerDied","Data":"3e3576992cc42eea5b9f122e2c7e7d390f4d461fba4f9d3956412e324dbb48c6"}
Jul 17 22:03:36.380360 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:36.380304    2573 scope.go:117] "RemoveContainer" containerID="3a633c57065034250397037aa339455365255e46a1692d1834b9b8274746c657"
Jul 17 22:03:36.380604 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:36.380573    2573 scope.go:117] "RemoveContainer" containerID="3e3576992cc42eea5b9f122e2c7e7d390f4d461fba4f9d3956412e324dbb48c6"
Jul 17 22:03:36.380784 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:03:36.380759    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c_odh-ai-gateway-infra(45e65e9f-d64a-448c-9ff0-01972d676213)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c" podUID="45e65e9f-d64a-448c-9ff0-01972d676213"
Jul 17 22:03:37.407259 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:37.407227    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c"]
Jul 17 22:03:37.536865 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:37.536839    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c"
Jul 17 22:03:37.620986 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:37.620955    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e65e9f-d64a-448c-9ff0-01972d676213-maas-api-service-ca\") pod \"45e65e9f-d64a-448c-9ff0-01972d676213\" (UID: \"45e65e9f-d64a-448c-9ff0-01972d676213\") "
Jul 17 22:03:37.621262 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:37.621241    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e65e9f-d64a-448c-9ff0-01972d676213-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "45e65e9f-d64a-448c-9ff0-01972d676213" (UID: "45e65e9f-d64a-448c-9ff0-01972d676213"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:03:37.722119 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:37.722045    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e65e9f-d64a-448c-9ff0-01972d676213-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:03:38.391898 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:38.391870    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c"
Jul 17 22:03:38.391898 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:38.391884    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c" event={"ID":"45e65e9f-d64a-448c-9ff0-01972d676213","Type":"ContainerDied","Data":"55b0b115c39fe91a13199f803400cf4ac40dd0c673cbf445537a225a95819a16"}
Jul 17 22:03:38.392130 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:38.391936    2573 scope.go:117] "RemoveContainer" containerID="3e3576992cc42eea5b9f122e2c7e7d390f4d461fba4f9d3956412e324dbb48c6"
Jul 17 22:03:38.413071 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:38.413048    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c"]
Jul 17 22:03:38.418253 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:38.418231    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-jms4c"]
Jul 17 22:03:38.529561 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:38.529537    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45e65e9f-d64a-448c-9ff0-01972d676213" path="/var/lib/kubelet/pods/45e65e9f-d64a-448c-9ff0-01972d676213/volumes"
Jul 17 22:03:45.525076 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:45.525004    2573 scope.go:117] "RemoveContainer" containerID="b335eadfc7eb78fa9ec265ca7965c4bca2f468d41faed4c63f9084ab7b664bb9"
Jul 17 22:03:46.432516 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:46.432425    2573 generic.go:356] "Generic (PLEG): container finished" podID="dcd26dcd-25f8-43a9-87ed-48f2d41ccf29" containerID="02c72c346e77f6cfdab1266ba0e188f316206bee824d99312059588c6a01e990" exitCode=6
Jul 17 22:03:46.432516 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:46.432471    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz" event={"ID":"dcd26dcd-25f8-43a9-87ed-48f2d41ccf29","Type":"ContainerDied","Data":"02c72c346e77f6cfdab1266ba0e188f316206bee824d99312059588c6a01e990"}
Jul 17 22:03:46.432736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:46.432531    2573 scope.go:117] "RemoveContainer" containerID="b335eadfc7eb78fa9ec265ca7965c4bca2f468d41faed4c63f9084ab7b664bb9"
Jul 17 22:03:46.432878 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:46.432854    2573 scope.go:117] "RemoveContainer" containerID="02c72c346e77f6cfdab1266ba0e188f316206bee824d99312059588c6a01e990"
Jul 17 22:03:46.433102 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:03:46.433077    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz_odh-ai-gateway-infra(dcd26dcd-25f8-43a9-87ed-48f2d41ccf29)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz" podUID="dcd26dcd-25f8-43a9-87ed-48f2d41ccf29"
Jul 17 22:03:47.457298 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:47.457265    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz"]
Jul 17 22:03:47.587347 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:47.587326    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz"
Jul 17 22:03:47.697055 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:47.697018    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd26dcd-25f8-43a9-87ed-48f2d41ccf29-maas-api-service-ca\") pod \"dcd26dcd-25f8-43a9-87ed-48f2d41ccf29\" (UID: \"dcd26dcd-25f8-43a9-87ed-48f2d41ccf29\") "
Jul 17 22:03:47.697368 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:47.697350    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd26dcd-25f8-43a9-87ed-48f2d41ccf29-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dcd26dcd-25f8-43a9-87ed-48f2d41ccf29" (UID: "dcd26dcd-25f8-43a9-87ed-48f2d41ccf29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:03:47.798333 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:47.798262    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd26dcd-25f8-43a9-87ed-48f2d41ccf29-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:03:48.443259 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:48.443225    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz"
Jul 17 22:03:48.443460 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:48.443231    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz" event={"ID":"dcd26dcd-25f8-43a9-87ed-48f2d41ccf29","Type":"ContainerDied","Data":"bf4a06d427047379c52a5618e26ab1f278437c4f333d2344d3242e9039680a48"}
Jul 17 22:03:48.443460 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:48.443364    2573 scope.go:117] "RemoveContainer" containerID="02c72c346e77f6cfdab1266ba0e188f316206bee824d99312059588c6a01e990"
Jul 17 22:03:48.462522 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:48.462502    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz"]
Jul 17 22:03:48.468757 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:48.468734    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-n95zz"]
Jul 17 22:03:48.529338 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:48.529312    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcd26dcd-25f8-43a9-87ed-48f2d41ccf29" path="/var/lib/kubelet/pods/dcd26dcd-25f8-43a9-87ed-48f2d41ccf29/volumes"
Jul 17 22:03:55.842606 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.842565    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6"]
Jul 17 22:03:55.842995 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.842839    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e65e9f-d64a-448c-9ff0-01972d676213" containerName="revoke-keys"
Jul 17 22:03:55.842995 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.842850    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e65e9f-d64a-448c-9ff0-01972d676213" containerName="revoke-keys"
Jul 17 22:03:55.842995 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.842860    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b1e0034-dbab-4300-b374-2ea283b0f18d" containerName="revoke-keys"
Jul 17 22:03:55.842995 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.842866    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b1e0034-dbab-4300-b374-2ea283b0f18d" containerName="revoke-keys"
Jul 17 22:03:55.842995 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.842872    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e65e9f-d64a-448c-9ff0-01972d676213" containerName="revoke-keys"
Jul 17 22:03:55.842995 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.842878    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e65e9f-d64a-448c-9ff0-01972d676213" containerName="revoke-keys"
Jul 17 22:03:55.842995 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.842884    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd26dcd-25f8-43a9-87ed-48f2d41ccf29" containerName="revoke-keys"
Jul 17 22:03:55.842995 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.842889    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd26dcd-25f8-43a9-87ed-48f2d41ccf29" containerName="revoke-keys"
Jul 17 22:03:55.842995 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.842894    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd26dcd-25f8-43a9-87ed-48f2d41ccf29" containerName="revoke-keys"
Jul 17 22:03:55.842995 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.842899    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd26dcd-25f8-43a9-87ed-48f2d41ccf29" containerName="revoke-keys"
Jul 17 22:03:55.842995 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.842905    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e65e9f-d64a-448c-9ff0-01972d676213" containerName="revoke-keys"
Jul 17 22:03:55.842995 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.842910    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e65e9f-d64a-448c-9ff0-01972d676213" containerName="revoke-keys"
Jul 17 22:03:55.842995 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.842962    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcd26dcd-25f8-43a9-87ed-48f2d41ccf29" containerName="revoke-keys"
Jul 17 22:03:55.842995 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.842969    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcd26dcd-25f8-43a9-87ed-48f2d41ccf29" containerName="revoke-keys"
Jul 17 22:03:55.842995 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.842974    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="45e65e9f-d64a-448c-9ff0-01972d676213" containerName="revoke-keys"
Jul 17 22:03:55.842995 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.842980    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="45e65e9f-d64a-448c-9ff0-01972d676213" containerName="revoke-keys"
Jul 17 22:03:55.847138 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.847121    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6"
Jul 17 22:03:55.849498 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.849479    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:03:55.849611 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.849577    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 22:03:55.852873 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.852854    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6"]
Jul 17 22:03:55.963756 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:55.963724    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcb7c3f2-3475-4321-953d-7c36c0bf9faa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6\" (UID: \"dcb7c3f2-3475-4321-953d-7c36c0bf9faa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6"
Jul 17 22:03:56.064440 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:56.064411    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcb7c3f2-3475-4321-953d-7c36c0bf9faa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6\" (UID: \"dcb7c3f2-3475-4321-953d-7c36c0bf9faa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6"
Jul 17 22:03:56.065032 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:56.065013    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcb7c3f2-3475-4321-953d-7c36c0bf9faa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6\" (UID: \"dcb7c3f2-3475-4321-953d-7c36c0bf9faa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6"
Jul 17 22:03:56.158395 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:56.158337    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6"
Jul 17 22:03:56.279944 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:56.279922    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6"]
Jul 17 22:03:56.281797 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:03:56.281774    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb7c3f2_3475_4321_953d_7c36c0bf9faa.slice/crio-4b22bde42a3de7f569ae136eaccd0cffc765c56893b63cd220fc1bcf38db14aa WatchSource:0}: Error finding container 4b22bde42a3de7f569ae136eaccd0cffc765c56893b63cd220fc1bcf38db14aa: Status 404 returned error can't find the container with id 4b22bde42a3de7f569ae136eaccd0cffc765c56893b63cd220fc1bcf38db14aa
Jul 17 22:03:56.483299 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:56.483252    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6" event={"ID":"dcb7c3f2-3475-4321-953d-7c36c0bf9faa","Type":"ContainerStarted","Data":"4b22bde42a3de7f569ae136eaccd0cffc765c56893b63cd220fc1bcf38db14aa"}
Jul 17 22:03:57.488965 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:57.488929    2573 generic.go:356] "Generic (PLEG): container finished" podID="dcb7c3f2-3475-4321-953d-7c36c0bf9faa" containerID="623a34cdbc2a884166ded4e5bf8cc96f0aabcbcdac1ab21ed515b8e0d6cb3967" exitCode=6
Jul 17 22:03:57.489380 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:57.489013    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6" event={"ID":"dcb7c3f2-3475-4321-953d-7c36c0bf9faa","Type":"ContainerDied","Data":"623a34cdbc2a884166ded4e5bf8cc96f0aabcbcdac1ab21ed515b8e0d6cb3967"}
Jul 17 22:03:57.489380 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:57.489162    2573 scope.go:117] "RemoveContainer" containerID="623a34cdbc2a884166ded4e5bf8cc96f0aabcbcdac1ab21ed515b8e0d6cb3967"
Jul 17 22:03:58.495769 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:58.495694    2573 generic.go:356] "Generic (PLEG): container finished" podID="dcb7c3f2-3475-4321-953d-7c36c0bf9faa" containerID="cd00d4286a7d3d7dcc903474ab81744a00657f89b5d08fc89795d92cd7cf16a4" exitCode=6
Jul 17 22:03:58.495769 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:58.495736    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6" event={"ID":"dcb7c3f2-3475-4321-953d-7c36c0bf9faa","Type":"ContainerDied","Data":"cd00d4286a7d3d7dcc903474ab81744a00657f89b5d08fc89795d92cd7cf16a4"}
Jul 17 22:03:58.496196 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:58.495784    2573 scope.go:117] "RemoveContainer" containerID="623a34cdbc2a884166ded4e5bf8cc96f0aabcbcdac1ab21ed515b8e0d6cb3967"
Jul 17 22:03:58.496196 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:58.495936    2573 scope.go:117] "RemoveContainer" containerID="cd00d4286a7d3d7dcc903474ab81744a00657f89b5d08fc89795d92cd7cf16a4"
Jul 17 22:03:58.496196 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:03:58.496131    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6_odh-ai-gateway-infra(dcb7c3f2-3475-4321-953d-7c36c0bf9faa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6" podUID="dcb7c3f2-3475-4321-953d-7c36c0bf9faa"
Jul 17 22:03:59.501457 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:03:59.501431    2573 scope.go:117] "RemoveContainer" containerID="cd00d4286a7d3d7dcc903474ab81744a00657f89b5d08fc89795d92cd7cf16a4"
Jul 17 22:03:59.501846 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:03:59.501588    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6_odh-ai-gateway-infra(dcb7c3f2-3475-4321-953d-7c36c0bf9faa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6" podUID="dcb7c3f2-3475-4321-953d-7c36c0bf9faa"
Jul 17 22:04:01.145371 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:01.145330    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn"]
Jul 17 22:04:01.145871 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:01.145764    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd26dcd-25f8-43a9-87ed-48f2d41ccf29" containerName="revoke-keys"
Jul 17 22:04:01.145871 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:01.145783    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd26dcd-25f8-43a9-87ed-48f2d41ccf29" containerName="revoke-keys"
Jul 17 22:04:01.145871 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:01.145846    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="45e65e9f-d64a-448c-9ff0-01972d676213" containerName="revoke-keys"
Jul 17 22:04:01.145871 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:01.145860    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcd26dcd-25f8-43a9-87ed-48f2d41ccf29" containerName="revoke-keys"
Jul 17 22:04:01.149947 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:01.149925    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn"
Jul 17 22:04:01.155278 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:01.155254    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn"]
Jul 17 22:04:01.201600 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:01.201562    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-svggn\" (UID: \"ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn"
Jul 17 22:04:01.301921 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:01.301901    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-svggn\" (UID: \"ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn"
Jul 17 22:04:01.302468 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:01.302452    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-svggn\" (UID: \"ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn"
Jul 17 22:04:01.462382 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:01.462317    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn"
Jul 17 22:04:01.587028 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:01.587005    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn"]
Jul 17 22:04:01.588955 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:04:01.588919    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd5a4a1_9e4e_4b70_b644_99cc6a7f9713.slice/crio-523af9e5fd869e666e15ba979b131ac52f1da34fde7a3625391b577526019483 WatchSource:0}: Error finding container 523af9e5fd869e666e15ba979b131ac52f1da34fde7a3625391b577526019483: Status 404 returned error can't find the container with id 523af9e5fd869e666e15ba979b131ac52f1da34fde7a3625391b577526019483
Jul 17 22:04:02.515782 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:02.515684    2573 generic.go:356] "Generic (PLEG): container finished" podID="ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713" containerID="8146e02b5d177b89a46148a690097e78f0028fe4999d49992ec7b741a432b134" exitCode=6
Jul 17 22:04:02.515782 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:02.515754    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn" event={"ID":"ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713","Type":"ContainerDied","Data":"8146e02b5d177b89a46148a690097e78f0028fe4999d49992ec7b741a432b134"}
Jul 17 22:04:02.516192 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:02.515802    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn" event={"ID":"ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713","Type":"ContainerStarted","Data":"523af9e5fd869e666e15ba979b131ac52f1da34fde7a3625391b577526019483"}
Jul 17 22:04:02.516192 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:02.516011    2573 scope.go:117] "RemoveContainer" containerID="8146e02b5d177b89a46148a690097e78f0028fe4999d49992ec7b741a432b134"
Jul 17 22:04:03.522038 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:03.521947    2573 generic.go:356] "Generic (PLEG): container finished" podID="ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713" containerID="50c4ef830e5012e06eb2b32c3d3634bb72c85b5d580d7b92bcd0afff632437cf" exitCode=6
Jul 17 22:04:03.522465 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:03.522037    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn" event={"ID":"ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713","Type":"ContainerDied","Data":"50c4ef830e5012e06eb2b32c3d3634bb72c85b5d580d7b92bcd0afff632437cf"}
Jul 17 22:04:03.522465 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:03.522131    2573 scope.go:117] "RemoveContainer" containerID="8146e02b5d177b89a46148a690097e78f0028fe4999d49992ec7b741a432b134"
Jul 17 22:04:03.522465 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:03.522210    2573 scope.go:117] "RemoveContainer" containerID="50c4ef830e5012e06eb2b32c3d3634bb72c85b5d580d7b92bcd0afff632437cf"
Jul 17 22:04:03.522465 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:04:03.522452    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-svggn_odh-ai-gateway-infra(ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn" podUID="ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713"
Jul 17 22:04:04.529299 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:04.529260    2573 scope.go:117] "RemoveContainer" containerID="50c4ef830e5012e06eb2b32c3d3634bb72c85b5d580d7b92bcd0afff632437cf"
Jul 17 22:04:04.529647 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:04:04.529458    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-svggn_odh-ai-gateway-infra(ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn" podUID="ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713"
Jul 17 22:04:10.036167 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:10.036134    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6"]
Jul 17 22:04:10.040701 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:10.040684    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6"
Jul 17 22:04:10.045981 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:10.045957    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6"]
Jul 17 22:04:10.169322 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:10.169263    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee26ff1d-7318-409f-81e3-cb08d3f575f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6\" (UID: \"ee26ff1d-7318-409f-81e3-cb08d3f575f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6"
Jul 17 22:04:10.271436 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:10.271381    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee26ff1d-7318-409f-81e3-cb08d3f575f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6\" (UID: \"ee26ff1d-7318-409f-81e3-cb08d3f575f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6"
Jul 17 22:04:10.272348 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:10.272323    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee26ff1d-7318-409f-81e3-cb08d3f575f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6\" (UID: \"ee26ff1d-7318-409f-81e3-cb08d3f575f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6"
Jul 17 22:04:10.352151 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:10.352110    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6"
Jul 17 22:04:10.479097 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:10.479070    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6"]
Jul 17 22:04:10.481248 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:04:10.481220    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee26ff1d_7318_409f_81e3_cb08d3f575f1.slice/crio-12750ecd365eaf29ebada14944f82282d824a260c0816028f2f9b906b1bb7e13 WatchSource:0}: Error finding container 12750ecd365eaf29ebada14944f82282d824a260c0816028f2f9b906b1bb7e13: Status 404 returned error can't find the container with id 12750ecd365eaf29ebada14944f82282d824a260c0816028f2f9b906b1bb7e13
Jul 17 22:04:10.556625 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:10.556590    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6" event={"ID":"ee26ff1d-7318-409f-81e3-cb08d3f575f1","Type":"ContainerStarted","Data":"12750ecd365eaf29ebada14944f82282d824a260c0816028f2f9b906b1bb7e13"}
Jul 17 22:04:11.524710 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:11.524682    2573 scope.go:117] "RemoveContainer" containerID="cd00d4286a7d3d7dcc903474ab81744a00657f89b5d08fc89795d92cd7cf16a4"
Jul 17 22:04:11.561930 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:11.561907    2573 generic.go:356] "Generic (PLEG): container finished" podID="ee26ff1d-7318-409f-81e3-cb08d3f575f1" containerID="bc0762771ae7487f25ce0b31dac32cfadade908c7a866110dd7b3d09a12d3c04" exitCode=6
Jul 17 22:04:11.562030 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:11.561967    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6" event={"ID":"ee26ff1d-7318-409f-81e3-cb08d3f575f1","Type":"ContainerDied","Data":"bc0762771ae7487f25ce0b31dac32cfadade908c7a866110dd7b3d09a12d3c04"}
Jul 17 22:04:11.562159 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:11.562135    2573 scope.go:117] "RemoveContainer" containerID="bc0762771ae7487f25ce0b31dac32cfadade908c7a866110dd7b3d09a12d3c04"
Jul 17 22:04:12.567527 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:12.567440    2573 generic.go:356] "Generic (PLEG): container finished" podID="ee26ff1d-7318-409f-81e3-cb08d3f575f1" containerID="1c83776b8e982830ec9ea963e564de79aa54a2756cab44cf4849b24cc8aa96d2" exitCode=6
Jul 17 22:04:12.567527 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:12.567512    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6" event={"ID":"ee26ff1d-7318-409f-81e3-cb08d3f575f1","Type":"ContainerDied","Data":"1c83776b8e982830ec9ea963e564de79aa54a2756cab44cf4849b24cc8aa96d2"}
Jul 17 22:04:12.568035 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:12.567577    2573 scope.go:117] "RemoveContainer" containerID="bc0762771ae7487f25ce0b31dac32cfadade908c7a866110dd7b3d09a12d3c04"
Jul 17 22:04:12.568035 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:12.567756    2573 scope.go:117] "RemoveContainer" containerID="1c83776b8e982830ec9ea963e564de79aa54a2756cab44cf4849b24cc8aa96d2"
Jul 17 22:04:12.568035 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:04:12.567952    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6_odh-ai-gateway-infra(ee26ff1d-7318-409f-81e3-cb08d3f575f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1"
Jul 17 22:04:12.569434 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:12.569415    2573 generic.go:356] "Generic (PLEG): container finished" podID="dcb7c3f2-3475-4321-953d-7c36c0bf9faa" containerID="8cfcfaeaaaec9355afd78ebf6c2cba0bb197c55b1e590be6a7501c4edc1c79e9" exitCode=6
Jul 17 22:04:12.569530 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:12.569478    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6" event={"ID":"dcb7c3f2-3475-4321-953d-7c36c0bf9faa","Type":"ContainerDied","Data":"8cfcfaeaaaec9355afd78ebf6c2cba0bb197c55b1e590be6a7501c4edc1c79e9"}
Jul 17 22:04:12.569733 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:12.569720    2573 scope.go:117] "RemoveContainer" containerID="8cfcfaeaaaec9355afd78ebf6c2cba0bb197c55b1e590be6a7501c4edc1c79e9"
Jul 17 22:04:12.569867 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:04:12.569852    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6_odh-ai-gateway-infra(dcb7c3f2-3475-4321-953d-7c36c0bf9faa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6" podUID="dcb7c3f2-3475-4321-953d-7c36c0bf9faa"
Jul 17 22:04:12.578068 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:12.578053    2573 scope.go:117] "RemoveContainer" containerID="cd00d4286a7d3d7dcc903474ab81744a00657f89b5d08fc89795d92cd7cf16a4"
Jul 17 22:04:13.575835 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:13.575798    2573 scope.go:117] "RemoveContainer" containerID="1c83776b8e982830ec9ea963e564de79aa54a2756cab44cf4849b24cc8aa96d2"
Jul 17 22:04:13.576221 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:04:13.575998    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6_odh-ai-gateway-infra(ee26ff1d-7318-409f-81e3-cb08d3f575f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1"
Jul 17 22:04:13.605520 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:13.605496    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6"]
Jul 17 22:04:13.738267 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:13.738242    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6"
Jul 17 22:04:13.802033 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:13.801962    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcb7c3f2-3475-4321-953d-7c36c0bf9faa-maas-api-service-ca\") pod \"dcb7c3f2-3475-4321-953d-7c36c0bf9faa\" (UID: \"dcb7c3f2-3475-4321-953d-7c36c0bf9faa\") "
Jul 17 22:04:13.802325 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:13.802304    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb7c3f2-3475-4321-953d-7c36c0bf9faa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dcb7c3f2-3475-4321-953d-7c36c0bf9faa" (UID: "dcb7c3f2-3475-4321-953d-7c36c0bf9faa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:04:13.903315 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:13.903260    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcb7c3f2-3475-4321-953d-7c36c0bf9faa-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:04:14.582791 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:14.582768    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6"
Jul 17 22:04:14.583208 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:14.582792    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6" event={"ID":"dcb7c3f2-3475-4321-953d-7c36c0bf9faa","Type":"ContainerDied","Data":"4b22bde42a3de7f569ae136eaccd0cffc765c56893b63cd220fc1bcf38db14aa"}
Jul 17 22:04:14.583208 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:14.582844    2573 scope.go:117] "RemoveContainer" containerID="8cfcfaeaaaec9355afd78ebf6c2cba0bb197c55b1e590be6a7501c4edc1c79e9"
Jul 17 22:04:14.599499 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:14.599474    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6"]
Jul 17 22:04:14.601886 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:14.601867    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-bdlj6"]
Jul 17 22:04:16.529189 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:16.529159    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcb7c3f2-3475-4321-953d-7c36c0bf9faa" path="/var/lib/kubelet/pods/dcb7c3f2-3475-4321-953d-7c36c0bf9faa/volumes"
Jul 17 22:04:17.525032 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:17.525005    2573 scope.go:117] "RemoveContainer" containerID="50c4ef830e5012e06eb2b32c3d3634bb72c85b5d580d7b92bcd0afff632437cf"
Jul 17 22:04:18.602475 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:18.602444    2573 generic.go:356] "Generic (PLEG): container finished" podID="ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713" containerID="eaf4ef1a11fad3efd7fcc6e20b79630ccf1bc059365dd9b844a2f59cefe41c8d" exitCode=6
Jul 17 22:04:18.602883 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:18.602489    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn" event={"ID":"ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713","Type":"ContainerDied","Data":"eaf4ef1a11fad3efd7fcc6e20b79630ccf1bc059365dd9b844a2f59cefe41c8d"}
Jul 17 22:04:18.602883 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:18.602538    2573 scope.go:117] "RemoveContainer" containerID="50c4ef830e5012e06eb2b32c3d3634bb72c85b5d580d7b92bcd0afff632437cf"
Jul 17 22:04:18.602883 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:18.602809    2573 scope.go:117] "RemoveContainer" containerID="eaf4ef1a11fad3efd7fcc6e20b79630ccf1bc059365dd9b844a2f59cefe41c8d"
Jul 17 22:04:18.603064 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:04:18.603015    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-svggn_odh-ai-gateway-infra(ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn" podUID="ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713"
Jul 17 22:04:19.625972 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:19.625939    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn"]
Jul 17 22:04:19.756606 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:19.756585    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn"
Jul 17 22:04:19.847592 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:19.847563    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713-maas-api-service-ca\") pod \"ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713\" (UID: \"ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713\") "
Jul 17 22:04:19.847881 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:19.847861    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713" (UID: "ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:04:19.948370 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:19.948302    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:04:20.614769 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:20.614737    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn"
Jul 17 22:04:20.614935 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:20.614771    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn" event={"ID":"ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713","Type":"ContainerDied","Data":"523af9e5fd869e666e15ba979b131ac52f1da34fde7a3625391b577526019483"}
Jul 17 22:04:20.614935 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:20.614831    2573 scope.go:117] "RemoveContainer" containerID="eaf4ef1a11fad3efd7fcc6e20b79630ccf1bc059365dd9b844a2f59cefe41c8d"
Jul 17 22:04:20.630406 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:20.630381    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn"]
Jul 17 22:04:20.636394 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:20.636373    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-svggn"]
Jul 17 22:04:22.530202 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:22.530165    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713" path="/var/lib/kubelet/pods/ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713/volumes"
Jul 17 22:04:26.524593 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:26.524562    2573 scope.go:117] "RemoveContainer" containerID="1c83776b8e982830ec9ea963e564de79aa54a2756cab44cf4849b24cc8aa96d2"
Jul 17 22:04:27.645250 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:27.645216    2573 generic.go:356] "Generic (PLEG): container finished" podID="ee26ff1d-7318-409f-81e3-cb08d3f575f1" containerID="fef03c8be71c10a85cca5c2fe874f18eef864c3bf7cfb437574bf1e7bd48308c" exitCode=6
Jul 17 22:04:27.645627 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:27.645268    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6" event={"ID":"ee26ff1d-7318-409f-81e3-cb08d3f575f1","Type":"ContainerDied","Data":"fef03c8be71c10a85cca5c2fe874f18eef864c3bf7cfb437574bf1e7bd48308c"}
Jul 17 22:04:27.645627 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:27.645350    2573 scope.go:117] "RemoveContainer" containerID="1c83776b8e982830ec9ea963e564de79aa54a2756cab44cf4849b24cc8aa96d2"
Jul 17 22:04:27.645742 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:27.645688    2573 scope.go:117] "RemoveContainer" containerID="fef03c8be71c10a85cca5c2fe874f18eef864c3bf7cfb437574bf1e7bd48308c"
Jul 17 22:04:27.645914 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:04:27.645892    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6_odh-ai-gateway-infra(ee26ff1d-7318-409f-81e3-cb08d3f575f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1"
Jul 17 22:04:37.786764 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:04:37.786724    2573 controller.go:195] "Failed to update lease" err="Put \"https://a2372d7ae419449aca06ea13091ff12e-ea0095fb6ecabfb5.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-128-246.ec2.internal?timeout=10s\": context deadline exceeded"
Jul 17 22:04:39.524674 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:39.524632    2573 scope.go:117] "RemoveContainer" containerID="fef03c8be71c10a85cca5c2fe874f18eef864c3bf7cfb437574bf1e7bd48308c"
Jul 17 22:04:39.525117 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:04:39.524809    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6_odh-ai-gateway-infra(ee26ff1d-7318-409f-81e3-cb08d3f575f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1"
Jul 17 22:04:47.787976 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:04:47.787941    2573 controller.go:195] "Failed to update lease" err="Put \"https://a2372d7ae419449aca06ea13091ff12e-ea0095fb6ecabfb5.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-128-246.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Jul 17 22:04:54.530181 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:54.530153    2573 scope.go:117] "RemoveContainer" containerID="fef03c8be71c10a85cca5c2fe874f18eef864c3bf7cfb437574bf1e7bd48308c"
Jul 17 22:04:55.768770 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:55.768738    2573 generic.go:356] "Generic (PLEG): container finished" podID="ee26ff1d-7318-409f-81e3-cb08d3f575f1" containerID="c6b742006b1512d40df765f5350b91e8ab8fffc0b97a5fb931566e575b0de041" exitCode=6
Jul 17 22:04:55.769202 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:55.768791    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6" event={"ID":"ee26ff1d-7318-409f-81e3-cb08d3f575f1","Type":"ContainerDied","Data":"c6b742006b1512d40df765f5350b91e8ab8fffc0b97a5fb931566e575b0de041"}
Jul 17 22:04:55.769202 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:55.768849    2573 scope.go:117] "RemoveContainer" containerID="fef03c8be71c10a85cca5c2fe874f18eef864c3bf7cfb437574bf1e7bd48308c"
Jul 17 22:04:55.769202 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:04:55.769090    2573 scope.go:117] "RemoveContainer" containerID="c6b742006b1512d40df765f5350b91e8ab8fffc0b97a5fb931566e575b0de041"
Jul 17 22:04:55.769377 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:04:55.769353    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 40s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6_odh-ai-gateway-infra(ee26ff1d-7318-409f-81e3-cb08d3f575f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1"
Jul 17 22:04:57.788878 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:04:57.788838    2573 controller.go:195] "Failed to update lease" err="Put \"https://a2372d7ae419449aca06ea13091ff12e-ea0095fb6ecabfb5.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-128-246.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Jul 17 22:05:07.524359 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:07.524328    2573 scope.go:117] "RemoveContainer" containerID="c6b742006b1512d40df765f5350b91e8ab8fffc0b97a5fb931566e575b0de041"
Jul 17 22:05:07.524716 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:05:07.524609    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 40s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6_odh-ai-gateway-infra(ee26ff1d-7318-409f-81e3-cb08d3f575f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1"
Jul 17 22:05:07.789697 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:05:07.789607    2573 controller.go:195] "Failed to update lease" err="Put \"https://a2372d7ae419449aca06ea13091ff12e-ea0095fb6ecabfb5.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-128-246.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Jul 17 22:05:09.143761 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143690    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.143761 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143695    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.143761 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143736    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.143761 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143752    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.144423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143794    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-q6bg6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.144423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143791    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-d9p27\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.144423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143803    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.144423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143819    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.144423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143895    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.144423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143899    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.144423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143912    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.144423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143906    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.144423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143894    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.144423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143901    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.144423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143933    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.144423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143944    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.144423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143945    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dnvbk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.144423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143962    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-przjx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.144423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143983    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.144423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.143988    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.144423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144002    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.144423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144014    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145096 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144023    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145096 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144039    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rdmx5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145096 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144042    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145096 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144051    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145096 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144069    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145096 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144050    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145096 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144084    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145096 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144098    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145096 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144108    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145096 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144113    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145096 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144128    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145096 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144131    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145096 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144137    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-pckxk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145096 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144151    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-051ea5-openshift-default-dockercfg-5hzm5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145096 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144116    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145096 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144164    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145096 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144160    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145096 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144146    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144181    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144177    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b86v4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144192    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144193    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144206    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144206    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rvgtv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144214    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j8vps\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144232    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144240    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144232    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144235    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"cacerts\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144254    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144232    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144264    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144272    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144278    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144274    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.145736 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144298    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8vkwx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144278    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144319    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144327    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ltpgv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144339    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144348    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144345    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-sz7w2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144357    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144376    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144400    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144405    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144412    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144405    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144425    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144449    2573 reflector.go:571] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144451    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144462    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144475    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144486    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-dockercfg-qpx9f\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146897 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144497    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146897 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144505    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7wgs9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146897 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144515    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cmtnp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146897 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:05:09.144518    2573 controller.go:195] "Failed to update lease" err="Put \"https://a2372d7ae419449aca06ea13091ff12e-ea0095fb6ecabfb5.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-128-246.ec2.internal?timeout=10s\": http2: client connection lost"
Jul 17 22:05:09.146897 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144533    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mcghg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146897 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144538    2573 controller.go:115] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease"
Jul 17 22:05:09.146897 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144560    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146897 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144581    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146897 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144583    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6lcn7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146897 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144602    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146897 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144612    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146897 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144617    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sqgsr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146897 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:05:09.144560    2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a2372d7ae419449aca06ea13091ff12e-ea0095fb6ecabfb5.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/events/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6.18c332e8419a81a6\": http2: client connection lost" event="&Event{ObjectMeta:{maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6.18c332e8419a81a6  odh-ai-gateway-infra   71888 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:odh-ai-gateway-infra,Name:maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6,UID:ee26ff1d-7318-409f-81e3-cb08d3f575f1,APIVersion:v1,ResourceVersion:71833,FieldPath:spec.containers{revoke-keys},},Reason:Pulled,Message:Container image \"registry.redhat.io/ubi9/ubi-minimal:9.7\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-128-246.ec2.internal,},FirstTimestamp:2026-07-17 22:04:10 +0000 UTC,LastTimestamp:2026-07-17 22:04:26.5255806 +0000 UTC m=+3366.489701187,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-246.ec2.internal,}"
Jul 17 22:05:09.146897 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144643    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146897 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144646    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"default-dockercfg-bnb28\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.146897 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144642    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.147432 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144656    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.147432 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144668    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.147432 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144680    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.147432 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144685    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.147432 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144682    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.147432 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144695    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-x4mt8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.147432 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144710    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.147432 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144707    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-st79t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.147432 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144721    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.147432 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144738    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.147432 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144745    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-kzfg4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.147432 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144748    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.147432 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144754    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.147432 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144760    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.147432 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144767    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tppfj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.147432 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144780    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-szcqz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.147432 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144778    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.147432 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.144794    2573 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Jul 17 22:05:09.958239 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.958192    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 22:05:09.963566 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.963541    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 17 22:05:09.965940 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.965919    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 22:05:09.968765 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.968746    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 22:05:09.974127 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.974111    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 17 22:05:09.978969 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.978945    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-kzfg4\""
Jul 17 22:05:09.980527 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.980511    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 22:05:09.981404 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:09.981388    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-pckxk\""
Jul 17 22:05:10.005012 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.004185    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 22:05:10.013787 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.013759    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 22:05:10.037209 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.037183    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 17 22:05:10.037209 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.037183    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 17 22:05:10.047019 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.047001    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6lcn7\""
Jul 17 22:05:10.047932 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.047916    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7wgs9\""
Jul 17 22:05:10.050507 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.050486    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 17 22:05:10.051606 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.051587    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qpx9f\""
Jul 17 22:05:10.057388 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.057370    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 22:05:10.062010 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.061993    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 22:05:10.063134 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.063118    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 22:05:10.065521 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.065505    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 22:05:10.073202 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.073183    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 17 22:05:10.080023 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.080006    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 22:05:10.081404 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.081387    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 22:05:10.086070 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.086054    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 17 22:05:10.087131 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.087114    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 17 22:05:10.087216 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.087147    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 17 22:05:10.101114 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.101092    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rvgtv\""
Jul 17 22:05:10.102354 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.102322    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 22:05:10.103592 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.103573    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-051ea5-openshift-default-dockercfg-5hzm5\""
Jul 17 22:05:10.103592 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.103583    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 17 22:05:10.107477 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.107458    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 22:05:10.119186 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.119165    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 22:05:10.119962 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.119941    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 17 22:05:10.126418 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.126396    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 22:05:10.135012 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.134994    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-x4mt8\""
Jul 17 22:05:10.155825 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.155804    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 22:05:10.156755 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.156740    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b86v4\""
Jul 17 22:05:10.163579 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.163563    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 22:05:10.165938 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.165920    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-d9p27\""
Jul 17 22:05:10.185838 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.185813    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 22:05:10.187179 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.187163    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 22:05:10.190160 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.190144    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 22:05:10.192225 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.192206    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5","odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh","odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9"]
Jul 17 22:05:10.192504 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.192492    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713" containerName="revoke-keys"
Jul 17 22:05:10.192543 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.192506    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713" containerName="revoke-keys"
Jul 17 22:05:10.192543 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.192516    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713" containerName="revoke-keys"
Jul 17 22:05:10.192543 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.192522    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713" containerName="revoke-keys"
Jul 17 22:05:10.192543 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.192542    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcb7c3f2-3475-4321-953d-7c36c0bf9faa" containerName="revoke-keys"
Jul 17 22:05:10.192671 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.192548    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcb7c3f2-3475-4321-953d-7c36c0bf9faa" containerName="revoke-keys"
Jul 17 22:05:10.192671 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.192562    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcb7c3f2-3475-4321-953d-7c36c0bf9faa" containerName="revoke-keys"
Jul 17 22:05:10.192671 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.192567    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcb7c3f2-3475-4321-953d-7c36c0bf9faa" containerName="revoke-keys"
Jul 17 22:05:10.192671 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.192611    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713" containerName="revoke-keys"
Jul 17 22:05:10.192671 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.192619    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713" containerName="revoke-keys"
Jul 17 22:05:10.192671 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.192624    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcb7c3f2-3475-4321-953d-7c36c0bf9faa" containerName="revoke-keys"
Jul 17 22:05:10.192671 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.192629    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcb7c3f2-3475-4321-953d-7c36c0bf9faa" containerName="revoke-keys"
Jul 17 22:05:10.192671 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.192636    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcb7c3f2-3475-4321-953d-7c36c0bf9faa" containerName="revoke-keys"
Jul 17 22:05:10.195344 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.195322    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5"
Jul 17 22:05:10.195634 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.195603    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713" containerName="revoke-keys"
Jul 17 22:05:10.195634 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.195630    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713" containerName="revoke-keys"
Jul 17 22:05:10.195787 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.195665    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcb7c3f2-3475-4321-953d-7c36c0bf9faa" containerName="revoke-keys"
Jul 17 22:05:10.195787 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.195679    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcb7c3f2-3475-4321-953d-7c36c0bf9faa" containerName="revoke-keys"
Jul 17 22:05:10.195885 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.195794    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd5a4a1-9e4e-4b70-b644-99cc6a7f9713" containerName="revoke-keys"
Jul 17 22:05:10.198477 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.198458    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh"
Jul 17 22:05:10.201217 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.201199    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9"
Jul 17 22:05:10.220567 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.220550    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 22:05:10.229003 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.228987    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 17 22:05:10.233391 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.233375    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 22:05:10.238626 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.238602    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/667bd155-d05a-49aa-b54d-946469be87e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh\" (UID: \"667bd155-d05a-49aa-b54d-946469be87e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh"
Jul 17 22:05:10.238808 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.238788    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af5a675-b3ec-4ccf-a167-b08132360788-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-ch4f9\" (UID: \"0af5a675-b3ec-4ccf-a167-b08132360788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9"
Jul 17 22:05:10.238984 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.238966    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d20353e0-f2fe-4a23-ba8c-c5594b2c82a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5\" (UID: \"d20353e0-f2fe-4a23-ba8c-c5594b2c82a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5"
Jul 17 22:05:10.256554 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.256530    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cmtnp\""
Jul 17 22:05:10.268070 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.268052    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 22:05:10.273830 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.273813    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 17 22:05:10.289509 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.289489    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 17 22:05:10.297048 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.297030    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 22:05:10.301740 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.301725    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 22:05:10.302960 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.302946    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 22:05:10.316664 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.316644    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sqgsr\""
Jul 17 22:05:10.324618 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.324595    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 17 22:05:10.325381 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.325363    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 22:05:10.335536 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.335520    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 22:05:10.339336 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.339318    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d20353e0-f2fe-4a23-ba8c-c5594b2c82a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5\" (UID: \"d20353e0-f2fe-4a23-ba8c-c5594b2c82a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5"
Jul 17 22:05:10.339403 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.339369    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/667bd155-d05a-49aa-b54d-946469be87e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh\" (UID: \"667bd155-d05a-49aa-b54d-946469be87e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh"
Jul 17 22:05:10.339403 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.339394    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af5a675-b3ec-4ccf-a167-b08132360788-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-ch4f9\" (UID: \"0af5a675-b3ec-4ccf-a167-b08132360788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9"
Jul 17 22:05:10.339960 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.339943    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d20353e0-f2fe-4a23-ba8c-c5594b2c82a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5\" (UID: \"d20353e0-f2fe-4a23-ba8c-c5594b2c82a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5"
Jul 17 22:05:10.340143 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.340123    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/667bd155-d05a-49aa-b54d-946469be87e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh\" (UID: \"667bd155-d05a-49aa-b54d-946469be87e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh"
Jul 17 22:05:10.340179 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.340123    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af5a675-b3ec-4ccf-a167-b08132360788-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-ch4f9\" (UID: \"0af5a675-b3ec-4ccf-a167-b08132360788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9"
Jul 17 22:05:10.347576 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.347561    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 22:05:10.350388 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.350375    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 22:05:10.353633 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.353617    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 22:05:10.356927 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.356906    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 22:05:10.361378 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.361354    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-przjx\""
Jul 17 22:05:10.364242 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.364206    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 22:05:10.364433 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.364421    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tppfj\""
Jul 17 22:05:10.365420 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.365405    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 22:05:10.366935 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.366920    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 22:05:10.367614 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.367600    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 17 22:05:10.374207 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.374185    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dnvbk\""
Jul 17 22:05:10.382960 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.382941    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j8vps\""
Jul 17 22:05:10.389734 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.389713    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 22:05:10.396348 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.396328    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 17 22:05:10.396593 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.396579    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 17 22:05:10.413293 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.413261    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 22:05:10.419872 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.419856    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 22:05:10.426296 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.426273    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 22:05:10.433784 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.433769    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 22:05:10.439392 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.439378    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 22:05:10.505928 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.505859    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5"
Jul 17 22:05:10.516456 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.516429    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh"
Jul 17 22:05:10.518725 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.518694    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 22:05:10.519044 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.519027    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9"
Jul 17 22:05:10.539584 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.539559    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-szcqz\""
Jul 17 22:05:10.539717 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.539671    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 17 22:05:10.546497 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.546474    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-st79t\""
Jul 17 22:05:10.561620 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.561596    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 17 22:05:10.562966 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.562903    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 22:05:10.563482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.563248    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 22:05:10.581988 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.581770    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-q6bg6\""
Jul 17 22:05:10.596965 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.596755    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ltpgv\""
Jul 17 22:05:10.598576 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.598317    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 22:05:10.603576 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.603555    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 22:05:10.614329 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.613519    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rdmx5\""
Jul 17 22:05:10.614329 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.613529    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mcghg\""
Jul 17 22:05:10.622800 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.622629    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 22:05:10.624014 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.623993    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:05:10.636995 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.636939    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 22:05:10.637815 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.637797    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 22:05:10.646504 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.643639    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 17 22:05:10.651885 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.651794    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 17 22:05:10.656005 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.655982    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 17 22:05:10.663695 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.663673    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bnb28\""
Jul 17 22:05:10.678681 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.678626    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5"]
Jul 17 22:05:10.683958 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.683626    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8vkwx\""
Jul 17 22:05:10.684075 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.683995    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 22:05:10.704488 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.704388    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 22:05:10.704651 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.704634    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 22:05:10.706164 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.706140    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 22:05:10.707089 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.707071    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-sz7w2\""
Jul 17 22:05:10.708982 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.708959    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 17 22:05:10.719716 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.719695    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 22:05:10.730842 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.730823    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 22:05:10.758121 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.758053    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 22:05:10.838566 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.838531    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5" event={"ID":"d20353e0-f2fe-4a23-ba8c-c5594b2c82a2","Type":"ContainerStarted","Data":"84c9b363a200a3b05461526b94900e33fb25355c4c719448ea91716e52bcec24"}
Jul 17 22:05:10.903224 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.903186    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh"]
Jul 17 22:05:10.905998 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:10.905976    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9"]
Jul 17 22:05:10.906123 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:05:10.906103    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667bd155_d05a_49aa_b54d_946469be87e5.slice/crio-3d957aaf86c2b3e74e770120068bbc129ceacb6ebacb05940f9ac616ae2f4c59 WatchSource:0}: Error finding container 3d957aaf86c2b3e74e770120068bbc129ceacb6ebacb05940f9ac616ae2f4c59: Status 404 returned error can't find the container with id 3d957aaf86c2b3e74e770120068bbc129ceacb6ebacb05940f9ac616ae2f4c59
Jul 17 22:05:10.908043 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:05:10.908026    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af5a675_b3ec_4ccf_a167_b08132360788.slice/crio-e70a7bfd0128686b30a757a98d6770d538306c914ad65197fff82b03eef04817 WatchSource:0}: Error finding container e70a7bfd0128686b30a757a98d6770d538306c914ad65197fff82b03eef04817: Status 404 returned error can't find the container with id e70a7bfd0128686b30a757a98d6770d538306c914ad65197fff82b03eef04817
Jul 17 22:05:11.246484 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.246458    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6"]
Jul 17 22:05:11.389121 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.389092    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6"
Jul 17 22:05:11.446912 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.446867    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee26ff1d-7318-409f-81e3-cb08d3f575f1-maas-api-service-ca\") pod \"ee26ff1d-7318-409f-81e3-cb08d3f575f1\" (UID: \"ee26ff1d-7318-409f-81e3-cb08d3f575f1\") "
Jul 17 22:05:11.447808 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.447453    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee26ff1d-7318-409f-81e3-cb08d3f575f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ee26ff1d-7318-409f-81e3-cb08d3f575f1" (UID: "ee26ff1d-7318-409f-81e3-cb08d3f575f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:05:11.547865 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.547835    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee26ff1d-7318-409f-81e3-cb08d3f575f1-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:05:11.844401 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.844370    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6"
Jul 17 22:05:11.844565 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.844369    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6" event={"ID":"ee26ff1d-7318-409f-81e3-cb08d3f575f1","Type":"ContainerDied","Data":"12750ecd365eaf29ebada14944f82282d824a260c0816028f2f9b906b1bb7e13"}
Jul 17 22:05:11.844565 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.844503    2573 scope.go:117] "RemoveContainer" containerID="c6b742006b1512d40df765f5350b91e8ab8fffc0b97a5fb931566e575b0de041"
Jul 17 22:05:11.845974 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.845938    2573 generic.go:356] "Generic (PLEG): container finished" podID="667bd155-d05a-49aa-b54d-946469be87e5" containerID="2e7ecb6932da9d4155fa58c0cf4f8738c2657ae6539a26a85bf275caf2baf178" exitCode=6
Jul 17 22:05:11.846084 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.846032    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh" event={"ID":"667bd155-d05a-49aa-b54d-946469be87e5","Type":"ContainerDied","Data":"2e7ecb6932da9d4155fa58c0cf4f8738c2657ae6539a26a85bf275caf2baf178"}
Jul 17 22:05:11.846084 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.846074    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh" event={"ID":"667bd155-d05a-49aa-b54d-946469be87e5","Type":"ContainerStarted","Data":"3d957aaf86c2b3e74e770120068bbc129ceacb6ebacb05940f9ac616ae2f4c59"}
Jul 17 22:05:11.846193 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.846184    2573 scope.go:117] "RemoveContainer" containerID="2e7ecb6932da9d4155fa58c0cf4f8738c2657ae6539a26a85bf275caf2baf178"
Jul 17 22:05:11.847698 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.847671    2573 generic.go:356] "Generic (PLEG): container finished" podID="d20353e0-f2fe-4a23-ba8c-c5594b2c82a2" containerID="3c3220846b985ca4cfc4f091d9e117963372294d5ad3f385275dfff87b0162de" exitCode=6
Jul 17 22:05:11.847810 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.847741    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5" event={"ID":"d20353e0-f2fe-4a23-ba8c-c5594b2c82a2","Type":"ContainerDied","Data":"3c3220846b985ca4cfc4f091d9e117963372294d5ad3f385275dfff87b0162de"}
Jul 17 22:05:11.848164 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.848141    2573 scope.go:117] "RemoveContainer" containerID="3c3220846b985ca4cfc4f091d9e117963372294d5ad3f385275dfff87b0162de"
Jul 17 22:05:11.849556 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.849530    2573 generic.go:356] "Generic (PLEG): container finished" podID="0af5a675-b3ec-4ccf-a167-b08132360788" containerID="a481fa4230b9f3cddbad019330c59be04161cd18971f28488708f453813dcdf9" exitCode=6
Jul 17 22:05:11.849671 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.849596    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9" event={"ID":"0af5a675-b3ec-4ccf-a167-b08132360788","Type":"ContainerDied","Data":"a481fa4230b9f3cddbad019330c59be04161cd18971f28488708f453813dcdf9"}
Jul 17 22:05:11.849671 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.849635    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9" event={"ID":"0af5a675-b3ec-4ccf-a167-b08132360788","Type":"ContainerStarted","Data":"e70a7bfd0128686b30a757a98d6770d538306c914ad65197fff82b03eef04817"}
Jul 17 22:05:11.849923 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.849798    2573 scope.go:117] "RemoveContainer" containerID="a481fa4230b9f3cddbad019330c59be04161cd18971f28488708f453813dcdf9"
Jul 17 22:05:11.907441 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.907414    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6"]
Jul 17 22:05:11.996171 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:11.996139    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-drmc6"]
Jul 17 22:05:12.530615 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:12.530582    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1" path="/var/lib/kubelet/pods/ee26ff1d-7318-409f-81e3-cb08d3f575f1/volumes"
Jul 17 22:05:12.855460 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:12.855392    2573 generic.go:356] "Generic (PLEG): container finished" podID="0af5a675-b3ec-4ccf-a167-b08132360788" containerID="c03f787639b6edc9bcf60ee740e65fdbdd22a611a79da19f10d9303d6a04e907" exitCode=6
Jul 17 22:05:12.855622 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:12.855488    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9" event={"ID":"0af5a675-b3ec-4ccf-a167-b08132360788","Type":"ContainerDied","Data":"c03f787639b6edc9bcf60ee740e65fdbdd22a611a79da19f10d9303d6a04e907"}
Jul 17 22:05:12.855622 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:12.855557    2573 scope.go:117] "RemoveContainer" containerID="a481fa4230b9f3cddbad019330c59be04161cd18971f28488708f453813dcdf9"
Jul 17 22:05:12.855745 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:12.855729    2573 scope.go:117] "RemoveContainer" containerID="c03f787639b6edc9bcf60ee740e65fdbdd22a611a79da19f10d9303d6a04e907"
Jul 17 22:05:12.855994 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:05:12.855963    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-ch4f9_odh-ai-gateway-infra(0af5a675-b3ec-4ccf-a167-b08132360788)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9" podUID="0af5a675-b3ec-4ccf-a167-b08132360788"
Jul 17 22:05:12.858127 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:12.858103    2573 generic.go:356] "Generic (PLEG): container finished" podID="667bd155-d05a-49aa-b54d-946469be87e5" containerID="98e1e447ad5674967d96357093465291c54c212dfc7665c381c8ab2424347c87" exitCode=6
Jul 17 22:05:12.858259 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:12.858184    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh" event={"ID":"667bd155-d05a-49aa-b54d-946469be87e5","Type":"ContainerDied","Data":"98e1e447ad5674967d96357093465291c54c212dfc7665c381c8ab2424347c87"}
Jul 17 22:05:12.858375 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:12.858361    2573 scope.go:117] "RemoveContainer" containerID="98e1e447ad5674967d96357093465291c54c212dfc7665c381c8ab2424347c87"
Jul 17 22:05:12.858556 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:05:12.858534    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh_odh-ai-gateway-infra(667bd155-d05a-49aa-b54d-946469be87e5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh" podUID="667bd155-d05a-49aa-b54d-946469be87e5"
Jul 17 22:05:12.859884 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:12.859862    2573 generic.go:356] "Generic (PLEG): container finished" podID="d20353e0-f2fe-4a23-ba8c-c5594b2c82a2" containerID="0e4fa5bdbb4eb8af0125bac4081b5c80ff3ade463b97b9d502c18c4e058f77ec" exitCode=6
Jul 17 22:05:12.859975 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:12.859893    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5" event={"ID":"d20353e0-f2fe-4a23-ba8c-c5594b2c82a2","Type":"ContainerDied","Data":"0e4fa5bdbb4eb8af0125bac4081b5c80ff3ade463b97b9d502c18c4e058f77ec"}
Jul 17 22:05:12.860081 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:12.860070    2573 scope.go:117] "RemoveContainer" containerID="0e4fa5bdbb4eb8af0125bac4081b5c80ff3ade463b97b9d502c18c4e058f77ec"
Jul 17 22:05:12.860225 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:05:12.860210    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5_odh-ai-gateway-infra(d20353e0-f2fe-4a23-ba8c-c5594b2c82a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5" podUID="d20353e0-f2fe-4a23-ba8c-c5594b2c82a2"
Jul 17 22:05:12.866249 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:12.866232    2573 scope.go:117] "RemoveContainer" containerID="2e7ecb6932da9d4155fa58c0cf4f8738c2657ae6539a26a85bf275caf2baf178"
Jul 17 22:05:12.877032 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:12.877013    2573 scope.go:117] "RemoveContainer" containerID="3c3220846b985ca4cfc4f091d9e117963372294d5ad3f385275dfff87b0162de"
Jul 17 22:05:13.866991 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:13.866965    2573 scope.go:117] "RemoveContainer" containerID="98e1e447ad5674967d96357093465291c54c212dfc7665c381c8ab2424347c87"
Jul 17 22:05:13.867434 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:05:13.867127    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh_odh-ai-gateway-infra(667bd155-d05a-49aa-b54d-946469be87e5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh" podUID="667bd155-d05a-49aa-b54d-946469be87e5"
Jul 17 22:05:13.868483 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:13.868461    2573 scope.go:117] "RemoveContainer" containerID="0e4fa5bdbb4eb8af0125bac4081b5c80ff3ade463b97b9d502c18c4e058f77ec"
Jul 17 22:05:13.868616 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:05:13.868597    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5_odh-ai-gateway-infra(d20353e0-f2fe-4a23-ba8c-c5594b2c82a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5" podUID="d20353e0-f2fe-4a23-ba8c-c5594b2c82a2"
Jul 17 22:05:13.869845 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:13.869826    2573 scope.go:117] "RemoveContainer" containerID="c03f787639b6edc9bcf60ee740e65fdbdd22a611a79da19f10d9303d6a04e907"
Jul 17 22:05:13.869968 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:05:13.869952    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-ch4f9_odh-ai-gateway-infra(0af5a675-b3ec-4ccf-a167-b08132360788)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9" podUID="0af5a675-b3ec-4ccf-a167-b08132360788"
Jul 17 22:05:14.690960 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:14.690930    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 22:05:14.691116 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:14.690968    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 22:05:14.691116 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:14.690982    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeHasSufficientPID"
Jul 17 22:05:14.691116 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:14.690997    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-246.ec2.internal" event="NodeReady"
Jul 17 22:05:25.524278 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:25.524248    2573 scope.go:117] "RemoveContainer" containerID="98e1e447ad5674967d96357093465291c54c212dfc7665c381c8ab2424347c87"
Jul 17 22:05:25.524673 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:25.524390    2573 scope.go:117] "RemoveContainer" containerID="c03f787639b6edc9bcf60ee740e65fdbdd22a611a79da19f10d9303d6a04e907"
Jul 17 22:05:25.923065 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:25.923031    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh" event={"ID":"667bd155-d05a-49aa-b54d-946469be87e5","Type":"ContainerStarted","Data":"273dbae26f1b3395408fa231b66a304d6ed062033be33bb7fd1c001de26b7ce6"}
Jul 17 22:05:25.924674 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:25.924652    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9" event={"ID":"0af5a675-b3ec-4ccf-a167-b08132360788","Type":"ContainerStarted","Data":"73bd5ffb9e3a08ae92ac27756eb85761b8b33c13a8fb008fb6540269991a463b"}
Jul 17 22:05:25.938745 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:25.938545    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh" podStartSLOduration=40.938528442 podStartE2EDuration="40.938528442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:05:25.937893161 +0000 UTC m=+3425.902013770" watchObservedRunningTime="2026-07-17 22:05:25.938528442 +0000 UTC m=+3425.902649050"
Jul 17 22:05:26.524911 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:26.524830    2573 scope.go:117] "RemoveContainer" containerID="0e4fa5bdbb4eb8af0125bac4081b5c80ff3ade463b97b9d502c18c4e058f77ec"
Jul 17 22:05:26.931373 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:26.931326    2573 generic.go:356] "Generic (PLEG): container finished" podID="667bd155-d05a-49aa-b54d-946469be87e5" containerID="273dbae26f1b3395408fa231b66a304d6ed062033be33bb7fd1c001de26b7ce6" exitCode=6
Jul 17 22:05:26.931516 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:26.931395    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh" event={"ID":"667bd155-d05a-49aa-b54d-946469be87e5","Type":"ContainerDied","Data":"273dbae26f1b3395408fa231b66a304d6ed062033be33bb7fd1c001de26b7ce6"}
Jul 17 22:05:26.931516 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:26.931456    2573 scope.go:117] "RemoveContainer" containerID="98e1e447ad5674967d96357093465291c54c212dfc7665c381c8ab2424347c87"
Jul 17 22:05:26.931809 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:26.931785    2573 scope.go:117] "RemoveContainer" containerID="273dbae26f1b3395408fa231b66a304d6ed062033be33bb7fd1c001de26b7ce6"
Jul 17 22:05:26.932043 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:05:26.932013    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh_odh-ai-gateway-infra(667bd155-d05a-49aa-b54d-946469be87e5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh" podUID="667bd155-d05a-49aa-b54d-946469be87e5"
Jul 17 22:05:26.933385 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:26.933364    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5" event={"ID":"d20353e0-f2fe-4a23-ba8c-c5594b2c82a2","Type":"ContainerStarted","Data":"8291b0a972c2d50c7890151c69d6aa78497e596773f7ebbea310ce22130d58b5"}
Jul 17 22:05:26.935062 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:26.935042    2573 generic.go:356] "Generic (PLEG): container finished" podID="0af5a675-b3ec-4ccf-a167-b08132360788" containerID="73bd5ffb9e3a08ae92ac27756eb85761b8b33c13a8fb008fb6540269991a463b" exitCode=6
Jul 17 22:05:26.935152 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:26.935140    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9" event={"ID":"0af5a675-b3ec-4ccf-a167-b08132360788","Type":"ContainerDied","Data":"73bd5ffb9e3a08ae92ac27756eb85761b8b33c13a8fb008fb6540269991a463b"}
Jul 17 22:05:26.935397 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:26.935384    2573 scope.go:117] "RemoveContainer" containerID="73bd5ffb9e3a08ae92ac27756eb85761b8b33c13a8fb008fb6540269991a463b"
Jul 17 22:05:26.935545 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:05:26.935521    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-ch4f9_odh-ai-gateway-infra(0af5a675-b3ec-4ccf-a167-b08132360788)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9" podUID="0af5a675-b3ec-4ccf-a167-b08132360788"
Jul 17 22:05:26.942033 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:26.942014    2573 scope.go:117] "RemoveContainer" containerID="c03f787639b6edc9bcf60ee740e65fdbdd22a611a79da19f10d9303d6a04e907"
Jul 17 22:05:26.945468 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:26.945424    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9" podStartSLOduration=35.945374355 podStartE2EDuration="35.945374355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:05:25.951685637 +0000 UTC m=+3425.915806252" watchObservedRunningTime="2026-07-17 22:05:26.945374355 +0000 UTC m=+3426.909494958"
Jul 17 22:05:26.953735 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:26.953716    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh"]
Jul 17 22:05:26.962654 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:26.962633    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9"]
Jul 17 22:05:26.963460 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:26.963423    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5" podStartSLOduration=58.963412395 podStartE2EDuration="58.963412395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:05:26.960616242 +0000 UTC m=+3426.924736854" watchObservedRunningTime="2026-07-17 22:05:26.963412395 +0000 UTC m=+3426.927533005"
Jul 17 22:05:27.942756 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:27.942727    2573 generic.go:356] "Generic (PLEG): container finished" podID="d20353e0-f2fe-4a23-ba8c-c5594b2c82a2" containerID="8291b0a972c2d50c7890151c69d6aa78497e596773f7ebbea310ce22130d58b5" exitCode=6
Jul 17 22:05:27.943087 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:27.942797    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5" event={"ID":"d20353e0-f2fe-4a23-ba8c-c5594b2c82a2","Type":"ContainerDied","Data":"8291b0a972c2d50c7890151c69d6aa78497e596773f7ebbea310ce22130d58b5"}
Jul 17 22:05:27.943087 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:27.942852    2573 scope.go:117] "RemoveContainer" containerID="0e4fa5bdbb4eb8af0125bac4081b5c80ff3ade463b97b9d502c18c4e058f77ec"
Jul 17 22:05:27.943204 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:27.943111    2573 scope.go:117] "RemoveContainer" containerID="8291b0a972c2d50c7890151c69d6aa78497e596773f7ebbea310ce22130d58b5"
Jul 17 22:05:27.943335 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:05:27.943315    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5_odh-ai-gateway-infra(d20353e0-f2fe-4a23-ba8c-c5594b2c82a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5" podUID="d20353e0-f2fe-4a23-ba8c-c5594b2c82a2"
Jul 17 22:05:27.977832 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:27.977803    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5"]
Jul 17 22:05:28.080730 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:28.080707    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh"
Jul 17 22:05:28.101344 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:28.101323    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9"
Jul 17 22:05:28.187575 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:28.187549    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af5a675-b3ec-4ccf-a167-b08132360788-maas-api-service-ca\") pod \"0af5a675-b3ec-4ccf-a167-b08132360788\" (UID: \"0af5a675-b3ec-4ccf-a167-b08132360788\") "
Jul 17 22:05:28.187689 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:28.187615    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/667bd155-d05a-49aa-b54d-946469be87e5-maas-api-service-ca\") pod \"667bd155-d05a-49aa-b54d-946469be87e5\" (UID: \"667bd155-d05a-49aa-b54d-946469be87e5\") "
Jul 17 22:05:28.187887 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:28.187868    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667bd155-d05a-49aa-b54d-946469be87e5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "667bd155-d05a-49aa-b54d-946469be87e5" (UID: "667bd155-d05a-49aa-b54d-946469be87e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:05:28.187929 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:28.187904    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af5a675-b3ec-4ccf-a167-b08132360788-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0af5a675-b3ec-4ccf-a167-b08132360788" (UID: "0af5a675-b3ec-4ccf-a167-b08132360788"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:05:28.288898 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:28.288847    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/667bd155-d05a-49aa-b54d-946469be87e5-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:05:28.288898 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:28.288869    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af5a675-b3ec-4ccf-a167-b08132360788-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:05:28.951249 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:28.951218    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh"
Jul 17 22:05:28.951249 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:28.951233    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh" event={"ID":"667bd155-d05a-49aa-b54d-946469be87e5","Type":"ContainerDied","Data":"3d957aaf86c2b3e74e770120068bbc129ceacb6ebacb05940f9ac616ae2f4c59"}
Jul 17 22:05:28.951828 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:28.951314    2573 scope.go:117] "RemoveContainer" containerID="273dbae26f1b3395408fa231b66a304d6ed062033be33bb7fd1c001de26b7ce6"
Jul 17 22:05:28.954758 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:28.954722    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9" event={"ID":"0af5a675-b3ec-4ccf-a167-b08132360788","Type":"ContainerDied","Data":"e70a7bfd0128686b30a757a98d6770d538306c914ad65197fff82b03eef04817"}
Jul 17 22:05:28.954758 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:28.954749    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9"
Jul 17 22:05:28.962229 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:28.962205    2573 scope.go:117] "RemoveContainer" containerID="73bd5ffb9e3a08ae92ac27756eb85761b8b33c13a8fb008fb6540269991a463b"
Jul 17 22:05:28.969754 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:28.969725    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh"]
Jul 17 22:05:28.975870 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:28.975838    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-mwcvh"]
Jul 17 22:05:28.997428 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:28.997403    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9"]
Jul 17 22:05:29.000997 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:29.000975    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-ch4f9"]
Jul 17 22:05:29.077603 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:29.077578    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5"
Jul 17 22:05:29.196536 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:29.196497    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d20353e0-f2fe-4a23-ba8c-c5594b2c82a2-maas-api-service-ca\") pod \"d20353e0-f2fe-4a23-ba8c-c5594b2c82a2\" (UID: \"d20353e0-f2fe-4a23-ba8c-c5594b2c82a2\") "
Jul 17 22:05:29.196850 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:29.196826    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20353e0-f2fe-4a23-ba8c-c5594b2c82a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d20353e0-f2fe-4a23-ba8c-c5594b2c82a2" (UID: "d20353e0-f2fe-4a23-ba8c-c5594b2c82a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:05:29.297792 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:29.297721    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d20353e0-f2fe-4a23-ba8c-c5594b2c82a2-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:05:29.961079 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:29.961048    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5"
Jul 17 22:05:29.961601 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:29.961049    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5" event={"ID":"d20353e0-f2fe-4a23-ba8c-c5594b2c82a2","Type":"ContainerDied","Data":"84c9b363a200a3b05461526b94900e33fb25355c4c719448ea91716e52bcec24"}
Jul 17 22:05:29.961601 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:29.961205    2573 scope.go:117] "RemoveContainer" containerID="8291b0a972c2d50c7890151c69d6aa78497e596773f7ebbea310ce22130d58b5"
Jul 17 22:05:29.981339 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:29.981317    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5"]
Jul 17 22:05:29.984690 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:29.984670    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-dljj5"]
Jul 17 22:05:30.531637 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:30.531605    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0af5a675-b3ec-4ccf-a167-b08132360788" path="/var/lib/kubelet/pods/0af5a675-b3ec-4ccf-a167-b08132360788/volumes"
Jul 17 22:05:30.531944 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:30.531931    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="667bd155-d05a-49aa-b54d-946469be87e5" path="/var/lib/kubelet/pods/667bd155-d05a-49aa-b54d-946469be87e5/volumes"
Jul 17 22:05:30.532220 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:30.532209    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d20353e0-f2fe-4a23-ba8c-c5594b2c82a2" path="/var/lib/kubelet/pods/d20353e0-f2fe-4a23-ba8c-c5594b2c82a2/volumes"
Jul 17 22:05:50.104095 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104061    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh"]
Jul 17 22:05:50.104481 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104371    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1" containerName="revoke-keys"
Jul 17 22:05:50.104481 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104383    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1" containerName="revoke-keys"
Jul 17 22:05:50.104481 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104392    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20353e0-f2fe-4a23-ba8c-c5594b2c82a2" containerName="revoke-keys"
Jul 17 22:05:50.104481 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104398    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20353e0-f2fe-4a23-ba8c-c5594b2c82a2" containerName="revoke-keys"
Jul 17 22:05:50.104481 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104406    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1" containerName="revoke-keys"
Jul 17 22:05:50.104481 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104412    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1" containerName="revoke-keys"
Jul 17 22:05:50.104481 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104418    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af5a675-b3ec-4ccf-a167-b08132360788" containerName="revoke-keys"
Jul 17 22:05:50.104481 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104424    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af5a675-b3ec-4ccf-a167-b08132360788" containerName="revoke-keys"
Jul 17 22:05:50.104481 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104435    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1" containerName="revoke-keys"
Jul 17 22:05:50.104481 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104441    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1" containerName="revoke-keys"
Jul 17 22:05:50.104481 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104447    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af5a675-b3ec-4ccf-a167-b08132360788" containerName="revoke-keys"
Jul 17 22:05:50.104481 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104452    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af5a675-b3ec-4ccf-a167-b08132360788" containerName="revoke-keys"
Jul 17 22:05:50.104481 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104457    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667bd155-d05a-49aa-b54d-946469be87e5" containerName="revoke-keys"
Jul 17 22:05:50.104481 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104462    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="667bd155-d05a-49aa-b54d-946469be87e5" containerName="revoke-keys"
Jul 17 22:05:50.104481 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104472    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667bd155-d05a-49aa-b54d-946469be87e5" containerName="revoke-keys"
Jul 17 22:05:50.104481 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104477    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="667bd155-d05a-49aa-b54d-946469be87e5" containerName="revoke-keys"
Jul 17 22:05:50.104481 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104482    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="667bd155-d05a-49aa-b54d-946469be87e5" containerName="revoke-keys"
Jul 17 22:05:50.104481 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104487    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="667bd155-d05a-49aa-b54d-946469be87e5" containerName="revoke-keys"
Jul 17 22:05:50.105039 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104493    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20353e0-f2fe-4a23-ba8c-c5594b2c82a2" containerName="revoke-keys"
Jul 17 22:05:50.105039 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104497    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20353e0-f2fe-4a23-ba8c-c5594b2c82a2" containerName="revoke-keys"
Jul 17 22:05:50.105039 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104542    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20353e0-f2fe-4a23-ba8c-c5594b2c82a2" containerName="revoke-keys"
Jul 17 22:05:50.105039 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104548    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af5a675-b3ec-4ccf-a167-b08132360788" containerName="revoke-keys"
Jul 17 22:05:50.105039 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104554    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af5a675-b3ec-4ccf-a167-b08132360788" containerName="revoke-keys"
Jul 17 22:05:50.105039 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104558    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1" containerName="revoke-keys"
Jul 17 22:05:50.105039 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104565    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1" containerName="revoke-keys"
Jul 17 22:05:50.105039 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104570    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="667bd155-d05a-49aa-b54d-946469be87e5" containerName="revoke-keys"
Jul 17 22:05:50.105039 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104575    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="667bd155-d05a-49aa-b54d-946469be87e5" containerName="revoke-keys"
Jul 17 22:05:50.105039 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104581    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1" containerName="revoke-keys"
Jul 17 22:05:50.105039 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104587    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af5a675-b3ec-4ccf-a167-b08132360788" containerName="revoke-keys"
Jul 17 22:05:50.105039 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104592    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="667bd155-d05a-49aa-b54d-946469be87e5" containerName="revoke-keys"
Jul 17 22:05:50.105039 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104597    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20353e0-f2fe-4a23-ba8c-c5594b2c82a2" containerName="revoke-keys"
Jul 17 22:05:50.105039 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104602    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1" containerName="revoke-keys"
Jul 17 22:05:50.105039 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.104606    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20353e0-f2fe-4a23-ba8c-c5594b2c82a2" containerName="revoke-keys"
Jul 17 22:05:50.107660 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.107643    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh"
Jul 17 22:05:50.109663 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.109638    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:05:50.109773 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.109660    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 22:05:50.115629 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.115605    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh"]
Jul 17 22:05:50.256368 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.256337    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35991ade-70b6-4dd3-a502-fc26ea1579f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh\" (UID: \"35991ade-70b6-4dd3-a502-fc26ea1579f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh"
Jul 17 22:05:50.356782 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.356699    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35991ade-70b6-4dd3-a502-fc26ea1579f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh\" (UID: \"35991ade-70b6-4dd3-a502-fc26ea1579f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh"
Jul 17 22:05:50.357344 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.357323    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35991ade-70b6-4dd3-a502-fc26ea1579f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh\" (UID: \"35991ade-70b6-4dd3-a502-fc26ea1579f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh"
Jul 17 22:05:50.418498 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.418464    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh"
Jul 17 22:05:50.541585 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:50.541565    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh"]
Jul 17 22:05:50.543596 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:05:50.543561    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35991ade_70b6_4dd3_a502_fc26ea1579f2.slice/crio-a1bd4a6ec12c55a6a57207da90c401ff1dd8159f83440a44fc6eee2a88bfc7e9 WatchSource:0}: Error finding container a1bd4a6ec12c55a6a57207da90c401ff1dd8159f83440a44fc6eee2a88bfc7e9: Status 404 returned error can't find the container with id a1bd4a6ec12c55a6a57207da90c401ff1dd8159f83440a44fc6eee2a88bfc7e9
Jul 17 22:05:51.060325 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:51.060235    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh" event={"ID":"35991ade-70b6-4dd3-a502-fc26ea1579f2","Type":"ContainerStarted","Data":"8254c131adc96cba8e614bc86a215d2e63d0f2b269ded36d37ea05b6ad789e83"}
Jul 17 22:05:51.060325 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:51.060299    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh" event={"ID":"35991ade-70b6-4dd3-a502-fc26ea1579f2","Type":"ContainerStarted","Data":"a1bd4a6ec12c55a6a57207da90c401ff1dd8159f83440a44fc6eee2a88bfc7e9"}
Jul 17 22:05:51.073957 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:51.073912    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh" podStartSLOduration=1.073897723 podStartE2EDuration="1.073897723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:05:51.072607496 +0000 UTC m=+3451.036728107" watchObservedRunningTime="2026-07-17 22:05:51.073897723 +0000 UTC m=+3451.038018332"
Jul 17 22:05:52.066167 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:52.066129    2573 generic.go:356] "Generic (PLEG): container finished" podID="35991ade-70b6-4dd3-a502-fc26ea1579f2" containerID="8254c131adc96cba8e614bc86a215d2e63d0f2b269ded36d37ea05b6ad789e83" exitCode=6
Jul 17 22:05:52.066684 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:52.066200    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh" event={"ID":"35991ade-70b6-4dd3-a502-fc26ea1579f2","Type":"ContainerDied","Data":"8254c131adc96cba8e614bc86a215d2e63d0f2b269ded36d37ea05b6ad789e83"}
Jul 17 22:05:52.066684 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:52.066481    2573 scope.go:117] "RemoveContainer" containerID="8254c131adc96cba8e614bc86a215d2e63d0f2b269ded36d37ea05b6ad789e83"
Jul 17 22:05:53.072222 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:53.072135    2573 generic.go:356] "Generic (PLEG): container finished" podID="35991ade-70b6-4dd3-a502-fc26ea1579f2" containerID="05482ac6a32ba21718cb4b0ee91410c449f8872f98dd5615a8fa69fb68d72383" exitCode=6
Jul 17 22:05:53.072690 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:53.072223    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh" event={"ID":"35991ade-70b6-4dd3-a502-fc26ea1579f2","Type":"ContainerDied","Data":"05482ac6a32ba21718cb4b0ee91410c449f8872f98dd5615a8fa69fb68d72383"}
Jul 17 22:05:53.072690 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:53.072305    2573 scope.go:117] "RemoveContainer" containerID="8254c131adc96cba8e614bc86a215d2e63d0f2b269ded36d37ea05b6ad789e83"
Jul 17 22:05:53.072690 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:53.072407    2573 scope.go:117] "RemoveContainer" containerID="05482ac6a32ba21718cb4b0ee91410c449f8872f98dd5615a8fa69fb68d72383"
Jul 17 22:05:53.072690 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:05:53.072579    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh_odh-ai-gateway-infra(35991ade-70b6-4dd3-a502-fc26ea1579f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh" podUID="35991ade-70b6-4dd3-a502-fc26ea1579f2"
Jul 17 22:05:54.077721 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:05:54.077696    2573 scope.go:117] "RemoveContainer" containerID="05482ac6a32ba21718cb4b0ee91410c449f8872f98dd5615a8fa69fb68d72383"
Jul 17 22:05:54.078101 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:05:54.077880    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh_odh-ai-gateway-infra(35991ade-70b6-4dd3-a502-fc26ea1579f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh" podUID="35991ade-70b6-4dd3-a502-fc26ea1579f2"
Jul 17 22:06:01.252024 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:01.251984    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9"]
Jul 17 22:06:01.252651 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:01.252434    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1" containerName="revoke-keys"
Jul 17 22:06:01.252651 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:01.252453    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee26ff1d-7318-409f-81e3-cb08d3f575f1" containerName="revoke-keys"
Jul 17 22:06:01.252651 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:01.252465    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af5a675-b3ec-4ccf-a167-b08132360788" containerName="revoke-keys"
Jul 17 22:06:01.252651 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:01.252474    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af5a675-b3ec-4ccf-a167-b08132360788" containerName="revoke-keys"
Jul 17 22:06:01.252651 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:01.252500    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20353e0-f2fe-4a23-ba8c-c5594b2c82a2" containerName="revoke-keys"
Jul 17 22:06:01.252651 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:01.252509    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20353e0-f2fe-4a23-ba8c-c5594b2c82a2" containerName="revoke-keys"
Jul 17 22:06:01.256499 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:01.256479    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9"
Jul 17 22:06:01.263137 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:01.263109    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9"]
Jul 17 22:06:01.441307 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:01.441258    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8f875b-830d-4ac5-a5df-d312362d74af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-jtqx9\" (UID: \"cd8f875b-830d-4ac5-a5df-d312362d74af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9"
Jul 17 22:06:01.542016 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:01.541940    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8f875b-830d-4ac5-a5df-d312362d74af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-jtqx9\" (UID: \"cd8f875b-830d-4ac5-a5df-d312362d74af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9"
Jul 17 22:06:01.542507 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:01.542491    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8f875b-830d-4ac5-a5df-d312362d74af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-jtqx9\" (UID: \"cd8f875b-830d-4ac5-a5df-d312362d74af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9"
Jul 17 22:06:01.568055 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:01.568034    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9"
Jul 17 22:06:01.897836 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:01.897812    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9"]
Jul 17 22:06:01.899686 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:06:01.899658    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8f875b_830d_4ac5_a5df_d312362d74af.slice/crio-7187dae51386df3063c73d857219c3170df1df0d07b49d6876daa02f844ec136 WatchSource:0}: Error finding container 7187dae51386df3063c73d857219c3170df1df0d07b49d6876daa02f844ec136: Status 404 returned error can't find the container with id 7187dae51386df3063c73d857219c3170df1df0d07b49d6876daa02f844ec136
Jul 17 22:06:02.118513 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:02.118482    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9" event={"ID":"cd8f875b-830d-4ac5-a5df-d312362d74af","Type":"ContainerStarted","Data":"7187dae51386df3063c73d857219c3170df1df0d07b49d6876daa02f844ec136"}
Jul 17 22:06:03.123869 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:03.123833    2573 generic.go:356] "Generic (PLEG): container finished" podID="cd8f875b-830d-4ac5-a5df-d312362d74af" containerID="333450a1711921aebbf554a4dc9bf972e65bbb56497ba29f81e6a84967dfebdf" exitCode=6
Jul 17 22:06:03.124257 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:03.123917    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9" event={"ID":"cd8f875b-830d-4ac5-a5df-d312362d74af","Type":"ContainerDied","Data":"333450a1711921aebbf554a4dc9bf972e65bbb56497ba29f81e6a84967dfebdf"}
Jul 17 22:06:03.124257 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:03.124051    2573 scope.go:117] "RemoveContainer" containerID="333450a1711921aebbf554a4dc9bf972e65bbb56497ba29f81e6a84967dfebdf"
Jul 17 22:06:04.129523 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:04.129490    2573 generic.go:356] "Generic (PLEG): container finished" podID="cd8f875b-830d-4ac5-a5df-d312362d74af" containerID="2d69acc6befe0c3c2dc90cc403c25580d361d6a214807bbc781642b4871b1a72" exitCode=6
Jul 17 22:06:04.129969 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:04.129581    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9" event={"ID":"cd8f875b-830d-4ac5-a5df-d312362d74af","Type":"ContainerDied","Data":"2d69acc6befe0c3c2dc90cc403c25580d361d6a214807bbc781642b4871b1a72"}
Jul 17 22:06:04.129969 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:04.129652    2573 scope.go:117] "RemoveContainer" containerID="333450a1711921aebbf554a4dc9bf972e65bbb56497ba29f81e6a84967dfebdf"
Jul 17 22:06:04.129969 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:04.129730    2573 scope.go:117] "RemoveContainer" containerID="2d69acc6befe0c3c2dc90cc403c25580d361d6a214807bbc781642b4871b1a72"
Jul 17 22:06:04.129969 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:06:04.129892    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-jtqx9_odh-ai-gateway-infra(cd8f875b-830d-4ac5-a5df-d312362d74af)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9" podUID="cd8f875b-830d-4ac5-a5df-d312362d74af"
Jul 17 22:06:05.135624 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:05.135600    2573 scope.go:117] "RemoveContainer" containerID="2d69acc6befe0c3c2dc90cc403c25580d361d6a214807bbc781642b4871b1a72"
Jul 17 22:06:05.136002 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:06:05.135764    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-jtqx9_odh-ai-gateway-infra(cd8f875b-830d-4ac5-a5df-d312362d74af)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9" podUID="cd8f875b-830d-4ac5-a5df-d312362d74af"
Jul 17 22:06:05.940812 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:05.940780    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc"]
Jul 17 22:06:05.945066 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:05.945051    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc"
Jul 17 22:06:05.950675 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:05.950644    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc"]
Jul 17 22:06:05.978835 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:05.978809    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f98d16-f580-4ee7-856d-2c5fac945ede-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc\" (UID: \"c5f98d16-f580-4ee7-856d-2c5fac945ede\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc"
Jul 17 22:06:06.079132 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:06.079097    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f98d16-f580-4ee7-856d-2c5fac945ede-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc\" (UID: \"c5f98d16-f580-4ee7-856d-2c5fac945ede\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc"
Jul 17 22:06:06.079812 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:06.079792    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f98d16-f580-4ee7-856d-2c5fac945ede-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc\" (UID: \"c5f98d16-f580-4ee7-856d-2c5fac945ede\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc"
Jul 17 22:06:06.256031 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:06.255956    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc"
Jul 17 22:06:06.376572 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:06.376551    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc"]
Jul 17 22:06:06.378508 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:06:06.378479    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f98d16_f580_4ee7_856d_2c5fac945ede.slice/crio-ce764443c724c7c0a4109d732b03a817e470d99b1baedc87e6771981438fa276 WatchSource:0}: Error finding container ce764443c724c7c0a4109d732b03a817e470d99b1baedc87e6771981438fa276: Status 404 returned error can't find the container with id ce764443c724c7c0a4109d732b03a817e470d99b1baedc87e6771981438fa276
Jul 17 22:06:07.146295 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:07.146258    2573 generic.go:356] "Generic (PLEG): container finished" podID="c5f98d16-f580-4ee7-856d-2c5fac945ede" containerID="1c86a1c08fd9de7f86203f1243d2ebfd76c4a23125283f4fbdf0a52d65319ddb" exitCode=6
Jul 17 22:06:07.146453 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:07.146312    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc" event={"ID":"c5f98d16-f580-4ee7-856d-2c5fac945ede","Type":"ContainerDied","Data":"1c86a1c08fd9de7f86203f1243d2ebfd76c4a23125283f4fbdf0a52d65319ddb"}
Jul 17 22:06:07.146453 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:07.146364    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc" event={"ID":"c5f98d16-f580-4ee7-856d-2c5fac945ede","Type":"ContainerStarted","Data":"ce764443c724c7c0a4109d732b03a817e470d99b1baedc87e6771981438fa276"}
Jul 17 22:06:07.146560 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:07.146554    2573 scope.go:117] "RemoveContainer" containerID="1c86a1c08fd9de7f86203f1243d2ebfd76c4a23125283f4fbdf0a52d65319ddb"
Jul 17 22:06:08.152115 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:08.152078    2573 generic.go:356] "Generic (PLEG): container finished" podID="c5f98d16-f580-4ee7-856d-2c5fac945ede" containerID="3f7d81c5f37f837458cd3d9e59f9fdd74f701e7e7deda1e428b2e0822e62f581" exitCode=6
Jul 17 22:06:08.152556 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:08.152163    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc" event={"ID":"c5f98d16-f580-4ee7-856d-2c5fac945ede","Type":"ContainerDied","Data":"3f7d81c5f37f837458cd3d9e59f9fdd74f701e7e7deda1e428b2e0822e62f581"}
Jul 17 22:06:08.152556 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:08.152220    2573 scope.go:117] "RemoveContainer" containerID="1c86a1c08fd9de7f86203f1243d2ebfd76c4a23125283f4fbdf0a52d65319ddb"
Jul 17 22:06:08.152556 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:08.152343    2573 scope.go:117] "RemoveContainer" containerID="3f7d81c5f37f837458cd3d9e59f9fdd74f701e7e7deda1e428b2e0822e62f581"
Jul 17 22:06:08.152675 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:06:08.152565    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc_odh-ai-gateway-infra(c5f98d16-f580-4ee7-856d-2c5fac945ede)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc" podUID="c5f98d16-f580-4ee7-856d-2c5fac945ede"
Jul 17 22:06:08.179151 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:08.179128    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm"]
Jul 17 22:06:08.183353 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:08.183338    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm"
Jul 17 22:06:08.190416 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:08.190393    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm"]
Jul 17 22:06:08.193925 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:08.193890    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/107ede0a-8f41-43ce-9a5d-1d8e018f2f6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm\" (UID: \"107ede0a-8f41-43ce-9a5d-1d8e018f2f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm"
Jul 17 22:06:08.294483 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:08.294453    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/107ede0a-8f41-43ce-9a5d-1d8e018f2f6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm\" (UID: \"107ede0a-8f41-43ce-9a5d-1d8e018f2f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm"
Jul 17 22:06:08.295030 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:08.295013    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/107ede0a-8f41-43ce-9a5d-1d8e018f2f6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm\" (UID: \"107ede0a-8f41-43ce-9a5d-1d8e018f2f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm"
Jul 17 22:06:08.494782 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:08.494692    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm"
Jul 17 22:06:08.525127 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:08.525105    2573 scope.go:117] "RemoveContainer" containerID="05482ac6a32ba21718cb4b0ee91410c449f8872f98dd5615a8fa69fb68d72383"
Jul 17 22:06:08.621854 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:08.621829    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm"]
Jul 17 22:06:08.623960 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:06:08.623934    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107ede0a_8f41_43ce_9a5d_1d8e018f2f6a.slice/crio-9b2fdffb6f568af3a752c83bd86849dc0577c96c3e7d935df1d1f10abe69fc60 WatchSource:0}: Error finding container 9b2fdffb6f568af3a752c83bd86849dc0577c96c3e7d935df1d1f10abe69fc60: Status 404 returned error can't find the container with id 9b2fdffb6f568af3a752c83bd86849dc0577c96c3e7d935df1d1f10abe69fc60
Jul 17 22:06:09.158813 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:09.158782    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh" event={"ID":"35991ade-70b6-4dd3-a502-fc26ea1579f2","Type":"ContainerStarted","Data":"e916a28ef6fca775d541d4e9c055139aa16699c9e68f0c702dd2ee5f14508dd2"}
Jul 17 22:06:09.160614 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:09.160586    2573 scope.go:117] "RemoveContainer" containerID="3f7d81c5f37f837458cd3d9e59f9fdd74f701e7e7deda1e428b2e0822e62f581"
Jul 17 22:06:09.160819 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:06:09.160795    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc_odh-ai-gateway-infra(c5f98d16-f580-4ee7-856d-2c5fac945ede)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc" podUID="c5f98d16-f580-4ee7-856d-2c5fac945ede"
Jul 17 22:06:09.161827 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:09.161791    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm" event={"ID":"107ede0a-8f41-43ce-9a5d-1d8e018f2f6a","Type":"ContainerStarted","Data":"b04a2b83368c944a3a1eed16a38a79195016e4144337109b98d079c38029abed"}
Jul 17 22:06:09.161949 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:09.161831    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm" event={"ID":"107ede0a-8f41-43ce-9a5d-1d8e018f2f6a","Type":"ContainerStarted","Data":"9b2fdffb6f568af3a752c83bd86849dc0577c96c3e7d935df1d1f10abe69fc60"}
Jul 17 22:06:09.199985 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:09.199938    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm" podStartSLOduration=1.199925133 podStartE2EDuration="1.199925133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:06:09.198527907 +0000 UTC m=+3469.162648518" watchObservedRunningTime="2026-07-17 22:06:09.199925133 +0000 UTC m=+3469.164045743"
Jul 17 22:06:10.167876 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:10.167842    2573 generic.go:356] "Generic (PLEG): container finished" podID="107ede0a-8f41-43ce-9a5d-1d8e018f2f6a" containerID="b04a2b83368c944a3a1eed16a38a79195016e4144337109b98d079c38029abed" exitCode=6
Jul 17 22:06:10.168354 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:10.167932    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm" event={"ID":"107ede0a-8f41-43ce-9a5d-1d8e018f2f6a","Type":"ContainerDied","Data":"b04a2b83368c944a3a1eed16a38a79195016e4144337109b98d079c38029abed"}
Jul 17 22:06:10.168354 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:10.168133    2573 scope.go:117] "RemoveContainer" containerID="b04a2b83368c944a3a1eed16a38a79195016e4144337109b98d079c38029abed"
Jul 17 22:06:10.169878 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:10.169858    2573 generic.go:356] "Generic (PLEG): container finished" podID="35991ade-70b6-4dd3-a502-fc26ea1579f2" containerID="e916a28ef6fca775d541d4e9c055139aa16699c9e68f0c702dd2ee5f14508dd2" exitCode=6
Jul 17 22:06:10.169994 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:10.169916    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh" event={"ID":"35991ade-70b6-4dd3-a502-fc26ea1579f2","Type":"ContainerDied","Data":"e916a28ef6fca775d541d4e9c055139aa16699c9e68f0c702dd2ee5f14508dd2"}
Jul 17 22:06:10.169994 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:10.169966    2573 scope.go:117] "RemoveContainer" containerID="05482ac6a32ba21718cb4b0ee91410c449f8872f98dd5615a8fa69fb68d72383"
Jul 17 22:06:10.170230 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:10.170209    2573 scope.go:117] "RemoveContainer" containerID="e916a28ef6fca775d541d4e9c055139aa16699c9e68f0c702dd2ee5f14508dd2"
Jul 17 22:06:10.170482 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:06:10.170461    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh_odh-ai-gateway-infra(35991ade-70b6-4dd3-a502-fc26ea1579f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh" podUID="35991ade-70b6-4dd3-a502-fc26ea1579f2"
Jul 17 22:06:10.186185 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:10.186160    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh"]
Jul 17 22:06:11.177827 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:11.177802    2573 generic.go:356] "Generic (PLEG): container finished" podID="107ede0a-8f41-43ce-9a5d-1d8e018f2f6a" containerID="2aed03e61a78db3fda17a8fc0ebdb57869bd6014d2348b46bd7dc55a8fe59c7d" exitCode=6
Jul 17 22:06:11.178176 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:11.177862    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm" event={"ID":"107ede0a-8f41-43ce-9a5d-1d8e018f2f6a","Type":"ContainerDied","Data":"2aed03e61a78db3fda17a8fc0ebdb57869bd6014d2348b46bd7dc55a8fe59c7d"}
Jul 17 22:06:11.178176 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:11.177905    2573 scope.go:117] "RemoveContainer" containerID="b04a2b83368c944a3a1eed16a38a79195016e4144337109b98d079c38029abed"
Jul 17 22:06:11.178176 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:11.178063    2573 scope.go:117] "RemoveContainer" containerID="2aed03e61a78db3fda17a8fc0ebdb57869bd6014d2348b46bd7dc55a8fe59c7d"
Jul 17 22:06:11.178373 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:06:11.178353    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm_odh-ai-gateway-infra(107ede0a-8f41-43ce-9a5d-1d8e018f2f6a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm" podUID="107ede0a-8f41-43ce-9a5d-1d8e018f2f6a"
Jul 17 22:06:11.305826 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:11.305801    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh"
Jul 17 22:06:11.322922 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:11.322896    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35991ade-70b6-4dd3-a502-fc26ea1579f2-maas-api-service-ca\") pod \"35991ade-70b6-4dd3-a502-fc26ea1579f2\" (UID: \"35991ade-70b6-4dd3-a502-fc26ea1579f2\") "
Jul 17 22:06:11.323789 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:11.323759    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35991ade-70b6-4dd3-a502-fc26ea1579f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "35991ade-70b6-4dd3-a502-fc26ea1579f2" (UID: "35991ade-70b6-4dd3-a502-fc26ea1579f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:06:11.424707 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:11.424678    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35991ade-70b6-4dd3-a502-fc26ea1579f2-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:06:12.189123 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:12.189082    2573 scope.go:117] "RemoveContainer" containerID="2aed03e61a78db3fda17a8fc0ebdb57869bd6014d2348b46bd7dc55a8fe59c7d"
Jul 17 22:06:12.189532 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:06:12.189277    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm_odh-ai-gateway-infra(107ede0a-8f41-43ce-9a5d-1d8e018f2f6a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm" podUID="107ede0a-8f41-43ce-9a5d-1d8e018f2f6a"
Jul 17 22:06:12.190098 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:12.190080    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh"
Jul 17 22:06:12.190207 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:12.190086    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh" event={"ID":"35991ade-70b6-4dd3-a502-fc26ea1579f2","Type":"ContainerDied","Data":"a1bd4a6ec12c55a6a57207da90c401ff1dd8159f83440a44fc6eee2a88bfc7e9"}
Jul 17 22:06:12.190253 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:12.190213    2573 scope.go:117] "RemoveContainer" containerID="e916a28ef6fca775d541d4e9c055139aa16699c9e68f0c702dd2ee5f14508dd2"
Jul 17 22:06:12.214813 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:12.214790    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh"]
Jul 17 22:06:12.218315 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:12.218277    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-6vqmh"]
Jul 17 22:06:12.529748 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:12.529678    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35991ade-70b6-4dd3-a502-fc26ea1579f2" path="/var/lib/kubelet/pods/35991ade-70b6-4dd3-a502-fc26ea1579f2/volumes"
Jul 17 22:06:18.524376 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:18.524350    2573 scope.go:117] "RemoveContainer" containerID="2d69acc6befe0c3c2dc90cc403c25580d361d6a214807bbc781642b4871b1a72"
Jul 17 22:06:19.222145 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:19.222116    2573 generic.go:356] "Generic (PLEG): container finished" podID="cd8f875b-830d-4ac5-a5df-d312362d74af" containerID="97bdf4ff7a8473632cdbfc22d2eb63142f0ffe16d67e865e39150a83815dcded" exitCode=6
Jul 17 22:06:19.222263 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:19.222151    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9" event={"ID":"cd8f875b-830d-4ac5-a5df-d312362d74af","Type":"ContainerDied","Data":"97bdf4ff7a8473632cdbfc22d2eb63142f0ffe16d67e865e39150a83815dcded"}
Jul 17 22:06:19.222263 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:19.222206    2573 scope.go:117] "RemoveContainer" containerID="2d69acc6befe0c3c2dc90cc403c25580d361d6a214807bbc781642b4871b1a72"
Jul 17 22:06:19.222508 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:19.222492    2573 scope.go:117] "RemoveContainer" containerID="97bdf4ff7a8473632cdbfc22d2eb63142f0ffe16d67e865e39150a83815dcded"
Jul 17 22:06:19.222730 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:06:19.222709    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-jtqx9_odh-ai-gateway-infra(cd8f875b-830d-4ac5-a5df-d312362d74af)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9" podUID="cd8f875b-830d-4ac5-a5df-d312362d74af"
Jul 17 22:06:20.245302 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:20.245259    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9"]
Jul 17 22:06:20.372524 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:20.372497    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9"
Jul 17 22:06:20.497182 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:20.497114    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8f875b-830d-4ac5-a5df-d312362d74af-maas-api-service-ca\") pod \"cd8f875b-830d-4ac5-a5df-d312362d74af\" (UID: \"cd8f875b-830d-4ac5-a5df-d312362d74af\") "
Jul 17 22:06:20.497451 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:20.497428    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8f875b-830d-4ac5-a5df-d312362d74af-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cd8f875b-830d-4ac5-a5df-d312362d74af" (UID: "cd8f875b-830d-4ac5-a5df-d312362d74af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:06:20.597658 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:20.597636    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8f875b-830d-4ac5-a5df-d312362d74af-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:06:21.234104 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:21.234024    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9"
Jul 17 22:06:21.234104 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:21.234041    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9" event={"ID":"cd8f875b-830d-4ac5-a5df-d312362d74af","Type":"ContainerDied","Data":"7187dae51386df3063c73d857219c3170df1df0d07b49d6876daa02f844ec136"}
Jul 17 22:06:21.234334 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:21.234115    2573 scope.go:117] "RemoveContainer" containerID="97bdf4ff7a8473632cdbfc22d2eb63142f0ffe16d67e865e39150a83815dcded"
Jul 17 22:06:21.250054 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:21.250033    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9"]
Jul 17 22:06:21.254324 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:21.254305    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-jtqx9"]
Jul 17 22:06:21.524835 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:21.524769    2573 scope.go:117] "RemoveContainer" containerID="3f7d81c5f37f837458cd3d9e59f9fdd74f701e7e7deda1e428b2e0822e62f581"
Jul 17 22:06:22.241773 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:22.241743    2573 generic.go:356] "Generic (PLEG): container finished" podID="c5f98d16-f580-4ee7-856d-2c5fac945ede" containerID="9aa7a1a84bc81b607a06c133aeabc697067b0a5bb96ed0d35fa4124275345c4e" exitCode=6
Jul 17 22:06:22.241898 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:22.241779    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc" event={"ID":"c5f98d16-f580-4ee7-856d-2c5fac945ede","Type":"ContainerDied","Data":"9aa7a1a84bc81b607a06c133aeabc697067b0a5bb96ed0d35fa4124275345c4e"}
Jul 17 22:06:22.241898 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:22.241833    2573 scope.go:117] "RemoveContainer" containerID="3f7d81c5f37f837458cd3d9e59f9fdd74f701e7e7deda1e428b2e0822e62f581"
Jul 17 22:06:22.242127 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:22.242108    2573 scope.go:117] "RemoveContainer" containerID="9aa7a1a84bc81b607a06c133aeabc697067b0a5bb96ed0d35fa4124275345c4e"
Jul 17 22:06:22.242369 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:06:22.242334    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc_odh-ai-gateway-infra(c5f98d16-f580-4ee7-856d-2c5fac945ede)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc" podUID="c5f98d16-f580-4ee7-856d-2c5fac945ede"
Jul 17 22:06:22.533503 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:22.533431    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd8f875b-830d-4ac5-a5df-d312362d74af" path="/var/lib/kubelet/pods/cd8f875b-830d-4ac5-a5df-d312362d74af/volumes"
Jul 17 22:06:23.265649 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:23.265618    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc"]
Jul 17 22:06:23.393657 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:23.393631    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc"
Jul 17 22:06:23.520316 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:23.520227    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f98d16-f580-4ee7-856d-2c5fac945ede-maas-api-service-ca\") pod \"c5f98d16-f580-4ee7-856d-2c5fac945ede\" (UID: \"c5f98d16-f580-4ee7-856d-2c5fac945ede\") "
Jul 17 22:06:23.520618 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:23.520596    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f98d16-f580-4ee7-856d-2c5fac945ede-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c5f98d16-f580-4ee7-856d-2c5fac945ede" (UID: "c5f98d16-f580-4ee7-856d-2c5fac945ede"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:06:23.621390 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:23.621350    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5f98d16-f580-4ee7-856d-2c5fac945ede-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:06:24.253981 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:24.253946    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc"
Jul 17 22:06:24.254149 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:24.253943    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc" event={"ID":"c5f98d16-f580-4ee7-856d-2c5fac945ede","Type":"ContainerDied","Data":"ce764443c724c7c0a4109d732b03a817e470d99b1baedc87e6771981438fa276"}
Jul 17 22:06:24.254149 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:24.254094    2573 scope.go:117] "RemoveContainer" containerID="9aa7a1a84bc81b607a06c133aeabc697067b0a5bb96ed0d35fa4124275345c4e"
Jul 17 22:06:24.273635 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:24.273611    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc"]
Jul 17 22:06:24.279439 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:24.279418    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-7csgc"]
Jul 17 22:06:24.529435 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:24.529372    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5f98d16-f580-4ee7-856d-2c5fac945ede" path="/var/lib/kubelet/pods/c5f98d16-f580-4ee7-856d-2c5fac945ede/volumes"
Jul 17 22:06:26.524660 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:26.524633    2573 scope.go:117] "RemoveContainer" containerID="2aed03e61a78db3fda17a8fc0ebdb57869bd6014d2348b46bd7dc55a8fe59c7d"
Jul 17 22:06:27.269545 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:27.269510    2573 generic.go:356] "Generic (PLEG): container finished" podID="107ede0a-8f41-43ce-9a5d-1d8e018f2f6a" containerID="59bb85406fac389af16b789e0e7e5ecd4d5b89d7d1682f12eca82eec4e6e7e85" exitCode=6
Jul 17 22:06:27.269687 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:27.269584    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm" event={"ID":"107ede0a-8f41-43ce-9a5d-1d8e018f2f6a","Type":"ContainerDied","Data":"59bb85406fac389af16b789e0e7e5ecd4d5b89d7d1682f12eca82eec4e6e7e85"}
Jul 17 22:06:27.269687 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:27.269657    2573 scope.go:117] "RemoveContainer" containerID="2aed03e61a78db3fda17a8fc0ebdb57869bd6014d2348b46bd7dc55a8fe59c7d"
Jul 17 22:06:27.269949 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:27.269926    2573 scope.go:117] "RemoveContainer" containerID="59bb85406fac389af16b789e0e7e5ecd4d5b89d7d1682f12eca82eec4e6e7e85"
Jul 17 22:06:27.270146 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:06:27.270125    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm_odh-ai-gateway-infra(107ede0a-8f41-43ce-9a5d-1d8e018f2f6a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm" podUID="107ede0a-8f41-43ce-9a5d-1d8e018f2f6a"
Jul 17 22:06:28.292837 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:28.292807    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm"]
Jul 17 22:06:28.420051 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:28.420028    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm"
Jul 17 22:06:28.562143 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:28.562079    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/107ede0a-8f41-43ce-9a5d-1d8e018f2f6a-maas-api-service-ca\") pod \"107ede0a-8f41-43ce-9a5d-1d8e018f2f6a\" (UID: \"107ede0a-8f41-43ce-9a5d-1d8e018f2f6a\") "
Jul 17 22:06:28.562417 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:28.562397    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107ede0a-8f41-43ce-9a5d-1d8e018f2f6a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "107ede0a-8f41-43ce-9a5d-1d8e018f2f6a" (UID: "107ede0a-8f41-43ce-9a5d-1d8e018f2f6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:06:28.663460 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:28.663429    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/107ede0a-8f41-43ce-9a5d-1d8e018f2f6a-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:06:29.281462 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:29.281432    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm"
Jul 17 22:06:29.281462 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:29.281441    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm" event={"ID":"107ede0a-8f41-43ce-9a5d-1d8e018f2f6a","Type":"ContainerDied","Data":"9b2fdffb6f568af3a752c83bd86849dc0577c96c3e7d935df1d1f10abe69fc60"}
Jul 17 22:06:29.281673 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:29.281497    2573 scope.go:117] "RemoveContainer" containerID="59bb85406fac389af16b789e0e7e5ecd4d5b89d7d1682f12eca82eec4e6e7e85"
Jul 17 22:06:29.305222 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:29.305202    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm"]
Jul 17 22:06:29.310853 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:29.310833    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-5kfbm"]
Jul 17 22:06:30.529691 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:30.529656    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="107ede0a-8f41-43ce-9a5d-1d8e018f2f6a" path="/var/lib/kubelet/pods/107ede0a-8f41-43ce-9a5d-1d8e018f2f6a/volumes"
Jul 17 22:06:45.600997 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.600913    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw"]
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601185    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd8f875b-830d-4ac5-a5df-d312362d74af" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601195    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd8f875b-830d-4ac5-a5df-d312362d74af" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601204    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35991ade-70b6-4dd3-a502-fc26ea1579f2" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601209    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="35991ade-70b6-4dd3-a502-fc26ea1579f2" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601216    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd8f875b-830d-4ac5-a5df-d312362d74af" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601221    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd8f875b-830d-4ac5-a5df-d312362d74af" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601229    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35991ade-70b6-4dd3-a502-fc26ea1579f2" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601234    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="35991ade-70b6-4dd3-a502-fc26ea1579f2" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601241    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5f98d16-f580-4ee7-856d-2c5fac945ede" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601246    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5f98d16-f580-4ee7-856d-2c5fac945ede" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601251    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="107ede0a-8f41-43ce-9a5d-1d8e018f2f6a" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601257    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="107ede0a-8f41-43ce-9a5d-1d8e018f2f6a" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601262    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5f98d16-f580-4ee7-856d-2c5fac945ede" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601267    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5f98d16-f580-4ee7-856d-2c5fac945ede" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601277    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="107ede0a-8f41-43ce-9a5d-1d8e018f2f6a" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601295    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="107ede0a-8f41-43ce-9a5d-1d8e018f2f6a" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601307    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="107ede0a-8f41-43ce-9a5d-1d8e018f2f6a" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601314    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="107ede0a-8f41-43ce-9a5d-1d8e018f2f6a" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601366    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5f98d16-f580-4ee7-856d-2c5fac945ede" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601373    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="35991ade-70b6-4dd3-a502-fc26ea1579f2" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601380    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd8f875b-830d-4ac5-a5df-d312362d74af" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601385    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="35991ade-70b6-4dd3-a502-fc26ea1579f2" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601390    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5f98d16-f580-4ee7-856d-2c5fac945ede" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601398    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="107ede0a-8f41-43ce-9a5d-1d8e018f2f6a" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601403    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd8f875b-830d-4ac5-a5df-d312362d74af" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601408    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd8f875b-830d-4ac5-a5df-d312362d74af" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601413    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="107ede0a-8f41-43ce-9a5d-1d8e018f2f6a" containerName="revoke-keys"
Jul 17 22:06:45.601482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.601420    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="107ede0a-8f41-43ce-9a5d-1d8e018f2f6a" containerName="revoke-keys"
Jul 17 22:06:45.604244 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.604226    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw"
Jul 17 22:06:45.606159 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.606135    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 22:06:45.606265 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.606223    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:06:45.611436 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.611217    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw"]
Jul 17 22:06:45.695147 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.695108    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d40f95-8c69-4013-836f-fec3fe6696ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw\" (UID: \"43d40f95-8c69-4013-836f-fec3fe6696ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw"
Jul 17 22:06:45.796053 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.796020    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d40f95-8c69-4013-836f-fec3fe6696ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw\" (UID: \"43d40f95-8c69-4013-836f-fec3fe6696ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw"
Jul 17 22:06:45.796621 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.796602    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d40f95-8c69-4013-836f-fec3fe6696ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw\" (UID: \"43d40f95-8c69-4013-836f-fec3fe6696ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw"
Jul 17 22:06:45.915116 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:45.915038    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw"
Jul 17 22:06:46.040318 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:46.040116    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw"]
Jul 17 22:06:46.044110 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:06:46.044085    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d40f95_8c69_4013_836f_fec3fe6696ae.slice/crio-6ec596a2d963eca1e6c35d64eaec7b8b9792a6de201bd9ec88747ea9e698feb3 WatchSource:0}: Error finding container 6ec596a2d963eca1e6c35d64eaec7b8b9792a6de201bd9ec88747ea9e698feb3: Status 404 returned error can't find the container with id 6ec596a2d963eca1e6c35d64eaec7b8b9792a6de201bd9ec88747ea9e698feb3
Jul 17 22:06:46.355079 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:46.355040    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw" event={"ID":"43d40f95-8c69-4013-836f-fec3fe6696ae","Type":"ContainerStarted","Data":"d12ddfb9bb658d7027fbc5cf3d109c78c8687531faf646c5cadeb1aa8f96d58c"}
Jul 17 22:06:46.355278 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:46.355092    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw" event={"ID":"43d40f95-8c69-4013-836f-fec3fe6696ae","Type":"ContainerStarted","Data":"6ec596a2d963eca1e6c35d64eaec7b8b9792a6de201bd9ec88747ea9e698feb3"}
Jul 17 22:06:46.369382 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:46.369341    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw" podStartSLOduration=1.36932692 podStartE2EDuration="1.36932692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:06:46.367747172 +0000 UTC m=+3506.331867783" watchObservedRunningTime="2026-07-17 22:06:46.36932692 +0000 UTC m=+3506.333447529"
Jul 17 22:06:47.360466 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:47.360412    2573 generic.go:356] "Generic (PLEG): container finished" podID="43d40f95-8c69-4013-836f-fec3fe6696ae" containerID="d12ddfb9bb658d7027fbc5cf3d109c78c8687531faf646c5cadeb1aa8f96d58c" exitCode=0
Jul 17 22:06:47.360847 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:47.360502    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw" event={"ID":"43d40f95-8c69-4013-836f-fec3fe6696ae","Type":"ContainerDied","Data":"d12ddfb9bb658d7027fbc5cf3d109c78c8687531faf646c5cadeb1aa8f96d58c"}
Jul 17 22:06:48.487562 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:48.487538    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw"
Jul 17 22:06:48.618162 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:48.618090    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d40f95-8c69-4013-836f-fec3fe6696ae-maas-api-service-ca\") pod \"43d40f95-8c69-4013-836f-fec3fe6696ae\" (UID: \"43d40f95-8c69-4013-836f-fec3fe6696ae\") "
Jul 17 22:06:48.618436 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:48.618416    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d40f95-8c69-4013-836f-fec3fe6696ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43d40f95-8c69-4013-836f-fec3fe6696ae" (UID: "43d40f95-8c69-4013-836f-fec3fe6696ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:06:48.719030 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:48.719000    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d40f95-8c69-4013-836f-fec3fe6696ae-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:06:49.370458 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:49.370426    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw" event={"ID":"43d40f95-8c69-4013-836f-fec3fe6696ae","Type":"ContainerDied","Data":"6ec596a2d963eca1e6c35d64eaec7b8b9792a6de201bd9ec88747ea9e698feb3"}
Jul 17 22:06:49.370458 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:49.370447    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw"
Jul 17 22:06:49.370669 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:49.370473    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ec596a2d963eca1e6c35d64eaec7b8b9792a6de201bd9ec88747ea9e698feb3"
Jul 17 22:06:50.154015 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.153980    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8"]
Jul 17 22:06:50.154416 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.154310    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd8f875b-830d-4ac5-a5df-d312362d74af" containerName="revoke-keys"
Jul 17 22:06:50.154416 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.154326    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd8f875b-830d-4ac5-a5df-d312362d74af" containerName="revoke-keys"
Jul 17 22:06:50.154416 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.154340    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35991ade-70b6-4dd3-a502-fc26ea1579f2" containerName="revoke-keys"
Jul 17 22:06:50.154416 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.154347    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="35991ade-70b6-4dd3-a502-fc26ea1579f2" containerName="revoke-keys"
Jul 17 22:06:50.154416 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.154355    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5f98d16-f580-4ee7-856d-2c5fac945ede" containerName="revoke-keys"
Jul 17 22:06:50.154416 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.154361    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5f98d16-f580-4ee7-856d-2c5fac945ede" containerName="revoke-keys"
Jul 17 22:06:50.154416 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.154372    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43d40f95-8c69-4013-836f-fec3fe6696ae" containerName="revoke-keys"
Jul 17 22:06:50.154416 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.154378    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="43d40f95-8c69-4013-836f-fec3fe6696ae" containerName="revoke-keys"
Jul 17 22:06:50.154677 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.154432    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="35991ade-70b6-4dd3-a502-fc26ea1579f2" containerName="revoke-keys"
Jul 17 22:06:50.154677 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.154439    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5f98d16-f580-4ee7-856d-2c5fac945ede" containerName="revoke-keys"
Jul 17 22:06:50.154677 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.154445    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="43d40f95-8c69-4013-836f-fec3fe6696ae" containerName="revoke-keys"
Jul 17 22:06:50.157389 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.157372    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8"
Jul 17 22:06:50.159890 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.159867    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:06:50.159987 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.159895    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 22:06:50.165703 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.165678    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8"]
Jul 17 22:06:50.333254 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.333213    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad4c5ca-7575-4943-b77b-43a8e6b25609-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8\" (UID: \"2ad4c5ca-7575-4943-b77b-43a8e6b25609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8"
Jul 17 22:06:50.434024 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.433950    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad4c5ca-7575-4943-b77b-43a8e6b25609-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8\" (UID: \"2ad4c5ca-7575-4943-b77b-43a8e6b25609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8"
Jul 17 22:06:50.434549 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.434524    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad4c5ca-7575-4943-b77b-43a8e6b25609-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8\" (UID: \"2ad4c5ca-7575-4943-b77b-43a8e6b25609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8"
Jul 17 22:06:50.468161 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.468139    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8"
Jul 17 22:06:50.592088 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:50.592062    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8"]
Jul 17 22:06:50.594342 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:06:50.594319    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad4c5ca_7575_4943_b77b_43a8e6b25609.slice/crio-40d4d549d5a52d701dde6212147287530a778db3e4f21bd305b505f3556199a8 WatchSource:0}: Error finding container 40d4d549d5a52d701dde6212147287530a778db3e4f21bd305b505f3556199a8: Status 404 returned error can't find the container with id 40d4d549d5a52d701dde6212147287530a778db3e4f21bd305b505f3556199a8
Jul 17 22:06:51.381227 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:51.381190    2573 generic.go:356] "Generic (PLEG): container finished" podID="2ad4c5ca-7575-4943-b77b-43a8e6b25609" containerID="1f707b8784f63d736d3d2d5f8a4790353dca8db7425908e7c86ea051af7b76ca" exitCode=6
Jul 17 22:06:51.381599 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:51.381293    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8" event={"ID":"2ad4c5ca-7575-4943-b77b-43a8e6b25609","Type":"ContainerDied","Data":"1f707b8784f63d736d3d2d5f8a4790353dca8db7425908e7c86ea051af7b76ca"}
Jul 17 22:06:51.381599 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:51.381351    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8" event={"ID":"2ad4c5ca-7575-4943-b77b-43a8e6b25609","Type":"ContainerStarted","Data":"40d4d549d5a52d701dde6212147287530a778db3e4f21bd305b505f3556199a8"}
Jul 17 22:06:51.381599 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:51.381423    2573 scope.go:117] "RemoveContainer" containerID="1f707b8784f63d736d3d2d5f8a4790353dca8db7425908e7c86ea051af7b76ca"
Jul 17 22:06:52.387274 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:52.387241    2573 generic.go:356] "Generic (PLEG): container finished" podID="2ad4c5ca-7575-4943-b77b-43a8e6b25609" containerID="96fd45a2fbe852443b6f166c279a802b0cf5fb129576f787ef17073134cea370" exitCode=6
Jul 17 22:06:52.387697 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:52.387310    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8" event={"ID":"2ad4c5ca-7575-4943-b77b-43a8e6b25609","Type":"ContainerDied","Data":"96fd45a2fbe852443b6f166c279a802b0cf5fb129576f787ef17073134cea370"}
Jul 17 22:06:52.387697 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:52.387367    2573 scope.go:117] "RemoveContainer" containerID="1f707b8784f63d736d3d2d5f8a4790353dca8db7425908e7c86ea051af7b76ca"
Jul 17 22:06:52.387697 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:52.387554    2573 scope.go:117] "RemoveContainer" containerID="96fd45a2fbe852443b6f166c279a802b0cf5fb129576f787ef17073134cea370"
Jul 17 22:06:52.387826 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:06:52.387736    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8_odh-ai-gateway-infra(2ad4c5ca-7575-4943-b77b-43a8e6b25609)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8" podUID="2ad4c5ca-7575-4943-b77b-43a8e6b25609"
Jul 17 22:06:53.393150 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:53.393124    2573 scope.go:117] "RemoveContainer" containerID="96fd45a2fbe852443b6f166c279a802b0cf5fb129576f787ef17073134cea370"
Jul 17 22:06:53.393558 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:06:53.393275    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8_odh-ai-gateway-infra(2ad4c5ca-7575-4943-b77b-43a8e6b25609)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8" podUID="2ad4c5ca-7575-4943-b77b-43a8e6b25609"
Jul 17 22:06:55.982696 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:55.982659    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m"]
Jul 17 22:06:55.987039 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:55.987014    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m"
Jul 17 22:06:55.992568 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:55.992540    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m"]
Jul 17 22:06:56.177622 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:56.177589    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70094acf-9eb8-4324-873c-2145a9b539e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m\" (UID: \"70094acf-9eb8-4324-873c-2145a9b539e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m"
Jul 17 22:06:56.205497 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:56.205459    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ac333d-29738760-4jchp"]
Jul 17 22:06:56.207209 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:56.207186    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-ac333d-29738760-4jchp"]
Jul 17 22:06:56.278416 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:56.278360    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70094acf-9eb8-4324-873c-2145a9b539e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m\" (UID: \"70094acf-9eb8-4324-873c-2145a9b539e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m"
Jul 17 22:06:56.278961 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:56.278943    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70094acf-9eb8-4324-873c-2145a9b539e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m\" (UID: \"70094acf-9eb8-4324-873c-2145a9b539e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m"
Jul 17 22:06:56.298426 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:56.298408    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m"
Jul 17 22:06:56.423767 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:56.423741    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m"]
Jul 17 22:06:56.425978 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:06:56.425952    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70094acf_9eb8_4324_873c_2145a9b539e4.slice/crio-268e2cc129b1dbb351a4fdfd6a2808cf86ed8702572213e93ce9bf3b77cd6a6d WatchSource:0}: Error finding container 268e2cc129b1dbb351a4fdfd6a2808cf86ed8702572213e93ce9bf3b77cd6a6d: Status 404 returned error can't find the container with id 268e2cc129b1dbb351a4fdfd6a2808cf86ed8702572213e93ce9bf3b77cd6a6d
Jul 17 22:06:56.530139 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:56.530073    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31fa2354-db49-41d0-8b4f-bb5f713faa84" path="/var/lib/kubelet/pods/31fa2354-db49-41d0-8b4f-bb5f713faa84/volumes"
Jul 17 22:06:57.415090 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:57.415057    2573 generic.go:356] "Generic (PLEG): container finished" podID="70094acf-9eb8-4324-873c-2145a9b539e4" containerID="550fee3c124a6c1d77f7ae2c844c80ecf5a6a206bbeba6aa11a9281223d5ccad" exitCode=6
Jul 17 22:06:57.415525 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:57.415144    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m" event={"ID":"70094acf-9eb8-4324-873c-2145a9b539e4","Type":"ContainerDied","Data":"550fee3c124a6c1d77f7ae2c844c80ecf5a6a206bbeba6aa11a9281223d5ccad"}
Jul 17 22:06:57.415525 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:57.415202    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m" event={"ID":"70094acf-9eb8-4324-873c-2145a9b539e4","Type":"ContainerStarted","Data":"268e2cc129b1dbb351a4fdfd6a2808cf86ed8702572213e93ce9bf3b77cd6a6d"}
Jul 17 22:06:57.415525 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:57.415326    2573 scope.go:117] "RemoveContainer" containerID="550fee3c124a6c1d77f7ae2c844c80ecf5a6a206bbeba6aa11a9281223d5ccad"
Jul 17 22:06:58.422238 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:58.422209    2573 generic.go:356] "Generic (PLEG): container finished" podID="70094acf-9eb8-4324-873c-2145a9b539e4" containerID="8f48947c8fed653db92f82cec7de9b1ffa604ffbe047244a931b996ca013b8b7" exitCode=6
Jul 17 22:06:58.422659 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:58.422312    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m" event={"ID":"70094acf-9eb8-4324-873c-2145a9b539e4","Type":"ContainerDied","Data":"8f48947c8fed653db92f82cec7de9b1ffa604ffbe047244a931b996ca013b8b7"}
Jul 17 22:06:58.422659 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:58.422369    2573 scope.go:117] "RemoveContainer" containerID="550fee3c124a6c1d77f7ae2c844c80ecf5a6a206bbeba6aa11a9281223d5ccad"
Jul 17 22:06:58.422659 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:58.422451    2573 scope.go:117] "RemoveContainer" containerID="8f48947c8fed653db92f82cec7de9b1ffa604ffbe047244a931b996ca013b8b7"
Jul 17 22:06:58.422659 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:06:58.422653    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m_odh-ai-gateway-infra(70094acf-9eb8-4324-873c-2145a9b539e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m" podUID="70094acf-9eb8-4324-873c-2145a9b539e4"
Jul 17 22:06:59.428455 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:06:59.428425    2573 scope.go:117] "RemoveContainer" containerID="8f48947c8fed653db92f82cec7de9b1ffa604ffbe047244a931b996ca013b8b7"
Jul 17 22:06:59.428840 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:06:59.428577    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m_odh-ai-gateway-infra(70094acf-9eb8-4324-873c-2145a9b539e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m" podUID="70094acf-9eb8-4324-873c-2145a9b539e4"
Jul 17 22:07:01.303547 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:01.303511    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7"]
Jul 17 22:07:01.307886 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:01.307869    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7"
Jul 17 22:07:01.313935 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:01.313909    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7"]
Jul 17 22:07:01.315065 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:01.315035    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1963bfd9-ca3d-468d-a6d2-b6a50e40ff65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-wxrn7\" (UID: \"1963bfd9-ca3d-468d-a6d2-b6a50e40ff65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7"
Jul 17 22:07:01.415748 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:01.415720    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1963bfd9-ca3d-468d-a6d2-b6a50e40ff65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-wxrn7\" (UID: \"1963bfd9-ca3d-468d-a6d2-b6a50e40ff65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7"
Jul 17 22:07:01.416318 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:01.416275    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1963bfd9-ca3d-468d-a6d2-b6a50e40ff65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-wxrn7\" (UID: \"1963bfd9-ca3d-468d-a6d2-b6a50e40ff65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7"
Jul 17 22:07:01.619350 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:01.619326    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7"
Jul 17 22:07:01.763964 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:01.763936    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7"]
Jul 17 22:07:01.766126 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:07:01.766096    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1963bfd9_ca3d_468d_a6d2_b6a50e40ff65.slice/crio-7ddcf2f59f6bcd1af4612e05ea0ef29914f23c22b48d8f27ded39dc9d6dc1c03 WatchSource:0}: Error finding container 7ddcf2f59f6bcd1af4612e05ea0ef29914f23c22b48d8f27ded39dc9d6dc1c03: Status 404 returned error can't find the container with id 7ddcf2f59f6bcd1af4612e05ea0ef29914f23c22b48d8f27ded39dc9d6dc1c03
Jul 17 22:07:02.443241 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:02.443214    2573 generic.go:356] "Generic (PLEG): container finished" podID="1963bfd9-ca3d-468d-a6d2-b6a50e40ff65" containerID="2aded5b9e14c209ce5140d19e27596575539e3d9575953e531a56a5975ea619e" exitCode=6
Jul 17 22:07:02.443531 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:02.443280    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7" event={"ID":"1963bfd9-ca3d-468d-a6d2-b6a50e40ff65","Type":"ContainerDied","Data":"2aded5b9e14c209ce5140d19e27596575539e3d9575953e531a56a5975ea619e"}
Jul 17 22:07:02.443531 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:02.443336    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7" event={"ID":"1963bfd9-ca3d-468d-a6d2-b6a50e40ff65","Type":"ContainerStarted","Data":"7ddcf2f59f6bcd1af4612e05ea0ef29914f23c22b48d8f27ded39dc9d6dc1c03"}
Jul 17 22:07:02.443531 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:02.443512    2573 scope.go:117] "RemoveContainer" containerID="2aded5b9e14c209ce5140d19e27596575539e3d9575953e531a56a5975ea619e"
Jul 17 22:07:03.449538 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:03.449505    2573 generic.go:356] "Generic (PLEG): container finished" podID="1963bfd9-ca3d-468d-a6d2-b6a50e40ff65" containerID="81ef1847047df9d252fb6f3a35d46d835e097ca0b8c08f681a062a9d410b5956" exitCode=6
Jul 17 22:07:03.449929 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:03.449584    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7" event={"ID":"1963bfd9-ca3d-468d-a6d2-b6a50e40ff65","Type":"ContainerDied","Data":"81ef1847047df9d252fb6f3a35d46d835e097ca0b8c08f681a062a9d410b5956"}
Jul 17 22:07:03.449929 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:03.449649    2573 scope.go:117] "RemoveContainer" containerID="2aded5b9e14c209ce5140d19e27596575539e3d9575953e531a56a5975ea619e"
Jul 17 22:07:03.449929 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:03.449815    2573 scope.go:117] "RemoveContainer" containerID="81ef1847047df9d252fb6f3a35d46d835e097ca0b8c08f681a062a9d410b5956"
Jul 17 22:07:03.450062 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:07:03.449975    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-wxrn7_odh-ai-gateway-infra(1963bfd9-ca3d-468d-a6d2-b6a50e40ff65)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7" podUID="1963bfd9-ca3d-468d-a6d2-b6a50e40ff65"
Jul 17 22:07:04.455713 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:04.455682    2573 scope.go:117] "RemoveContainer" containerID="81ef1847047df9d252fb6f3a35d46d835e097ca0b8c08f681a062a9d410b5956"
Jul 17 22:07:04.456083 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:07:04.455841    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-wxrn7_odh-ai-gateway-infra(1963bfd9-ca3d-468d-a6d2-b6a50e40ff65)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7" podUID="1963bfd9-ca3d-468d-a6d2-b6a50e40ff65"
Jul 17 22:07:05.524731 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:05.524703    2573 scope.go:117] "RemoveContainer" containerID="96fd45a2fbe852443b6f166c279a802b0cf5fb129576f787ef17073134cea370"
Jul 17 22:07:06.466179 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:06.466103    2573 generic.go:356] "Generic (PLEG): container finished" podID="2ad4c5ca-7575-4943-b77b-43a8e6b25609" containerID="e4b8836807696cb53c5e471d7d1d223b32433a86db2dd42012951e94a7629b12" exitCode=6
Jul 17 22:07:06.466329 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:06.466175    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8" event={"ID":"2ad4c5ca-7575-4943-b77b-43a8e6b25609","Type":"ContainerDied","Data":"e4b8836807696cb53c5e471d7d1d223b32433a86db2dd42012951e94a7629b12"}
Jul 17 22:07:06.466329 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:06.466264    2573 scope.go:117] "RemoveContainer" containerID="96fd45a2fbe852443b6f166c279a802b0cf5fb129576f787ef17073134cea370"
Jul 17 22:07:06.466510 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:06.466497    2573 scope.go:117] "RemoveContainer" containerID="e4b8836807696cb53c5e471d7d1d223b32433a86db2dd42012951e94a7629b12"
Jul 17 22:07:06.466711 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:07:06.466690    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8_odh-ai-gateway-infra(2ad4c5ca-7575-4943-b77b-43a8e6b25609)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8" podUID="2ad4c5ca-7575-4943-b77b-43a8e6b25609"
Jul 17 22:07:07.491193 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:07.491164    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8"]
Jul 17 22:07:07.626527 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:07.626504    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8"
Jul 17 22:07:07.660390 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:07.660366    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad4c5ca-7575-4943-b77b-43a8e6b25609-maas-api-service-ca\") pod \"2ad4c5ca-7575-4943-b77b-43a8e6b25609\" (UID: \"2ad4c5ca-7575-4943-b77b-43a8e6b25609\") "
Jul 17 22:07:07.660707 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:07.660684    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad4c5ca-7575-4943-b77b-43a8e6b25609-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ad4c5ca-7575-4943-b77b-43a8e6b25609" (UID: "2ad4c5ca-7575-4943-b77b-43a8e6b25609"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:07:07.761786 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:07.761719    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad4c5ca-7575-4943-b77b-43a8e6b25609-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:07:08.228604 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.228574    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq"]
Jul 17 22:07:08.228893 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.228880    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ad4c5ca-7575-4943-b77b-43a8e6b25609" containerName="revoke-keys"
Jul 17 22:07:08.228935 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.228895    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ad4c5ca-7575-4943-b77b-43a8e6b25609" containerName="revoke-keys"
Jul 17 22:07:08.228935 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.228916    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ad4c5ca-7575-4943-b77b-43a8e6b25609" containerName="revoke-keys"
Jul 17 22:07:08.228935 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.228922    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ad4c5ca-7575-4943-b77b-43a8e6b25609" containerName="revoke-keys"
Jul 17 22:07:08.229027 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.228974    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ad4c5ca-7575-4943-b77b-43a8e6b25609" containerName="revoke-keys"
Jul 17 22:07:08.229027 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.228983    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ad4c5ca-7575-4943-b77b-43a8e6b25609" containerName="revoke-keys"
Jul 17 22:07:08.229027 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.228989    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ad4c5ca-7575-4943-b77b-43a8e6b25609" containerName="revoke-keys"
Jul 17 22:07:08.231728 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.231711    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq"
Jul 17 22:07:08.238610 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.238535    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq"]
Jul 17 22:07:08.265487 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.265456    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77aff67-1031-42ff-aaf7-ea8e4bb6a860-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq\" (UID: \"f77aff67-1031-42ff-aaf7-ea8e4bb6a860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq"
Jul 17 22:07:08.366051 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.366021    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77aff67-1031-42ff-aaf7-ea8e4bb6a860-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq\" (UID: \"f77aff67-1031-42ff-aaf7-ea8e4bb6a860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq"
Jul 17 22:07:08.366628 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.366608    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77aff67-1031-42ff-aaf7-ea8e4bb6a860-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq\" (UID: \"f77aff67-1031-42ff-aaf7-ea8e4bb6a860\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq"
Jul 17 22:07:08.477915 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.477895    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8"
Jul 17 22:07:08.478078 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.477943    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8" event={"ID":"2ad4c5ca-7575-4943-b77b-43a8e6b25609","Type":"ContainerDied","Data":"40d4d549d5a52d701dde6212147287530a778db3e4f21bd305b505f3556199a8"}
Jul 17 22:07:08.478078 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.478005    2573 scope.go:117] "RemoveContainer" containerID="e4b8836807696cb53c5e471d7d1d223b32433a86db2dd42012951e94a7629b12"
Jul 17 22:07:08.497620 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.497597    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8"]
Jul 17 22:07:08.503041 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.503018    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc1422e3-9z9m8"]
Jul 17 22:07:08.529091 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.529066    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ad4c5ca-7575-4943-b77b-43a8e6b25609" path="/var/lib/kubelet/pods/2ad4c5ca-7575-4943-b77b-43a8e6b25609/volumes"
Jul 17 22:07:08.542782 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.542761    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq"
Jul 17 22:07:08.667651 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:08.667626    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq"]
Jul 17 22:07:08.669654 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:07:08.669628    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf77aff67_1031_42ff_aaf7_ea8e4bb6a860.slice/crio-57526cbe58854bbf94f4fa221543a2a409e389779fba69b7afd1fe36547936fd WatchSource:0}: Error finding container 57526cbe58854bbf94f4fa221543a2a409e389779fba69b7afd1fe36547936fd: Status 404 returned error can't find the container with id 57526cbe58854bbf94f4fa221543a2a409e389779fba69b7afd1fe36547936fd
Jul 17 22:07:09.484554 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:09.484472    2573 generic.go:356] "Generic (PLEG): container finished" podID="f77aff67-1031-42ff-aaf7-ea8e4bb6a860" containerID="d51d01e9f620496039c68fe8f75048936ad2e0be0ba45cfe2b179309f90cf46b" exitCode=6
Jul 17 22:07:09.484554 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:09.484510    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq" event={"ID":"f77aff67-1031-42ff-aaf7-ea8e4bb6a860","Type":"ContainerDied","Data":"d51d01e9f620496039c68fe8f75048936ad2e0be0ba45cfe2b179309f90cf46b"}
Jul 17 22:07:09.484554 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:09.484545    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq" event={"ID":"f77aff67-1031-42ff-aaf7-ea8e4bb6a860","Type":"ContainerStarted","Data":"57526cbe58854bbf94f4fa221543a2a409e389779fba69b7afd1fe36547936fd"}
Jul 17 22:07:09.484767 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:09.484689    2573 scope.go:117] "RemoveContainer" containerID="d51d01e9f620496039c68fe8f75048936ad2e0be0ba45cfe2b179309f90cf46b"
Jul 17 22:07:10.490749 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:10.490654    2573 generic.go:356] "Generic (PLEG): container finished" podID="f77aff67-1031-42ff-aaf7-ea8e4bb6a860" containerID="f2f419db3cc0a460d9e98fce553db5abd4ce1e7524ac55590a594064f4edf4bf" exitCode=6
Jul 17 22:07:10.490749 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:10.490710    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq" event={"ID":"f77aff67-1031-42ff-aaf7-ea8e4bb6a860","Type":"ContainerDied","Data":"f2f419db3cc0a460d9e98fce553db5abd4ce1e7524ac55590a594064f4edf4bf"}
Jul 17 22:07:10.491258 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:10.490806    2573 scope.go:117] "RemoveContainer" containerID="d51d01e9f620496039c68fe8f75048936ad2e0be0ba45cfe2b179309f90cf46b"
Jul 17 22:07:10.491258 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:10.491013    2573 scope.go:117] "RemoveContainer" containerID="f2f419db3cc0a460d9e98fce553db5abd4ce1e7524ac55590a594064f4edf4bf"
Jul 17 22:07:10.491258 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:07:10.491213    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq_odh-ai-gateway-infra(f77aff67-1031-42ff-aaf7-ea8e4bb6a860)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq" podUID="f77aff67-1031-42ff-aaf7-ea8e4bb6a860"
Jul 17 22:07:11.496701 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:11.496671    2573 scope.go:117] "RemoveContainer" containerID="f2f419db3cc0a460d9e98fce553db5abd4ce1e7524ac55590a594064f4edf4bf"
Jul 17 22:07:11.497132 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:07:11.496910    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq_odh-ai-gateway-infra(f77aff67-1031-42ff-aaf7-ea8e4bb6a860)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq" podUID="f77aff67-1031-42ff-aaf7-ea8e4bb6a860"
Jul 17 22:07:11.524333 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:11.524273    2573 scope.go:117] "RemoveContainer" containerID="8f48947c8fed653db92f82cec7de9b1ffa604ffbe047244a931b996ca013b8b7"
Jul 17 22:07:12.503946 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:12.503865    2573 generic.go:356] "Generic (PLEG): container finished" podID="70094acf-9eb8-4324-873c-2145a9b539e4" containerID="6f4758bbde226c69a1c046c788d67119d53b287c88f258f835607406d57f4eee" exitCode=6
Jul 17 22:07:12.503946 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:12.503936    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m" event={"ID":"70094acf-9eb8-4324-873c-2145a9b539e4","Type":"ContainerDied","Data":"6f4758bbde226c69a1c046c788d67119d53b287c88f258f835607406d57f4eee"}
Jul 17 22:07:12.504405 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:12.503995    2573 scope.go:117] "RemoveContainer" containerID="8f48947c8fed653db92f82cec7de9b1ffa604ffbe047244a931b996ca013b8b7"
Jul 17 22:07:12.504405 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:12.504347    2573 scope.go:117] "RemoveContainer" containerID="6f4758bbde226c69a1c046c788d67119d53b287c88f258f835607406d57f4eee"
Jul 17 22:07:12.504551 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:07:12.504522    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m_odh-ai-gateway-infra(70094acf-9eb8-4324-873c-2145a9b539e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m" podUID="70094acf-9eb8-4324-873c-2145a9b539e4"
Jul 17 22:07:13.527203 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:13.527175    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m"]
Jul 17 22:07:13.645110 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:13.645084    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m"
Jul 17 22:07:13.709010 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:13.708973    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70094acf-9eb8-4324-873c-2145a9b539e4-maas-api-service-ca\") pod \"70094acf-9eb8-4324-873c-2145a9b539e4\" (UID: \"70094acf-9eb8-4324-873c-2145a9b539e4\") "
Jul 17 22:07:13.709311 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:13.709268    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70094acf-9eb8-4324-873c-2145a9b539e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70094acf-9eb8-4324-873c-2145a9b539e4" (UID: "70094acf-9eb8-4324-873c-2145a9b539e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:07:13.810497 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:13.810440    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70094acf-9eb8-4324-873c-2145a9b539e4-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:07:14.515887 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:14.515853    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m" event={"ID":"70094acf-9eb8-4324-873c-2145a9b539e4","Type":"ContainerDied","Data":"268e2cc129b1dbb351a4fdfd6a2808cf86ed8702572213e93ce9bf3b77cd6a6d"}
Jul 17 22:07:14.516084 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:14.515910    2573 scope.go:117] "RemoveContainer" containerID="6f4758bbde226c69a1c046c788d67119d53b287c88f258f835607406d57f4eee"
Jul 17 22:07:14.516084 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:14.515869    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m"
Jul 17 22:07:14.538105 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:14.538083    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m"]
Jul 17 22:07:14.542672 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:14.542646    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eecee1fd-6tl8m"]
Jul 17 22:07:16.529021 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:16.528986    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70094acf-9eb8-4324-873c-2145a9b539e4" path="/var/lib/kubelet/pods/70094acf-9eb8-4324-873c-2145a9b539e4/volumes"
Jul 17 22:07:18.525304 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:18.525262    2573 scope.go:117] "RemoveContainer" containerID="81ef1847047df9d252fb6f3a35d46d835e097ca0b8c08f681a062a9d410b5956"
Jul 17 22:07:19.540406 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:19.540330    2573 generic.go:356] "Generic (PLEG): container finished" podID="1963bfd9-ca3d-468d-a6d2-b6a50e40ff65" containerID="951d539d0924bce0415161be65a994b99fa5b515a3f28030dd7e635911f64860" exitCode=6
Jul 17 22:07:19.540406 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:19.540363    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7" event={"ID":"1963bfd9-ca3d-468d-a6d2-b6a50e40ff65","Type":"ContainerDied","Data":"951d539d0924bce0415161be65a994b99fa5b515a3f28030dd7e635911f64860"}
Jul 17 22:07:19.540406 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:19.540405    2573 scope.go:117] "RemoveContainer" containerID="81ef1847047df9d252fb6f3a35d46d835e097ca0b8c08f681a062a9d410b5956"
Jul 17 22:07:19.540909 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:19.540654    2573 scope.go:117] "RemoveContainer" containerID="951d539d0924bce0415161be65a994b99fa5b515a3f28030dd7e635911f64860"
Jul 17 22:07:19.540909 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:07:19.540832    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-wxrn7_odh-ai-gateway-infra(1963bfd9-ca3d-468d-a6d2-b6a50e40ff65)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7" podUID="1963bfd9-ca3d-468d-a6d2-b6a50e40ff65"
Jul 17 22:07:20.566700 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:20.566669    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7"]
Jul 17 22:07:20.695093 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:20.695071    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7"
Jul 17 22:07:20.764371 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:20.764339    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1963bfd9-ca3d-468d-a6d2-b6a50e40ff65-maas-api-service-ca\") pod \"1963bfd9-ca3d-468d-a6d2-b6a50e40ff65\" (UID: \"1963bfd9-ca3d-468d-a6d2-b6a50e40ff65\") "
Jul 17 22:07:20.764670 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:20.764650    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1963bfd9-ca3d-468d-a6d2-b6a50e40ff65-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1963bfd9-ca3d-468d-a6d2-b6a50e40ff65" (UID: "1963bfd9-ca3d-468d-a6d2-b6a50e40ff65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:07:20.865858 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:20.865830    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1963bfd9-ca3d-468d-a6d2-b6a50e40ff65-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:07:20.985998 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:20.985971    2573 scope.go:117] "RemoveContainer" containerID="e38b762369701c3eade9894c9c48e47e1a7b14d38de0da6fd21e5ae6b0efe1e3"
Jul 17 22:07:21.550842 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:21.550813    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7"
Jul 17 22:07:21.551030 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:21.550854    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7" event={"ID":"1963bfd9-ca3d-468d-a6d2-b6a50e40ff65","Type":"ContainerDied","Data":"7ddcf2f59f6bcd1af4612e05ea0ef29914f23c22b48d8f27ded39dc9d6dc1c03"}
Jul 17 22:07:21.551030 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:21.550915    2573 scope.go:117] "RemoveContainer" containerID="951d539d0924bce0415161be65a994b99fa5b515a3f28030dd7e635911f64860"
Jul 17 22:07:21.577117 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:21.577087    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7"]
Jul 17 22:07:21.578586 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:21.578569    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-wxrn7"]
Jul 17 22:07:22.257270 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:22.257239    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-051ea5-29738760-s4f5f"]
Jul 17 22:07:22.259625 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:22.259603    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-051ea5-29738760-s4f5f"]
Jul 17 22:07:22.530131 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:22.530054    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1963bfd9-ca3d-468d-a6d2-b6a50e40ff65" path="/var/lib/kubelet/pods/1963bfd9-ca3d-468d-a6d2-b6a50e40ff65/volumes"
Jul 17 22:07:22.530367 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:22.530353    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3801dad-7ae9-4cf5-a24d-e60376ff1490" path="/var/lib/kubelet/pods/f3801dad-7ae9-4cf5-a24d-e60376ff1490/volumes"
Jul 17 22:07:26.529582 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:26.529558    2573 scope.go:117] "RemoveContainer" containerID="f2f419db3cc0a460d9e98fce553db5abd4ce1e7524ac55590a594064f4edf4bf"
Jul 17 22:07:27.576954 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:27.576867    2573 generic.go:356] "Generic (PLEG): container finished" podID="f77aff67-1031-42ff-aaf7-ea8e4bb6a860" containerID="03f9b6f612d62daf78a3dcd2fae576c2422bb386446faa2d292d981e77269bbe" exitCode=6
Jul 17 22:07:27.576954 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:27.576941    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq" event={"ID":"f77aff67-1031-42ff-aaf7-ea8e4bb6a860","Type":"ContainerDied","Data":"03f9b6f612d62daf78a3dcd2fae576c2422bb386446faa2d292d981e77269bbe"}
Jul 17 22:07:27.577425 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:27.577043    2573 scope.go:117] "RemoveContainer" containerID="f2f419db3cc0a460d9e98fce553db5abd4ce1e7524ac55590a594064f4edf4bf"
Jul 17 22:07:27.577425 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:27.577408    2573 scope.go:117] "RemoveContainer" containerID="03f9b6f612d62daf78a3dcd2fae576c2422bb386446faa2d292d981e77269bbe"
Jul 17 22:07:27.577631 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:07:27.577612    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq_odh-ai-gateway-infra(f77aff67-1031-42ff-aaf7-ea8e4bb6a860)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq" podUID="f77aff67-1031-42ff-aaf7-ea8e4bb6a860"
Jul 17 22:07:28.599423 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:28.599394    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq"]
Jul 17 22:07:28.724258 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:28.724236    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq"
Jul 17 22:07:28.828923 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:28.828894    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77aff67-1031-42ff-aaf7-ea8e4bb6a860-maas-api-service-ca\") pod \"f77aff67-1031-42ff-aaf7-ea8e4bb6a860\" (UID: \"f77aff67-1031-42ff-aaf7-ea8e4bb6a860\") "
Jul 17 22:07:28.829251 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:28.829232    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f77aff67-1031-42ff-aaf7-ea8e4bb6a860-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f77aff67-1031-42ff-aaf7-ea8e4bb6a860" (UID: "f77aff67-1031-42ff-aaf7-ea8e4bb6a860"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:07:28.929375 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:28.929310    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f77aff67-1031-42ff-aaf7-ea8e4bb6a860-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:07:29.587884 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:29.587850    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq" event={"ID":"f77aff67-1031-42ff-aaf7-ea8e4bb6a860","Type":"ContainerDied","Data":"57526cbe58854bbf94f4fa221543a2a409e389779fba69b7afd1fe36547936fd"}
Jul 17 22:07:29.587884 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:29.587869    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq"
Jul 17 22:07:29.588134 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:29.587903    2573 scope.go:117] "RemoveContainer" containerID="03f9b6f612d62daf78a3dcd2fae576c2422bb386446faa2d292d981e77269bbe"
Jul 17 22:07:29.610195 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:29.610169    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq"]
Jul 17 22:07:29.615953 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:29.615931    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-9clzq"]
Jul 17 22:07:30.530852 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:30.530811    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f77aff67-1031-42ff-aaf7-ea8e4bb6a860" path="/var/lib/kubelet/pods/f77aff67-1031-42ff-aaf7-ea8e4bb6a860/volumes"
Jul 17 22:07:37.465205 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:37.465164    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"]
Jul 17 22:07:37.465674 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:37.465430    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9" podUID="41e9349f-4a00-4427-bcc6-88f26802c273" containerName="istio-proxy" containerID="cri-o://c310df8531813fc7af23eb294e59d3b0ce83de0e107ea79da4d243b7fd19cf9a" gracePeriod=30
Jul 17 22:07:42.643956 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.643921    2573 generic.go:356] "Generic (PLEG): container finished" podID="41e9349f-4a00-4427-bcc6-88f26802c273" containerID="c310df8531813fc7af23eb294e59d3b0ce83de0e107ea79da4d243b7fd19cf9a" exitCode=0
Jul 17 22:07:42.644272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.643968    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9" event={"ID":"41e9349f-4a00-4427-bcc6-88f26802c273","Type":"ContainerDied","Data":"c310df8531813fc7af23eb294e59d3b0ce83de0e107ea79da4d243b7fd19cf9a"}
Jul 17 22:07:42.644272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.644028    2573 scope.go:117] "RemoveContainer" containerID="dfe0c924cf0cad18b44866d294090a46d03183e7ada180c42be1de97261e31e2"
Jul 17 22:07:42.710609 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.710582    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 22:07:42.828228 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.828148    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-istio-data\") pod \"41e9349f-4a00-4427-bcc6-88f26802c273\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") "
Jul 17 22:07:42.828228 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.828188    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-workload-certs\") pod \"41e9349f-4a00-4427-bcc6-88f26802c273\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") "
Jul 17 22:07:42.828228 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.828209    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-workload-socket\") pod \"41e9349f-4a00-4427-bcc6-88f26802c273\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") "
Jul 17 22:07:42.828533 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.828265    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e9349f-4a00-4427-bcc6-88f26802c273-istio-token\") pod \"41e9349f-4a00-4427-bcc6-88f26802c273\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") "
Jul 17 22:07:42.828533 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.828280    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-istio-envoy\") pod \"41e9349f-4a00-4427-bcc6-88f26802c273\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") "
Jul 17 22:07:42.828533 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.828318    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nng9\" (UniqueName: \"kubernetes.io/projected/41e9349f-4a00-4427-bcc6-88f26802c273-kube-api-access-4nng9\") pod \"41e9349f-4a00-4427-bcc6-88f26802c273\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") "
Jul 17 22:07:42.828533 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.828337    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-credential-socket\") pod \"41e9349f-4a00-4427-bcc6-88f26802c273\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") "
Jul 17 22:07:42.828533 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.828392    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e9349f-4a00-4427-bcc6-88f26802c273-istiod-ca-cert\") pod \"41e9349f-4a00-4427-bcc6-88f26802c273\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") "
Jul 17 22:07:42.828533 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.828424    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e9349f-4a00-4427-bcc6-88f26802c273-istio-podinfo\") pod \"41e9349f-4a00-4427-bcc6-88f26802c273\" (UID: \"41e9349f-4a00-4427-bcc6-88f26802c273\") "
Jul 17 22:07:42.828533 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.828529    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "41e9349f-4a00-4427-bcc6-88f26802c273" (UID: "41e9349f-4a00-4427-bcc6-88f26802c273"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:07:42.828891 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.828633    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "41e9349f-4a00-4427-bcc6-88f26802c273" (UID: "41e9349f-4a00-4427-bcc6-88f26802c273"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:07:42.828891 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.828730    2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-workload-certs\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:07:42.828891 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.828748    2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-workload-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:07:42.828891 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.828819    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "41e9349f-4a00-4427-bcc6-88f26802c273" (UID: "41e9349f-4a00-4427-bcc6-88f26802c273"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:07:42.829091 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.828945    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e9349f-4a00-4427-bcc6-88f26802c273-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "41e9349f-4a00-4427-bcc6-88f26802c273" (UID: "41e9349f-4a00-4427-bcc6-88f26802c273"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:07:42.829358 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.829338    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-istio-data" (OuterVolumeSpecName: "istio-data") pod "41e9349f-4a00-4427-bcc6-88f26802c273" (UID: "41e9349f-4a00-4427-bcc6-88f26802c273"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:07:42.830575 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.830544    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e9349f-4a00-4427-bcc6-88f26802c273-istio-token" (OuterVolumeSpecName: "istio-token") pod "41e9349f-4a00-4427-bcc6-88f26802c273" (UID: "41e9349f-4a00-4427-bcc6-88f26802c273"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:07:42.830575 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.830563    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e9349f-4a00-4427-bcc6-88f26802c273-kube-api-access-4nng9" (OuterVolumeSpecName: "kube-api-access-4nng9") pod "41e9349f-4a00-4427-bcc6-88f26802c273" (UID: "41e9349f-4a00-4427-bcc6-88f26802c273"). InnerVolumeSpecName "kube-api-access-4nng9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:07:42.830833 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.830816    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "41e9349f-4a00-4427-bcc6-88f26802c273" (UID: "41e9349f-4a00-4427-bcc6-88f26802c273"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:07:42.830872 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.830853    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41e9349f-4a00-4427-bcc6-88f26802c273-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "41e9349f-4a00-4427-bcc6-88f26802c273" (UID: "41e9349f-4a00-4427-bcc6-88f26802c273"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:07:42.929692 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.929658    2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e9349f-4a00-4427-bcc6-88f26802c273-istio-podinfo\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:07:42.929692 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.929691    2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-istio-data\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:07:42.929692 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.929701    2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e9349f-4a00-4427-bcc6-88f26802c273-istio-token\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:07:42.929870 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.929708    2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-istio-envoy\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:07:42.929870 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.929718    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nng9\" (UniqueName: \"kubernetes.io/projected/41e9349f-4a00-4427-bcc6-88f26802c273-kube-api-access-4nng9\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:07:42.929870 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.929726    2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e9349f-4a00-4427-bcc6-88f26802c273-credential-socket\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:07:42.929870 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:42.929734    2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e9349f-4a00-4427-bcc6-88f26802c273-istiod-ca-cert\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:07:43.649394 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:43.649358    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9" event={"ID":"41e9349f-4a00-4427-bcc6-88f26802c273","Type":"ContainerDied","Data":"628b5ed22f057299eaefe82e096f18745ec87f2a478239d21ee87feb8a83845a"}
Jul 17 22:07:43.649394 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:43.649386    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"
Jul 17 22:07:43.649921 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:43.649416    2573 scope.go:117] "RemoveContainer" containerID="c310df8531813fc7af23eb294e59d3b0ce83de0e107ea79da4d243b7fd19cf9a"
Jul 17 22:07:43.669659 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:43.669632    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"]
Jul 17 22:07:43.675212 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:43.675190    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-051ea5-openshift-default-6955b57579-wt8b9"]
Jul 17 22:07:43.965176 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:43.965108    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8f6b79df-smxrt_8501dc13-6ab5-4a53-84b0-bb02ef46e114/copy-manifests/0.log"
Jul 17 22:07:43.972057 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:43.972038    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8f6b79df-smxrt_8501dc13-6ab5-4a53-84b0-bb02ef46e114/manager/0.log"
Jul 17 22:07:44.529059 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:44.529027    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41e9349f-4a00-4427-bcc6-88f26802c273" path="/var/lib/kubelet/pods/41e9349f-4a00-4427-bcc6-88f26802c273/volumes"
Jul 17 22:07:44.741719 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:44.741693    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738760-569qx_c38b12d6-9aed-4f15-b927-a25b17914f27/cleanup/0.log"
Jul 17 22:07:44.850103 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:44.850076    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-380122-mcwr7_4b807a2a-edb2-4d29-b12b-08bac7aac22f/revoke-keys/0.log"
Jul 17 22:07:44.953482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:44.953456    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-dd0e35-kt7r8_55b3fcf9-6a2e-423e-934d-4b6fdcf9fb89/revoke-keys/0.log"
Jul 17 22:07:45.061770 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:45.061737    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-ffd161-fsth4_76dd8293-4d21-4387-bf38-8b63670fb88e/revoke-keys/0.log"
Jul 17 22:07:45.603850 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:45.603821    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-ac333d-46gkw_43d40f95-8c69-4013-836f-fec3fe6696ae/revoke-keys/0.log"
Jul 17 22:07:48.196857 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:48.196827    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vlswt_3ea85e23-ab25-4b26-83f4-cd0b3dfffe06/istio-proxy/0.log"
Jul 17 22:07:48.320731 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:48.320701    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-ghm49_054e5029-b4c2-465f-ad91-2dd5e116f847/discovery/0.log"
Jul 17 22:07:49.435633 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:49.435601    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-hzrq7_933175c9-8925-4726-aec4-6ab6c52e0a5e/main/0.log"
Jul 17 22:07:49.768717 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:49.768627    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-ckqnw_aa2aed99-b52e-4278-8c33-87ced547b65a/main/0.log"
Jul 17 22:07:57.515476 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:57.515444    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6lpt4_ce0a4720-2597-4f74-8e8a-012be51716e8/global-pull-secret-syncer/0.log"
Jul 17 22:07:57.646169 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:57.646139    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-brktb_6205c591-20ac-4b53-aa08-656d2aecc409/konnectivity-agent/0.log"
Jul 17 22:07:57.697193 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:07:57.697166    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-246.ec2.internal_a9ca977836d346c8c6c32a4fb71a6efd/haproxy/0.log"
Jul 17 22:08:01.355482 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355447    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55"]
Jul 17 22:08:01.355848 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355724    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f77aff67-1031-42ff-aaf7-ea8e4bb6a860" containerName="revoke-keys"
Jul 17 22:08:01.355848 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355739    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f77aff67-1031-42ff-aaf7-ea8e4bb6a860" containerName="revoke-keys"
Jul 17 22:08:01.355848 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355749    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f77aff67-1031-42ff-aaf7-ea8e4bb6a860" containerName="revoke-keys"
Jul 17 22:08:01.355848 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355755    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f77aff67-1031-42ff-aaf7-ea8e4bb6a860" containerName="revoke-keys"
Jul 17 22:08:01.355848 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355765    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70094acf-9eb8-4324-873c-2145a9b539e4" containerName="revoke-keys"
Jul 17 22:08:01.355848 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355771    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="70094acf-9eb8-4324-873c-2145a9b539e4" containerName="revoke-keys"
Jul 17 22:08:01.355848 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355777    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70094acf-9eb8-4324-873c-2145a9b539e4" containerName="revoke-keys"
Jul 17 22:08:01.355848 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355782    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="70094acf-9eb8-4324-873c-2145a9b539e4" containerName="revoke-keys"
Jul 17 22:08:01.355848 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355789    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1963bfd9-ca3d-468d-a6d2-b6a50e40ff65" containerName="revoke-keys"
Jul 17 22:08:01.355848 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355796    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1963bfd9-ca3d-468d-a6d2-b6a50e40ff65" containerName="revoke-keys"
Jul 17 22:08:01.355848 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355807    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f77aff67-1031-42ff-aaf7-ea8e4bb6a860" containerName="revoke-keys"
Jul 17 22:08:01.355848 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355814    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f77aff67-1031-42ff-aaf7-ea8e4bb6a860" containerName="revoke-keys"
Jul 17 22:08:01.355848 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355822    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e9349f-4a00-4427-bcc6-88f26802c273" containerName="istio-proxy"
Jul 17 22:08:01.355848 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355830    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e9349f-4a00-4427-bcc6-88f26802c273" containerName="istio-proxy"
Jul 17 22:08:01.355848 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355844    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ad4c5ca-7575-4943-b77b-43a8e6b25609" containerName="revoke-keys"
Jul 17 22:08:01.355848 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355849    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ad4c5ca-7575-4943-b77b-43a8e6b25609" containerName="revoke-keys"
Jul 17 22:08:01.355848 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355856    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1963bfd9-ca3d-468d-a6d2-b6a50e40ff65" containerName="revoke-keys"
Jul 17 22:08:01.356397 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355861    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1963bfd9-ca3d-468d-a6d2-b6a50e40ff65" containerName="revoke-keys"
Jul 17 22:08:01.356397 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355914    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1963bfd9-ca3d-468d-a6d2-b6a50e40ff65" containerName="revoke-keys"
Jul 17 22:08:01.356397 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355921    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="70094acf-9eb8-4324-873c-2145a9b539e4" containerName="revoke-keys"
Jul 17 22:08:01.356397 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355929    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e9349f-4a00-4427-bcc6-88f26802c273" containerName="istio-proxy"
Jul 17 22:08:01.356397 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355935    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f77aff67-1031-42ff-aaf7-ea8e4bb6a860" containerName="revoke-keys"
Jul 17 22:08:01.356397 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355940    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f77aff67-1031-42ff-aaf7-ea8e4bb6a860" containerName="revoke-keys"
Jul 17 22:08:01.356397 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355946    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="70094acf-9eb8-4324-873c-2145a9b539e4" containerName="revoke-keys"
Jul 17 22:08:01.356397 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355952    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1963bfd9-ca3d-468d-a6d2-b6a50e40ff65" containerName="revoke-keys"
Jul 17 22:08:01.356397 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.355958    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e9349f-4a00-4427-bcc6-88f26802c273" containerName="istio-proxy"
Jul 17 22:08:01.360648 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.360630    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55"
Jul 17 22:08:01.362622 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.362597    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 22:08:01.362622 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.362619    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qmg6\""
Jul 17 22:08:01.366061 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.366034    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55"]
Jul 17 22:08:01.469456 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.469428    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad107e5-50b1-4b9b-86cb-61d9a163fa32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-b9l55\" (UID: \"fad107e5-50b1-4b9b-86cb-61d9a163fa32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55"
Jul 17 22:08:01.570539 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.570509    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad107e5-50b1-4b9b-86cb-61d9a163fa32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-b9l55\" (UID: \"fad107e5-50b1-4b9b-86cb-61d9a163fa32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55"
Jul 17 22:08:01.571092 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.571071    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad107e5-50b1-4b9b-86cb-61d9a163fa32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-52039111-b9l55\" (UID: \"fad107e5-50b1-4b9b-86cb-61d9a163fa32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55"
Jul 17 22:08:01.671350 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.671260    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55"
Jul 17 22:08:01.793749 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:01.793723    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55"]
Jul 17 22:08:01.796455 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:08:01.796432    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad107e5_50b1_4b9b_86cb_61d9a163fa32.slice/crio-4a244c749a0a84dc76cd050b28249f628ebf9fc2d56732a0e84b6229c4a444ca WatchSource:0}: Error finding container 4a244c749a0a84dc76cd050b28249f628ebf9fc2d56732a0e84b6229c4a444ca: Status 404 returned error can't find the container with id 4a244c749a0a84dc76cd050b28249f628ebf9fc2d56732a0e84b6229c4a444ca
Jul 17 22:08:02.730456 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:02.730383    2573 generic.go:356] "Generic (PLEG): container finished" podID="fad107e5-50b1-4b9b-86cb-61d9a163fa32" containerID="d592514ed95a983f53e017579fe830381cc2b623b8f08e3be73520b6df114b4e" exitCode=6
Jul 17 22:08:02.730810 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:02.730472    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55" event={"ID":"fad107e5-50b1-4b9b-86cb-61d9a163fa32","Type":"ContainerDied","Data":"d592514ed95a983f53e017579fe830381cc2b623b8f08e3be73520b6df114b4e"}
Jul 17 22:08:02.730810 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:02.730528    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55" event={"ID":"fad107e5-50b1-4b9b-86cb-61d9a163fa32","Type":"ContainerStarted","Data":"4a244c749a0a84dc76cd050b28249f628ebf9fc2d56732a0e84b6229c4a444ca"}
Jul 17 22:08:02.730810 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:02.730612    2573 scope.go:117] "RemoveContainer" containerID="d592514ed95a983f53e017579fe830381cc2b623b8f08e3be73520b6df114b4e"
Jul 17 22:08:03.737341 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:03.737279    2573 generic.go:356] "Generic (PLEG): container finished" podID="fad107e5-50b1-4b9b-86cb-61d9a163fa32" containerID="6b8fb3b57ae89cc3feb119debedcf47b8c58058a13fc5965e09a98a810218955" exitCode=6
Jul 17 22:08:03.737743 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:03.737349    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55" event={"ID":"fad107e5-50b1-4b9b-86cb-61d9a163fa32","Type":"ContainerDied","Data":"6b8fb3b57ae89cc3feb119debedcf47b8c58058a13fc5965e09a98a810218955"}
Jul 17 22:08:03.737743 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:03.737433    2573 scope.go:117] "RemoveContainer" containerID="d592514ed95a983f53e017579fe830381cc2b623b8f08e3be73520b6df114b4e"
Jul 17 22:08:03.737743 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:03.737621    2573 scope.go:117] "RemoveContainer" containerID="6b8fb3b57ae89cc3feb119debedcf47b8c58058a13fc5965e09a98a810218955"
Jul 17 22:08:03.737881 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:08:03.737812    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-b9l55_odh-ai-gateway-infra(fad107e5-50b1-4b9b-86cb-61d9a163fa32)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55" podUID="fad107e5-50b1-4b9b-86cb-61d9a163fa32"
Jul 17 22:08:04.742963 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:04.742928    2573 scope.go:117] "RemoveContainer" containerID="6b8fb3b57ae89cc3feb119debedcf47b8c58058a13fc5965e09a98a810218955"
Jul 17 22:08:04.743468 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:08:04.743137    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-b9l55_odh-ai-gateway-infra(fad107e5-50b1-4b9b-86cb-61d9a163fa32)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55" podUID="fad107e5-50b1-4b9b-86cb-61d9a163fa32"
Jul 17 22:08:05.991089 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:05.991064    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-njrz6_3c558f3b-21ce-49a4-95ec-82537abae35a/node-exporter/0.log"
Jul 17 22:08:06.011477 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:06.011453    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-njrz6_3c558f3b-21ce-49a4-95ec-82537abae35a/kube-rbac-proxy/0.log"
Jul 17 22:08:06.035189 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:06.035166    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-njrz6_3c558f3b-21ce-49a4-95ec-82537abae35a/init-textfile/0.log"
Jul 17 22:08:06.336809 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:06.336783    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-ctqcz_ddbb92d2-c9c7-4e7c-ae12-28693c93de88/prometheus-operator-admission-webhook/0.log"
Jul 17 22:08:08.103599 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.103565    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"]
Jul 17 22:08:08.103977 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.103834    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70094acf-9eb8-4324-873c-2145a9b539e4" containerName="revoke-keys"
Jul 17 22:08:08.103977 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.103844    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="70094acf-9eb8-4324-873c-2145a9b539e4" containerName="revoke-keys"
Jul 17 22:08:08.103977 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.103852    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1963bfd9-ca3d-468d-a6d2-b6a50e40ff65" containerName="revoke-keys"
Jul 17 22:08:08.103977 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.103859    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1963bfd9-ca3d-468d-a6d2-b6a50e40ff65" containerName="revoke-keys"
Jul 17 22:08:08.103977 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.103866    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e9349f-4a00-4427-bcc6-88f26802c273" containerName="istio-proxy"
Jul 17 22:08:08.103977 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.103872    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e9349f-4a00-4427-bcc6-88f26802c273" containerName="istio-proxy"
Jul 17 22:08:08.103977 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.103920    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f77aff67-1031-42ff-aaf7-ea8e4bb6a860" containerName="revoke-keys"
Jul 17 22:08:08.103977 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.103930    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1963bfd9-ca3d-468d-a6d2-b6a50e40ff65" containerName="revoke-keys"
Jul 17 22:08:08.103977 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.103936    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="70094acf-9eb8-4324-873c-2145a9b539e4" containerName="revoke-keys"
Jul 17 22:08:08.106667 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.106649    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:08.108518 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.108496    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4qhz6\"/\"openshift-service-ca.crt\""
Jul 17 22:08:08.108617 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.108596    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4qhz6\"/\"default-dockercfg-tq2z6\""
Jul 17 22:08:08.108680 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.108623    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4qhz6\"/\"kube-root-ca.crt\""
Jul 17 22:08:08.114178 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.114154    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"]
Jul 17 22:08:08.114501 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.114482    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b00767bd-0775-4f2b-8e8a-f4df961c7705-podres\") pod \"perf-node-gather-daemonset-pwlch\" (UID: \"b00767bd-0775-4f2b-8e8a-f4df961c7705\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:08.114590 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.114515    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b00767bd-0775-4f2b-8e8a-f4df961c7705-lib-modules\") pod \"perf-node-gather-daemonset-pwlch\" (UID: \"b00767bd-0775-4f2b-8e8a-f4df961c7705\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:08.114590 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.114552    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjv5q\" (UniqueName: \"kubernetes.io/projected/b00767bd-0775-4f2b-8e8a-f4df961c7705-kube-api-access-tjv5q\") pod \"perf-node-gather-daemonset-pwlch\" (UID: \"b00767bd-0775-4f2b-8e8a-f4df961c7705\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:08.114696 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.114594    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b00767bd-0775-4f2b-8e8a-f4df961c7705-sys\") pod \"perf-node-gather-daemonset-pwlch\" (UID: \"b00767bd-0775-4f2b-8e8a-f4df961c7705\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:08.114696 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.114623    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b00767bd-0775-4f2b-8e8a-f4df961c7705-proc\") pod \"perf-node-gather-daemonset-pwlch\" (UID: \"b00767bd-0775-4f2b-8e8a-f4df961c7705\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:08.215067 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.215032    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b00767bd-0775-4f2b-8e8a-f4df961c7705-lib-modules\") pod \"perf-node-gather-daemonset-pwlch\" (UID: \"b00767bd-0775-4f2b-8e8a-f4df961c7705\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:08.215249 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.215084    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjv5q\" (UniqueName: \"kubernetes.io/projected/b00767bd-0775-4f2b-8e8a-f4df961c7705-kube-api-access-tjv5q\") pod \"perf-node-gather-daemonset-pwlch\" (UID: \"b00767bd-0775-4f2b-8e8a-f4df961c7705\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:08.215249 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.215115    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b00767bd-0775-4f2b-8e8a-f4df961c7705-sys\") pod \"perf-node-gather-daemonset-pwlch\" (UID: \"b00767bd-0775-4f2b-8e8a-f4df961c7705\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:08.215249 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.215142    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b00767bd-0775-4f2b-8e8a-f4df961c7705-proc\") pod \"perf-node-gather-daemonset-pwlch\" (UID: \"b00767bd-0775-4f2b-8e8a-f4df961c7705\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:08.215249 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.215200    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b00767bd-0775-4f2b-8e8a-f4df961c7705-podres\") pod \"perf-node-gather-daemonset-pwlch\" (UID: \"b00767bd-0775-4f2b-8e8a-f4df961c7705\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:08.215249 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.215212    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b00767bd-0775-4f2b-8e8a-f4df961c7705-sys\") pod \"perf-node-gather-daemonset-pwlch\" (UID: \"b00767bd-0775-4f2b-8e8a-f4df961c7705\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:08.215501 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.215252    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b00767bd-0775-4f2b-8e8a-f4df961c7705-lib-modules\") pod \"perf-node-gather-daemonset-pwlch\" (UID: \"b00767bd-0775-4f2b-8e8a-f4df961c7705\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:08.215501 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.215269    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b00767bd-0775-4f2b-8e8a-f4df961c7705-proc\") pod \"perf-node-gather-daemonset-pwlch\" (UID: \"b00767bd-0775-4f2b-8e8a-f4df961c7705\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:08.215501 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.215352    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b00767bd-0775-4f2b-8e8a-f4df961c7705-podres\") pod \"perf-node-gather-daemonset-pwlch\" (UID: \"b00767bd-0775-4f2b-8e8a-f4df961c7705\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:08.222620 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.222600    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjv5q\" (UniqueName: \"kubernetes.io/projected/b00767bd-0775-4f2b-8e8a-f4df961c7705-kube-api-access-tjv5q\") pod \"perf-node-gather-daemonset-pwlch\" (UID: \"b00767bd-0775-4f2b-8e8a-f4df961c7705\") " pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:08.275917 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.275891    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62"]
Jul 17 22:08:08.279037 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.279022    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62"
Jul 17 22:08:08.285703 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.285683    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62"]
Jul 17 22:08:08.315679 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.315650    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d5e559-efef-4ad4-8b1d-c5822a82386e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62\" (UID: \"c0d5e559-efef-4ad4-8b1d-c5822a82386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62"
Jul 17 22:08:08.416074 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.415999    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:08.416196 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.416175    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d5e559-efef-4ad4-8b1d-c5822a82386e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62\" (UID: \"c0d5e559-efef-4ad4-8b1d-c5822a82386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62"
Jul 17 22:08:08.416767 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.416749    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d5e559-efef-4ad4-8b1d-c5822a82386e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62\" (UID: \"c0d5e559-efef-4ad4-8b1d-c5822a82386e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62"
Jul 17 22:08:08.542125 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.542103    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"]
Jul 17 22:08:08.544484 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:08:08.544457    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb00767bd_0775_4f2b_8e8a_f4df961c7705.slice/crio-251edd4f54ae212a25a9fc0b672226268b87468f4ae9b905214987564feb1c81 WatchSource:0}: Error finding container 251edd4f54ae212a25a9fc0b672226268b87468f4ae9b905214987564feb1c81: Status 404 returned error can't find the container with id 251edd4f54ae212a25a9fc0b672226268b87468f4ae9b905214987564feb1c81
Jul 17 22:08:08.590362 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.590341    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62"
Jul 17 22:08:08.760508 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.760476    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch" event={"ID":"b00767bd-0775-4f2b-8e8a-f4df961c7705","Type":"ContainerStarted","Data":"ec7c5a5a3c0e7716ec1490f08db03fe488ec77208bf7943f80cf9f7cab337980"}
Jul 17 22:08:08.760674 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.760526    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch" event={"ID":"b00767bd-0775-4f2b-8e8a-f4df961c7705","Type":"ContainerStarted","Data":"251edd4f54ae212a25a9fc0b672226268b87468f4ae9b905214987564feb1c81"}
Jul 17 22:08:08.760674 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.760598    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:08.918408 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.918306    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch" podStartSLOduration=0.918260945 podStartE2EDuration="918.260945ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 22:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:08:08.790578693 +0000 UTC m=+3588.754699302" watchObservedRunningTime="2026-07-17 22:08:08.918260945 +0000 UTC m=+3588.882381555"
Jul 17 22:08:08.920044 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:08.920012    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62"]
Jul 17 22:08:08.922937 ip-10-0-128-246 kubenswrapper[2573]: W0717 22:08:08.922909    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d5e559_efef_4ad4_8b1d_c5822a82386e.slice/crio-a16587e83636c780161c98e2dfe8e14c8ac91d784a310cdcf3aa300316af38a8 WatchSource:0}: Error finding container a16587e83636c780161c98e2dfe8e14c8ac91d784a310cdcf3aa300316af38a8: Status 404 returned error can't find the container with id a16587e83636c780161c98e2dfe8e14c8ac91d784a310cdcf3aa300316af38a8
Jul 17 22:08:09.765613 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:09.765585    2573 generic.go:356] "Generic (PLEG): container finished" podID="c0d5e559-efef-4ad4-8b1d-c5822a82386e" containerID="44426021541f9213a8e9773e0a67ea0ffe4d1f7ade4a6a62e73cc6819bbb1530" exitCode=6
Jul 17 22:08:09.766002 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:09.765671    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62" event={"ID":"c0d5e559-efef-4ad4-8b1d-c5822a82386e","Type":"ContainerDied","Data":"44426021541f9213a8e9773e0a67ea0ffe4d1f7ade4a6a62e73cc6819bbb1530"}
Jul 17 22:08:09.766002 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:09.765723    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62" event={"ID":"c0d5e559-efef-4ad4-8b1d-c5822a82386e","Type":"ContainerStarted","Data":"a16587e83636c780161c98e2dfe8e14c8ac91d784a310cdcf3aa300316af38a8"}
Jul 17 22:08:09.766002 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:09.765820    2573 scope.go:117] "RemoveContainer" containerID="44426021541f9213a8e9773e0a67ea0ffe4d1f7ade4a6a62e73cc6819bbb1530"
Jul 17 22:08:10.573929 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:10.573901    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2m9sr_febc06a9-decb-4fb6-9b31-443a339262b9/dns/0.log"
Jul 17 22:08:10.596246 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:10.596222    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2m9sr_febc06a9-decb-4fb6-9b31-443a339262b9/kube-rbac-proxy/0.log"
Jul 17 22:08:10.739656 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:10.739627    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xgwzb_35528beb-a7c1-446b-bfae-6eb51d69a48c/dns-node-resolver/0.log"
Jul 17 22:08:10.772945 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:10.772919    2573 generic.go:356] "Generic (PLEG): container finished" podID="c0d5e559-efef-4ad4-8b1d-c5822a82386e" containerID="0058f4c446f7719e5ff79eefdd7b4025163adbdceecfd8ba59f5e0babc573290" exitCode=6
Jul 17 22:08:10.773308 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:10.772978    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62" event={"ID":"c0d5e559-efef-4ad4-8b1d-c5822a82386e","Type":"ContainerDied","Data":"0058f4c446f7719e5ff79eefdd7b4025163adbdceecfd8ba59f5e0babc573290"}
Jul 17 22:08:10.773308 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:10.773023    2573 scope.go:117] "RemoveContainer" containerID="44426021541f9213a8e9773e0a67ea0ffe4d1f7ade4a6a62e73cc6819bbb1530"
Jul 17 22:08:10.773308 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:10.773186    2573 scope.go:117] "RemoveContainer" containerID="0058f4c446f7719e5ff79eefdd7b4025163adbdceecfd8ba59f5e0babc573290"
Jul 17 22:08:10.773449 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:08:10.773419    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62_odh-ai-gateway-infra(c0d5e559-efef-4ad4-8b1d-c5822a82386e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62" podUID="c0d5e559-efef-4ad4-8b1d-c5822a82386e"
Jul 17 22:08:11.289050 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:11.289023    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6vr5m_d9d08404-9306-407f-b32a-a0e2d25473b1/node-ca/0.log"
Jul 17 22:08:11.778712 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:11.778682    2573 scope.go:117] "RemoveContainer" containerID="0058f4c446f7719e5ff79eefdd7b4025163adbdceecfd8ba59f5e0babc573290"
Jul 17 22:08:11.779158 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:08:11.778883    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62_odh-ai-gateway-infra(c0d5e559-efef-4ad4-8b1d-c5822a82386e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62" podUID="c0d5e559-efef-4ad4-8b1d-c5822a82386e"
Jul 17 22:08:12.253794 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:12.253763    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vlswt_3ea85e23-ab25-4b26-83f4-cd0b3dfffe06/istio-proxy/0.log"
Jul 17 22:08:12.368088 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:12.368064    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-ghm49_054e5029-b4c2-465f-ad91-2dd5e116f847/discovery/0.log"
Jul 17 22:08:13.274421 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:13.274348    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m6kll_8cf5abf4-6b79-4ed0-bf87-a45cddb80aa7/serve-healthcheck-canary/0.log"
Jul 17 22:08:13.756079 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:13.756055    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dnp9k_96edb730-a088-46cb-89e6-731a4d05f8cc/kube-rbac-proxy/0.log"
Jul 17 22:08:13.777037 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:13.777019    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dnp9k_96edb730-a088-46cb-89e6-731a4d05f8cc/exporter/0.log"
Jul 17 22:08:13.797828 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:13.797803    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dnp9k_96edb730-a088-46cb-89e6-731a4d05f8cc/extractor/0.log"
Jul 17 22:08:14.775526 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:14.775502    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4qhz6/perf-node-gather-daemonset-pwlch"
Jul 17 22:08:15.524839 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:15.524809    2573 scope.go:117] "RemoveContainer" containerID="6b8fb3b57ae89cc3feb119debedcf47b8c58058a13fc5965e09a98a810218955"
Jul 17 22:08:15.797935 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:15.797873    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55" event={"ID":"fad107e5-50b1-4b9b-86cb-61d9a163fa32","Type":"ContainerStarted","Data":"3744e2a88826b0c8b65285260e0eee8afc835be78761bbfaeae4534ae519dcd2"}
Jul 17 22:08:15.813256 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:15.813211    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55" podStartSLOduration=14.813197603999999 podStartE2EDuration="14.813197604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:08:15.811249264 +0000 UTC m=+3595.775369873" watchObservedRunningTime="2026-07-17 22:08:15.813197604 +0000 UTC m=+3595.777318215"
Jul 17 22:08:16.646514 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:16.646485    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8f6b79df-smxrt_8501dc13-6ab5-4a53-84b0-bb02ef46e114/manager/0.log"
Jul 17 22:08:16.667598 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:16.667576    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b8f6b79df-smxrt_8501dc13-6ab5-4a53-84b0-bb02ef46e114/copy-manifests/0.log"
Jul 17 22:08:16.803334 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:16.803278    2573 generic.go:356] "Generic (PLEG): container finished" podID="fad107e5-50b1-4b9b-86cb-61d9a163fa32" containerID="3744e2a88826b0c8b65285260e0eee8afc835be78761bbfaeae4534ae519dcd2" exitCode=6
Jul 17 22:08:16.803334 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:16.803321    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55" event={"ID":"fad107e5-50b1-4b9b-86cb-61d9a163fa32","Type":"ContainerDied","Data":"3744e2a88826b0c8b65285260e0eee8afc835be78761bbfaeae4534ae519dcd2"}
Jul 17 22:08:16.803748 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:16.803384    2573 scope.go:117] "RemoveContainer" containerID="6b8fb3b57ae89cc3feb119debedcf47b8c58058a13fc5965e09a98a810218955"
Jul 17 22:08:16.803748 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:16.803629    2573 scope.go:117] "RemoveContainer" containerID="3744e2a88826b0c8b65285260e0eee8afc835be78761bbfaeae4534ae519dcd2"
Jul 17 22:08:16.803836 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:08:16.803819    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-52039111-b9l55_odh-ai-gateway-infra(fad107e5-50b1-4b9b-86cb-61d9a163fa32)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55" podUID="fad107e5-50b1-4b9b-86cb-61d9a163fa32"
Jul 17 22:08:16.821893 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:16.821867    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55"]
Jul 17 22:08:17.960213 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:17.960190    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55"
Jul 17 22:08:17.985143 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:17.985122    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad107e5-50b1-4b9b-86cb-61d9a163fa32-maas-api-service-ca\") pod \"fad107e5-50b1-4b9b-86cb-61d9a163fa32\" (UID: \"fad107e5-50b1-4b9b-86cb-61d9a163fa32\") "
Jul 17 22:08:17.985475 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:17.985452    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad107e5-50b1-4b9b-86cb-61d9a163fa32-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fad107e5-50b1-4b9b-86cb-61d9a163fa32" (UID: "fad107e5-50b1-4b9b-86cb-61d9a163fa32"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:08:18.086142 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:18.086120    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad107e5-50b1-4b9b-86cb-61d9a163fa32-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:08:18.285335 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:18.285307    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7f487bdccb-fj54j_6b7f6c48-59a8-4a21-9002-1ad27a25cef5/manager/0.log"
Jul 17 22:08:18.718896 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:18.718859    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vlswt_3ea85e23-ab25-4b26-83f4-cd0b3dfffe06/istio-proxy/0.log"
Jul 17 22:08:18.814164 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:18.814026    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55"
Jul 17 22:08:18.814164 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:18.814053    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55" event={"ID":"fad107e5-50b1-4b9b-86cb-61d9a163fa32","Type":"ContainerDied","Data":"4a244c749a0a84dc76cd050b28249f628ebf9fc2d56732a0e84b6229c4a444ca"}
Jul 17 22:08:18.814164 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:18.814104    2573 scope.go:117] "RemoveContainer" containerID="3744e2a88826b0c8b65285260e0eee8afc835be78761bbfaeae4534ae519dcd2"
Jul 17 22:08:18.830967 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:18.830925    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55"]
Jul 17 22:08:18.833473 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:18.833455    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-ghm49_054e5029-b4c2-465f-ad91-2dd5e116f847/discovery/0.log"
Jul 17 22:08:18.836138 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:18.836115    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-52039111-b9l55"]
Jul 17 22:08:20.149006 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:20.148977    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vlswt_3ea85e23-ab25-4b26-83f4-cd0b3dfffe06/istio-proxy/0.log"
Jul 17 22:08:20.261222 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:20.261193    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-ghm49_054e5029-b4c2-465f-ad91-2dd5e116f847/discovery/0.log"
Jul 17 22:08:20.530021 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:20.529942    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fad107e5-50b1-4b9b-86cb-61d9a163fa32" path="/var/lib/kubelet/pods/fad107e5-50b1-4b9b-86cb-61d9a163fa32/volumes"
Jul 17 22:08:21.019709 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:21.019676    2573 scope.go:117] "RemoveContainer" containerID="c6b3ddb3d2c0974d25e3c0cd1aa2d526d7379a312d3428e8d8c12dbb77e0588b"
Jul 17 22:08:22.525329 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:22.525275    2573 scope.go:117] "RemoveContainer" containerID="0058f4c446f7719e5ff79eefdd7b4025163adbdceecfd8ba59f5e0babc573290"
Jul 17 22:08:22.835102 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:22.835034    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62" event={"ID":"c0d5e559-efef-4ad4-8b1d-c5822a82386e","Type":"ContainerStarted","Data":"a89935eaf8d9c44e48d37541f9acdf6d7336168ec67052979b9fe88408e4a8a1"}
Jul 17 22:08:22.850082 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:22.850033    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62" podStartSLOduration=14.850020422 podStartE2EDuration="14.850020422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:08:22.848659541 +0000 UTC m=+3602.812780164" watchObservedRunningTime="2026-07-17 22:08:22.850020422 +0000 UTC m=+3602.814141032"
Jul 17 22:08:23.841345 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:23.841306    2573 generic.go:356] "Generic (PLEG): container finished" podID="c0d5e559-efef-4ad4-8b1d-c5822a82386e" containerID="a89935eaf8d9c44e48d37541f9acdf6d7336168ec67052979b9fe88408e4a8a1" exitCode=6
Jul 17 22:08:23.841757 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:23.841315    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62" event={"ID":"c0d5e559-efef-4ad4-8b1d-c5822a82386e","Type":"ContainerDied","Data":"a89935eaf8d9c44e48d37541f9acdf6d7336168ec67052979b9fe88408e4a8a1"}
Jul 17 22:08:23.841757 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:23.841453    2573 scope.go:117] "RemoveContainer" containerID="0058f4c446f7719e5ff79eefdd7b4025163adbdceecfd8ba59f5e0babc573290"
Jul 17 22:08:23.841757 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:23.841738    2573 scope.go:117] "RemoveContainer" containerID="a89935eaf8d9c44e48d37541f9acdf6d7336168ec67052979b9fe88408e4a8a1"
Jul 17 22:08:23.841949 ip-10-0-128-246 kubenswrapper[2573]: E0717 22:08:23.841924    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62_odh-ai-gateway-infra(c0d5e559-efef-4ad4-8b1d-c5822a82386e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62" podUID="c0d5e559-efef-4ad4-8b1d-c5822a82386e"
Jul 17 22:08:23.863683 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:23.863652    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62"]
Jul 17 22:08:24.776265 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:24.776241    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-99pvf_2297d3f6-c109-4b3d-883c-1b0a411f9512/kube-multus-additional-cni-plugins/0.log"
Jul 17 22:08:24.831009 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:24.830987    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-99pvf_2297d3f6-c109-4b3d-883c-1b0a411f9512/egress-router-binary-copy/0.log"
Jul 17 22:08:24.855337 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:24.855312    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-99pvf_2297d3f6-c109-4b3d-883c-1b0a411f9512/cni-plugins/0.log"
Jul 17 22:08:24.879245 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:24.879219    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-99pvf_2297d3f6-c109-4b3d-883c-1b0a411f9512/bond-cni-plugin/0.log"
Jul 17 22:08:24.900279 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:24.900259    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-99pvf_2297d3f6-c109-4b3d-883c-1b0a411f9512/routeoverride-cni/0.log"
Jul 17 22:08:24.921369 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:24.921344    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-99pvf_2297d3f6-c109-4b3d-883c-1b0a411f9512/whereabouts-cni-bincopy/0.log"
Jul 17 22:08:24.943336 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:24.943315    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-99pvf_2297d3f6-c109-4b3d-883c-1b0a411f9512/whereabouts-cni/0.log"
Jul 17 22:08:24.974502 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:24.974481    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62"
Jul 17 22:08:25.035272 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:25.035187    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d5e559-efef-4ad4-8b1d-c5822a82386e-maas-api-service-ca\") pod \"c0d5e559-efef-4ad4-8b1d-c5822a82386e\" (UID: \"c0d5e559-efef-4ad4-8b1d-c5822a82386e\") "
Jul 17 22:08:25.035505 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:25.035484    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d5e559-efef-4ad4-8b1d-c5822a82386e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c0d5e559-efef-4ad4-8b1d-c5822a82386e" (UID: "c0d5e559-efef-4ad4-8b1d-c5822a82386e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:08:25.136069 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:25.136038    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d5e559-efef-4ad4-8b1d-c5822a82386e-maas-api-service-ca\") on node \"ip-10-0-128-246.ec2.internal\" DevicePath \"\""
Jul 17 22:08:25.220223 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:25.220188    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-k9jkf_72709796-e935-4cba-a007-cc17fb595e10/kube-multus/0.log"
Jul 17 22:08:25.258267 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:25.258239    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-87nv2_4147724d-819d-4372-b2c8-c170b2d71411/network-metrics-daemon/0.log"
Jul 17 22:08:25.274457 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:25.274427    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-87nv2_4147724d-819d-4372-b2c8-c170b2d71411/kube-rbac-proxy/0.log"
Jul 17 22:08:25.852057 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:25.852028    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62" event={"ID":"c0d5e559-efef-4ad4-8b1d-c5822a82386e","Type":"ContainerDied","Data":"a16587e83636c780161c98e2dfe8e14c8ac91d784a310cdcf3aa300316af38a8"}
Jul 17 22:08:25.852244 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:25.852060    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db4eb08b-4ld62"
Jul 17 22:08:25.852244 ip-10-0-128-246 kubenswrapper[2573]: I0717 22:08:25.852085    2573 scope.go:117] "RemoveContainer" containerID="a89935eaf8d9c44e48d37541f9acdf6d7336168ec67052979b9fe88408e4a8a1"