Jul 16 22:33:09.142573 ip-10-0-139-23 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 22:33:09.593363 ip-10-0-139-23 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 22:33:09.593363 ip-10-0-139-23 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 22:33:09.593363 ip-10-0-139-23 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 22:33:09.593363 ip-10-0-139-23 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 22:33:09.593363 ip-10-0-139-23 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 22:33:09.593363 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.545035    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 22:33:09.593363 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550117    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 22:33:09.593363 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550131    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 22:33:09.593363 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550135    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 22:33:09.593363 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550140    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 22:33:09.593363 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550144    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 22:33:09.593363 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550149    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 22:33:09.593363 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550153    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 22:33:09.593363 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550158    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550161    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550165    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550169    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550173    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550176    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550182    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550186    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550190    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550193    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550197    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550201    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550204    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550208    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550212    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550216    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550223    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550228    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550232    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550237    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 22:33:09.595744 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550241    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550302    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550308    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550313    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550317    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550321    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550326    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550330    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550334    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550338    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550343    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550346    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550350    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550354    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550358    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550362    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550366    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550370    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550373    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550378    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 22:33:09.605058 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550383    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550389    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550393    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550398    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550402    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550406    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550410    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550413    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550417    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550421    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550425    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550429    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550434    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550438    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550442    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550448    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550451    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550455    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550459    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550462    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 22:33:09.607497 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550468    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550472    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550475    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550479    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550483    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550489    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550493    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550498    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550502    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550508    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550513    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550517    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550523    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550527    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550531    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550535    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550539    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550542    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550546    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 22:33:09.615478 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550550    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 22:33:09.610232 ip-10-0-139-23 systemd[1]: Started Kubernetes Kubelet.
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550554    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550558    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550562    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550566    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550569    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550573    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550577    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550581    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550585    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550589    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550593    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.550597    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551598    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551608    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551612    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551616    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551620    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551624    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 22:33:09.622818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551629    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551632    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551637    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551640    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551644    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551648    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551652    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551657    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551661    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551664    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551669    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551673    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551676    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551680    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551684    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551688    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551692    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551696    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551700    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551704    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 22:33:09.628182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551708    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551712    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551717    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551724    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551728    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551734    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551739    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551744    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551748    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551752    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551757    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551761    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551765    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551771    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551775    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551778    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551782    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551787    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551791    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 22:33:09.635379 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551795    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551800    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551803    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551808    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551812    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551817    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551820    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551825    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551830    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551834    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551838    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551842    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551846    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551850    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551854    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551858    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551862    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551866    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551870    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551874    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 22:33:09.636308 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551878    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551882    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551886    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551890    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551894    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551898    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551902    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551906    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551909    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551914    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551942    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551948    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551952    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551956    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551962    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551966    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551969    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551974    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551978    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 22:33:09.637319 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551981    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551988    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551993    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.551998    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552005    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552009    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552013    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552018    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552022    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552026    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552030    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552035    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552039    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552043    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552046    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552566    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552574    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552579    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552583    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 22:33:09.638128 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552587    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552591    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552597    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552602    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552606    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552610    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552614    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552619    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552624    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552629    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552636    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552640    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552644    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552648    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552653    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552657    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552661    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552667    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552672    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 22:33:09.638876 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552676    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552679    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552683    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552688    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552692    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552695    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552699    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552702    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552706    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552711    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552715    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552719    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552722    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552726    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552729    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552733    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552737    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552741    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552744    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552748    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 22:33:09.639623 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552752    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552756    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552760    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552764    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552767    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552772    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552777    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552781    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552786    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552789    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552797    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552802    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552806    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552812    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552816    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552820    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552823    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552830    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552833    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552838    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 22:33:09.640414 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552842    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552846    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552849    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552853    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552857    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552861    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552864    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552868    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552872    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552875    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552879    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552883    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552886    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552890    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552893    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552898    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552902    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552906    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552909    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552913    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 22:33:09.641309 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552917    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552921    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552926    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552930    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552936    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552940    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552944    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552948    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552951    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552955    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552958    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552962    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552966    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552970    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552974    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.552978    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553061    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553071    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553080    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553085    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 22:33:09.642182 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553091    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553096    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553101    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553107    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553111    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553116    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553121    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553126    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553131    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553136    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553140    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553144    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553148    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553153    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553159    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553164    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553170    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553174    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553179    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553185    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553189    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553194    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553198    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553203    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 22:33:09.642996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553207    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553212    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553216    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553220    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553226    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553230    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553234    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553239    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553243    2565 flags.go:64] FLAG: --enable-server="true"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553266    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553272    2565 flags.go:64] FLAG: --event-burst="100"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553277    2565 flags.go:64] FLAG: --event-qps="50"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553281    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553285    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553290    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553295    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553300    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553305    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553309    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553314    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553319    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553323    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553328    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553332    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553338    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 22:33:09.643803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553342    2565 flags.go:64] FLAG: --feature-gates=""
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553348    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553353    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553357    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553361    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553366    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553370    2565 flags.go:64] FLAG: --help="false"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553375    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-139-23.ec2.internal"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553380    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553384    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553388    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553393    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553398    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553403    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553407    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553411    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553416    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553419    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553424    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553428    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553432    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553436    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553441    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553446    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 22:33:09.644829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553450    2565 flags.go:64] FLAG: --lock-file=""
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553455    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553460    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553465    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553473    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553479    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553484    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553490    2565 flags.go:64] FLAG: --logging-format="text"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553495    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553499    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553504    2565 flags.go:64] FLAG: --manifest-url=""
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553509    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553515    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553519    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553525    2565 flags.go:64] FLAG: --max-pods="110"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553529    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553533    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553537    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553541    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553546    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553551    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553555    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553569    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553574    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553578    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 22:33:09.645842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553583    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553587    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553594    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553599    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553603    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553607    2565 flags.go:64] FLAG: --port="10250"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553612    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553616    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08df24f449990255e"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553621    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553626    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553630    2565 flags.go:64] FLAG: --register-node="true"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553634    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553639    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553645    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553651    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553655    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553660    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553664    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553669    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553673    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553677    2565 flags.go:64] FLAG: --runonce="false"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553681    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553686    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553690    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553695    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 22:33:09.646777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553699    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553703    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553708    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553712    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553717    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553721    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553725    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553729    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553734    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553739    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553743    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553750    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553754    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553758    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553762    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553767    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553772    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553776    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553780    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553784    2565 flags.go:64] FLAG: --v="2"
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553791    2565 flags.go:64] FLAG: --version="false"
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553798    2565 flags.go:64] FLAG: --vmodule=""
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553803    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.553808    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.553961    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 22:33:09.647531 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.553967    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.553971    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.553975    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.553979    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.553983    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.553987    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.553991    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.553995    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.553999    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554003    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554006    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554010    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554014    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554019    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554023    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554026    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554030    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554034    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554038    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554041    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 22:33:09.648388 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554046    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554049    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554053    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554057    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554060    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554065    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554069    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554074    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554080    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554084    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554087    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554091    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554095    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554098    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554103    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554106    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554110    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554114    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554117    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 22:33:09.648921 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554122    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554125    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554130    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554136    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554140    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554144    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554148    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554151    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554156    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554160    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554164    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554168    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554172    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554175    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554179    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554183    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554186    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554190    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554194    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554199    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 22:33:09.649542 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554205    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554209    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554214    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554218    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554222    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554226    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554230    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554234    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554238    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554242    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554264    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554269    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554273    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554277    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554281    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554284    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554288    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554292    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554297    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554300    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 22:33:09.650097 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554304    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 22:33:09.650670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554311    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 22:33:09.650670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554316    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 22:33:09.650670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554321    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 22:33:09.650670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554326    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 22:33:09.650670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554329    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 22:33:09.650670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554333    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 22:33:09.650670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554337    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 22:33:09.650670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554341    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 22:33:09.650670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554344    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 22:33:09.650670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554348    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 22:33:09.650670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554352    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 22:33:09.650670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554358    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 22:33:09.650670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554363    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 22:33:09.650670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554366    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 22:33:09.650670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554371    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 22:33:09.650670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554375    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 22:33:09.650670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554378    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 22:33:09.650670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.554382    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 22:33:09.651215 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.554390    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 22:33:09.651215 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.562193    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 22:33:09.651215 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.562207    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 22:33:09.651215 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.562783    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 22:33:09.651215 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.562794    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 22:33:09.651215 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562849    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 22:33:09.651215 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562854    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 22:33:09.651215 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562858    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 22:33:09.651215 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562861    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 22:33:09.651215 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562864    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 22:33:09.651215 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562867    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 22:33:09.651215 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562871    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 22:33:09.651215 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562875    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 22:33:09.651215 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562878    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562881    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562884    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562887    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562890    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562892    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562895    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562897    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562900    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562902    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562905    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562908    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562911    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562913    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562916    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562918    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562921    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562924    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562926    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562929    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 22:33:09.651627 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562932    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562935    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562937    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562940    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562943    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562945    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562948    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562950    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562953    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562956    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562958    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562961    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562964    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562966    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562969    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562971    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562974    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562976    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562980    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562983    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 22:33:09.652114 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562985    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562988    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562991    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562993    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562995    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.562998    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563000    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563003    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563006    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563010    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563013    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563016    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563019    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563021    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563024    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563028    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563032    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563035    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563037    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 22:33:09.652641 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563040    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563042    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563045    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563047    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563050    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563052    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563055    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563057    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563061    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563063    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563066    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563070    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563073    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563076    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563079    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563082    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563085    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563088    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563091    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563093    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 22:33:09.653184 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563096    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 22:33:09.653836 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563099    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 22:33:09.653836 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563105    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 22:33:09.653836 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563107    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 22:33:09.653836 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563110    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 22:33:09.653836 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563113    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 22:33:09.653836 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563116    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 22:33:09.653836 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563118    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 22:33:09.653836 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563121    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 22:33:09.653836 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563123    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 22:33:09.653836 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563126    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 22:33:09.653836 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563129    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 22:33:09.653836 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.563134    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 22:33:09.653836 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563238    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 22:33:09.653836 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563244    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 22:33:09.653836 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563267    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563270    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563273    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563275    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563278    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563281    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563283    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563286    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563289    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563291    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563295    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563298    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563301    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563305    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563309    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563312    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563314    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563317    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563320    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563323    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 22:33:09.654212 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563328    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563332    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563335    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563338    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563341    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563344    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563346    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563349    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563352    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563354    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563357    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563360    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563362    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563365    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563367    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563370    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563372    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563375    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563377    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 22:33:09.654732 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563379    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563382    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563384    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563387    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563390    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563393    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563396    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563398    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563401    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563403    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563406    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563409    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563412    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563416    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563420    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563423    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563426    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563428    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563431    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563433    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 22:33:09.655348 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563436    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563439    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563441    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563443    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563446    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563448    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563451    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563453    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563456    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563459    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563461    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563463    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563466    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563469    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563471    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563473    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563476    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563482    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563485    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563487    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 22:33:09.655971 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563490    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 22:33:09.656481 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563492    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 22:33:09.656481 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563495    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 22:33:09.656481 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563498    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 22:33:09.656481 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563500    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 22:33:09.656481 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563504    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 22:33:09.656481 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563508    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 22:33:09.656481 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563511    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 22:33:09.656481 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563513    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 22:33:09.656481 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563516    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 22:33:09.656481 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563518    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 22:33:09.656481 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563521    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 22:33:09.656481 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563523    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 22:33:09.656481 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563526    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 22:33:09.656481 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563530    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 22:33:09.656481 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563534    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 22:33:09.656481 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563537    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 22:33:09.656481 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:09.563540    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 22:33:09.657006 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.563545    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 22:33:09.657006 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.563637    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 22:33:09.657006 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.565900    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 22:33:09.657006 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.567372    2565 server.go:1019] "Starting client certificate rotation"
Jul 16 22:33:09.657006 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.567466    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 22:33:09.657006 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.567495    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 22:33:09.657006 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.569712    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 22:33:09.657006 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.570817    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 22:33:09.657006 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.578649    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 16 22:33:09.657006 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.580781    2565 log.go:25] "Validated CRI v1 image API"
Jul 16 22:33:09.657006 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.581531    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 22:33:09.657006 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.584071    2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ceef9d1c-2798-489e-8eda-b96e7fbdeadc:/dev/nvme0n1p4 f4de3228-0f3e-47fe-aa2b-e4fd98a89436:/dev/nvme0n1p3]
Jul 16 22:33:09.657369 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.584088    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 22:33:09.657404 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.590645    2565 manager.go:216] Machine: {Timestamp:2026-07-16 22:33:09.58992813 +0000 UTC m=+0.338412194 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098597 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29c15bab291e098f27d95d475579d9 SystemUUID:ec29c15b-ab29-1e09-8f27-d95d475579d9 BootID:ac2314f6-b9ac-48dd-b2b1-c29160cfa840 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:99:ce:e6:64:ed Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:99:ce:e6:64:ed Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:c0:70:1b:80:9c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 22:33:09.657404 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.590779    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 22:33:09.657404 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.590869    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 22:33:09.657404 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.593059    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 22:33:09.657404 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.593087    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-23.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 22:33:09.657404 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.593805    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 22:33:09.657404 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.593815    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 22:33:09.657404 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.593822    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.594643    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.596309    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.598779    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.599764    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.599868    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.599895    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.599912    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.599925    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.599933    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.599941    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.600265    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.600282    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.602119    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.604478    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.604508    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.605633    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.605647    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.605653    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.605659    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.605669    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.605677    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.605688    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.605694    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.605699    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.605705    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.605721    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.605727    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.606347    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.606354    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.609456    2565 server.go:1268] "Started kubelet"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.609543    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.609604    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.610034    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.611945    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.612157    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 22:33:09.657651 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.614444    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-23.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.614006    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-23.ec2.internal.18c2e5e89c4778f9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-23.ec2.internal,UID:ip-10-0-139-23.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-23.ec2.internal,},FirstTimestamp:2026-07-16 22:33:09.609240825 +0000 UTC m=+0.357724887,LastTimestamp:2026-07-16 22:33:09.609240825 +0000 UTC m=+0.357724887,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-23.ec2.internal,}"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.615016    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-23.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.615029    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.616479    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.620323    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.620336    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.620536    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.621496    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.621631    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.621670    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.621781    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.621791    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.622707    2565 factory.go:55] Registering systemd factory
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.622748    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.623009    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.623031    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-23.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.623899    2565 factory.go:153] Registering CRI-O factory
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.623915    2565 factory.go:223] Registration of the crio container factory successfully
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.623978    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.624001    2565 factory.go:103] Registering Raw factory
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.624015    2565 manager.go:1196] Started watching for new ooms in manager
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.624546    2565 manager.go:349] Starting recovery of all containers
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.635708    2565 manager.go:354] Recovery completed
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.640816    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.642645    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.642680    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.642693    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.643209    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.643222    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 22:33:09.658745 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.643238    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:33:09.659764 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.643242    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 22:33:09.659764 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.643931    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:33:09.659764 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.643954    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:33:09.659764 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.643963    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:33:09.659764 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.644491    2565 policy_none.go:49] "None policy: Start"
Jul 16 22:33:09.659764 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.644504    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 22:33:09.659764 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.644514    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 22:33:09.659764 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.645367    2565 policy_none.go:47] "Start"
Jul 16 22:33:09.659764 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.655140    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-23.ec2.internal.18c2e5e89e4589b4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-23.ec2.internal,UID:ip-10-0-139-23.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-23.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-23.ec2.internal,},FirstTimestamp:2026-07-16 22:33:09.642668468 +0000 UTC m=+0.391152530,LastTimestamp:2026-07-16 22:33:09.642668468 +0000 UTC m=+0.391152530,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-23.ec2.internal,}"
Jul 16 22:33:09.665463 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.665321    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-23.ec2.internal.18c2e5e89e45cb0c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-23.ec2.internal,UID:ip-10-0-139-23.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-23.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-23.ec2.internal,},FirstTimestamp:2026-07-16 22:33:09.642685196 +0000 UTC m=+0.391169257,LastTimestamp:2026-07-16 22:33:09.642685196 +0000 UTC m=+0.391169257,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-23.ec2.internal,}"
Jul 16 22:33:09.666883 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.666863    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9bs7q"
Jul 16 22:33:09.674034 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.673976    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-23.ec2.internal.18c2e5e89e45f924  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-23.ec2.internal,UID:ip-10-0-139-23.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-23.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-23.ec2.internal,},FirstTimestamp:2026-07-16 22:33:09.642696996 +0000 UTC m=+0.391181057,LastTimestamp:2026-07-16 22:33:09.642696996 +0000 UTC m=+0.391181057,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-23.ec2.internal,}"
Jul 16 22:33:09.685302 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.674928    2565 manager.go:337] "Starting Device Plugin manager"
Jul 16 22:33:09.685302 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.674951    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 22:33:09.685302 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.674959    2565 server.go:87] "Starting device plugin registration server"
Jul 16 22:33:09.685302 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.675151    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 22:33:09.685302 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.675161    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 22:33:09.685302 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.675287    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 22:33:09.685302 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.675356    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 22:33:09.685302 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.675364    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 22:33:09.685302 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.675814    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 22:33:09.685302 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.675865    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-23.ec2.internal\" not found"
Jul 16 22:33:09.685302 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.678700    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9bs7q"
Jul 16 22:33:09.718566 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.718512    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 22:33:09.719685 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.719670    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 22:33:09.719759 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.719711    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 22:33:09.719759 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.719732    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 22:33:09.719838 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.719767    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 22:33:09.722121 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.722099    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 22:33:09.776064 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.776046    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:33:09.776855 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.776840    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:33:09.776918 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.776872    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:33:09.776918 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.776887    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:33:09.776918 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.776910    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-23.ec2.internal"
Jul 16 22:33:09.815941 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.815921    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-23.ec2.internal"
Jul 16 22:33:09.816022 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.815943    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-23.ec2.internal\": node \"ip-10-0-139-23.ec2.internal\" not found"
Jul 16 22:33:09.819976 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.819961    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal"]
Jul 16 22:33:09.820024 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.820012    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:33:09.820729 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.820714    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:33:09.820729 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.820738    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:33:09.820879 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.820753    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:33:09.823236 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.821900    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:33:09.823236 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.822135    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal"
Jul 16 22:33:09.823236 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.822170    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:33:09.823236 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.822350    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8cf190d7ed5bcce5f5086d7c54f3dfe0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal\" (UID: \"8cf190d7ed5bcce5f5086d7c54f3dfe0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal"
Jul 16 22:33:09.823236 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.822381    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cf190d7ed5bcce5f5086d7c54f3dfe0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal\" (UID: \"8cf190d7ed5bcce5f5086d7c54f3dfe0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal"
Jul 16 22:33:09.823536 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.823519    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:33:09.823573 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.823533    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:33:09.823889 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.823866    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:33:09.823968 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.823899    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:33:09.823968 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.823876    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:33:09.823968 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.823963    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:33:09.825608 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.825592    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal"
Jul 16 22:33:09.825677 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.825621    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:33:09.826337 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.826307    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:33:09.826431 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.826341    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:33:09.826431 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.826355    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:33:09.840744 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.840722    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-23.ec2.internal\" not found" node="ip-10-0-139-23.ec2.internal"
Jul 16 22:33:09.844849 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.844835    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-23.ec2.internal\" not found" node="ip-10-0-139-23.ec2.internal"
Jul 16 22:33:09.892614 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.892595    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found"
Jul 16 22:33:09.922789 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.922771    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8cf190d7ed5bcce5f5086d7c54f3dfe0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal\" (UID: \"8cf190d7ed5bcce5f5086d7c54f3dfe0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal"
Jul 16 22:33:09.922858 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.922795    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cf190d7ed5bcce5f5086d7c54f3dfe0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal\" (UID: \"8cf190d7ed5bcce5f5086d7c54f3dfe0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal"
Jul 16 22:33:09.922858 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.922814    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f6e975acdd98c0499b69a0b9bfc9cbb0-config\") pod \"kube-apiserver-proxy-ip-10-0-139-23.ec2.internal\" (UID: \"f6e975acdd98c0499b69a0b9bfc9cbb0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal"
Jul 16 22:33:09.922946 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.922861    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cf190d7ed5bcce5f5086d7c54f3dfe0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal\" (UID: \"8cf190d7ed5bcce5f5086d7c54f3dfe0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal"
Jul 16 22:33:09.922946 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:09.922879    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8cf190d7ed5bcce5f5086d7c54f3dfe0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal\" (UID: \"8cf190d7ed5bcce5f5086d7c54f3dfe0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal"
Jul 16 22:33:09.993035 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:09.992990    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found"
Jul 16 22:33:10.023326 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:10.023303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f6e975acdd98c0499b69a0b9bfc9cbb0-config\") pod \"kube-apiserver-proxy-ip-10-0-139-23.ec2.internal\" (UID: \"f6e975acdd98c0499b69a0b9bfc9cbb0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal"
Jul 16 22:33:10.023391 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:10.023353    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f6e975acdd98c0499b69a0b9bfc9cbb0-config\") pod \"kube-apiserver-proxy-ip-10-0-139-23.ec2.internal\" (UID: \"f6e975acdd98c0499b69a0b9bfc9cbb0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal"
Jul 16 22:33:10.093612 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:10.093590    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found"
Jul 16 22:33:10.144100 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:10.144080    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal"
Jul 16 22:33:10.148294 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:10.148112    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal"
Jul 16 22:33:10.194506 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:10.194483    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found"
Jul 16 22:33:10.294773 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:10.294728    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found"
Jul 16 22:33:10.395254 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:10.395233    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found"
Jul 16 22:33:10.499748 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:10.499725    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found"
Jul 16 22:33:10.567393 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:10.567343    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 22:33:10.567494 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:10.567475    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 22:33:10.579262 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:10.579245    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 22:33:10.600605 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:10.600583    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found"
Jul 16 22:33:10.684617 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:10.684582    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 22:28:10 +0000 UTC" deadline="2028-03-07 15:32:05.135202305 +0000 UTC"
Jul 16 22:33:10.684617 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:10.684615    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14392h58m54.450590997s"
Jul 16 22:33:10.701132 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:10.701108    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found"
Jul 16 22:33:10.760980 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:10.760951    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:33:10.762234 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:10.762199    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:33:10.762234 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:10.762236    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:33:10.762430 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:10.762250    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:33:10.774244 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:10.774216    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e975acdd98c0499b69a0b9bfc9cbb0.slice/crio-02db4ff75c1929ed77da290245c51638daef6cd483cf60c5d964588e6457eabe WatchSource:0}: Error finding container 02db4ff75c1929ed77da290245c51638daef6cd483cf60c5d964588e6457eabe: Status 404 returned error can't find the container with id 02db4ff75c1929ed77da290245c51638daef6cd483cf60c5d964588e6457eabe
Jul 16 22:33:10.782864 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:10.782843    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:33:10.788452 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:10.788427    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf190d7ed5bcce5f5086d7c54f3dfe0.slice/crio-e7de9dea47030816be87bf72230ec5f70794f5ad58418761e13026b2492bf9af WatchSource:0}: Error finding container e7de9dea47030816be87bf72230ec5f70794f5ad58418761e13026b2492bf9af: Status 404 returned error can't find the container with id e7de9dea47030816be87bf72230ec5f70794f5ad58418761e13026b2492bf9af
Jul 16 22:33:10.801541 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:10.801516    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found"
Jul 16 22:33:10.902572 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:10.902508    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-23.ec2.internal\" not found"
Jul 16 22:33:10.907773 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:10.907756    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 22:33:10.977396 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:10.977372    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 22:33:11.021243 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.021221    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal"
Jul 16 22:33:11.036024 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.036002    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 22:33:11.037008 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.036995    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal"
Jul 16 22:33:11.046389 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.046374    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 22:33:11.442885 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.442855    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 22:33:11.601127 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.601094    2565 apiserver.go:52] "Watching apiserver"
Jul 16 22:33:11.606704 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.606680    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 22:33:11.607169 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.607141    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-gkmrf","openshift-cluster-node-tuning-operator/tuned-9nmvs","openshift-image-registry/node-ca-r7xmd","openshift-multus/multus-8c8p5","openshift-multus/multus-additional-cni-plugins-z5q2j","kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z","openshift-dns/node-resolver-cgrb2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal","openshift-multus/network-metrics-daemon-pbp8p","openshift-network-diagnostics/network-check-target-lvsgm","openshift-network-operator/iptables-alerter-b5j8x","openshift-ovn-kubernetes/ovnkube-node-s4s28","kube-system/global-pull-secret-syncer-lgwp5"]
Jul 16 22:33:11.609967 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.609939    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.613415 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.613396    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.613526 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.613481    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 22:33:11.613685 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.613660    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8pkjh\""
Jul 16 22:33:11.614169 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.614147    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 22:33:11.614301 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.614149    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 22:33:11.615736 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.615642    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r7xmd"
Jul 16 22:33:11.615936 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.615861    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 22:33:11.615936 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.615920    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 22:33:11.616216 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.616196    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7gbsf\""
Jul 16 22:33:11.616322 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.616236    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 22:33:11.616516 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.616495    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 22:33:11.616596 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.616557    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 22:33:11.616654 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.616557    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 22:33:11.619064 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.619040    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 22:33:11.619715 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.619694    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 22:33:11.620661 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.620482    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 22:33:11.620661 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.620577    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.621415 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.621391    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-85sfg\""
Jul 16 22:33:11.623159 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.623138    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 22:33:11.625206 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.625179    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.625346 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.625308    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 22:33:11.625346 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.625315    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 22:33:11.625557 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.625539    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 22:33:11.625919 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.625901    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 22:33:11.627283 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.627262    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6c898\""
Jul 16 22:33:11.627685 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.627665    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-774t7\""
Jul 16 22:33:11.627947 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.627925    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 22:33:11.628173 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.628155    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cgrb2"
Jul 16 22:33:11.628254 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.628210    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 22:33:11.630316 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.630296    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 22:33:11.630572 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.630553    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 22:33:11.630660 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.630595    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b5j8x"
Jul 16 22:33:11.630984 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.630965    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rcqq4\""
Jul 16 22:33:11.632352 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.631942    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4pm7\" (UniqueName: \"kubernetes.io/projected/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-kube-api-access-p4pm7\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.632352 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.631986    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7a448609-0972-4f56-953a-e0231a116488-registration-dir\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.632352 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632013    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-run-openvswitch\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.632352 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632040    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8bce50b5-5616-4134-b9ee-f1264bd569dd-serviceca\") pod \"node-ca-r7xmd\" (UID: \"8bce50b5-5616-4134-b9ee-f1264bd569dd\") " pod="openshift-image-registry/node-ca-r7xmd"
Jul 16 22:33:11.632352 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632065    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljpxm\" (UniqueName: \"kubernetes.io/projected/2190bbd8-a2fb-4522-9254-092c9f78d6bc-kube-api-access-ljpxm\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.632352 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632099    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-os-release\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.632352 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632123    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-host-var-lib-cni-bin\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.632352 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632149    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwhdp\" (UniqueName: \"kubernetes.io/projected/778abed0-00de-488b-84e0-26c1497fa68a-kube-api-access-kwhdp\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.632352 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632174    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-multus-socket-dir-parent\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.632352 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632197    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-etc-kubernetes\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.632352 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632221    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.632352 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632245    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-cni-bin\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.632352 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632267    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-cnibin\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.632352 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632340    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-run-systemd\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.633177 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632374    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-system-cni-dir\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.633177 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632415    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-host-var-lib-cni-multus\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.633177 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632445    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-multus-cni-dir\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.633177 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632470    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-host-run-k8s-cni-cncf-io\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.633177 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632494    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-hostroot\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.633177 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632518    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-slash\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.633177 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-node-log\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.633177 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632567    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-log-socket\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.633177 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632632    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-run-ovn-kubernetes\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.633177 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632660    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/778abed0-00de-488b-84e0-26c1497fa68a-ovnkube-script-lib\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.633177 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632686    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2190bbd8-a2fb-4522-9254-092c9f78d6bc-multus-daemon-config\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.633177 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8bce50b5-5616-4134-b9ee-f1264bd569dd-host\") pod \"node-ca-r7xmd\" (UID: \"8bce50b5-5616-4134-b9ee-f1264bd569dd\") " pod="openshift-image-registry/node-ca-r7xmd"
Jul 16 22:33:11.633177 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632751    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-system-cni-dir\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.633177 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7a448609-0972-4f56-953a-e0231a116488-etc-selinux\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.633964 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632844    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-host-var-lib-kubelet\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.633964 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.633964 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632935    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-run-netns\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.633964 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632960    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/778abed0-00de-488b-84e0-26c1497fa68a-env-overrides\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.633964 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.632986    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nlv7\" (UniqueName: \"kubernetes.io/projected/7a448609-0972-4f56-953a-e0231a116488-kube-api-access-5nlv7\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.633964 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633013    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-cni-netd\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.633964 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633043    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.633964 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/778abed0-00de-488b-84e0-26c1497fa68a-ovnkube-config\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.633964 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633097    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr2vc\" (UniqueName: \"kubernetes.io/projected/8bce50b5-5616-4134-b9ee-f1264bd569dd-kube-api-access-lr2vc\") pod \"node-ca-r7xmd\" (UID: \"8bce50b5-5616-4134-b9ee-f1264bd569dd\") " pod="openshift-image-registry/node-ca-r7xmd"
Jul 16 22:33:11.633964 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633167    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-cnibin\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.633964 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633228    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7a448609-0972-4f56-953a-e0231a116488-socket-dir\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.633964 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633255    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-var-lib-openvswitch\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.633964 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633302    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-run-ovn\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.634706 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633345    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.634706 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633372    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7a448609-0972-4f56-953a-e0231a116488-device-dir\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.634706 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-kubelet\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.634706 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633449    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-etc-openvswitch\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.634706 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633473    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/778abed0-00de-488b-84e0-26c1497fa68a-ovn-node-metrics-cert\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.634706 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633497    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-host-run-netns\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.634706 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-multus-conf-dir\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.634706 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633546    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7a448609-0972-4f56-953a-e0231a116488-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.634706 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633570    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-host-run-multus-certs\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.634706 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633614    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-os-release\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.634706 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633637    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.634706 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633664    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7a448609-0972-4f56-953a-e0231a116488-sys-fs\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.634706 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633692    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-systemd-units\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.634706 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633718    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 22:33:11.635449 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633733    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2190bbd8-a2fb-4522-9254-092c9f78d6bc-cni-binary-copy\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.635449 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633718    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 22:33:11.635449 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633960    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 22:33:11.635449 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.633960    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5v7lw\""
Jul 16 22:33:11.635449 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.634312    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gkmrf"
Jul 16 22:33:11.636409 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.636380    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 22:33:11.636553 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.636506    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 22:33:11.636641 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.636562    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 22:33:11.636641 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.636578    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vn57h\""
Jul 16 22:33:11.636862 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.636842    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.639018 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.638995    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 22:33:11.639116 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.638995    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fpg4c\""
Jul 16 22:33:11.639116 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.639023    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 22:33:11.639591 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.639571    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:11.639710 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:11.639637    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbp8p" podUID="d8dfffe2-b127-481e-90c8-d0372f409044"
Jul 16 22:33:11.642800 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.642177    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:11.642800 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:11.642243    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvsgm" podUID="cca1b284-455e-4795-a955-930aad902e9f"
Jul 16 22:33:11.644864 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.644845    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:11.645008 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:11.644912    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgwp5" podUID="61e408a2-709b-4928-b6b2-018a505d8d4d"
Jul 16 22:33:11.666932 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.666909    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kzpf9"
Jul 16 22:33:11.675769 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.675637    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kzpf9"
Jul 16 22:33:11.723412 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.723313    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 22:33:11.725292 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.725228    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal" event={"ID":"8cf190d7ed5bcce5f5086d7c54f3dfe0","Type":"ContainerStarted","Data":"e7de9dea47030816be87bf72230ec5f70794f5ad58418761e13026b2492bf9af"}
Jul 16 22:33:11.726576 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.726539    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal" event={"ID":"f6e975acdd98c0499b69a0b9bfc9cbb0","Type":"ContainerStarted","Data":"02db4ff75c1929ed77da290245c51638daef6cd483cf60c5d964588e6457eabe"}
Jul 16 22:33:11.734979 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.734950    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-multus-cni-dir\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.735108 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735013    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-host-run-k8s-cni-cncf-io\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.735108 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735054    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-hostroot\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.735108 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735078    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-slash\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.735108 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-node-log\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.735381 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735127    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-log-socket\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.735381 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735151    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-run-ovn-kubernetes\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.735381 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/778abed0-00de-488b-84e0-26c1497fa68a-ovnkube-script-lib\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.735381 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735189    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-node-log\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.735381 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735202    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2190bbd8-a2fb-4522-9254-092c9f78d6bc-multus-daemon-config\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.735381 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735217    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-hostroot\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.735381 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735232    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-run-ovn-kubernetes\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.735381 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735242    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4045c278-1bda-4f6c-9db4-80c4c9c3b202-host-slash\") pod \"iptables-alerter-b5j8x\" (UID: \"4045c278-1bda-4f6c-9db4-80c4c9c3b202\") " pod="openshift-network-operator/iptables-alerter-b5j8x"
Jul 16 22:33:11.735381 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735180    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-slash\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.735381 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735280    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-multus-cni-dir\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.735381 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-run\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.735381 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8bce50b5-5616-4134-b9ee-f1264bd569dd-host\") pod \"node-ca-r7xmd\" (UID: \"8bce50b5-5616-4134-b9ee-f1264bd569dd\") " pod="openshift-image-registry/node-ca-r7xmd"
Jul 16 22:33:11.735381 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735325    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-host-run-k8s-cni-cncf-io\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.735381 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735217    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-log-socket\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.735381 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735349    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-system-cni-dir\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.736098 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8bce50b5-5616-4134-b9ee-f1264bd569dd-host\") pod \"node-ca-r7xmd\" (UID: \"8bce50b5-5616-4134-b9ee-f1264bd569dd\") " pod="openshift-image-registry/node-ca-r7xmd"
Jul 16 22:33:11.736098 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735375    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7a448609-0972-4f56-953a-e0231a116488-etc-selinux\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.736098 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735410    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-var-lib-kubelet\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.736098 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735411    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-system-cni-dir\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.736098 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-host-var-lib-kubelet\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.736098 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735462    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.736098 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7a448609-0972-4f56-953a-e0231a116488-etc-selinux\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.736098 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735488    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-run-netns\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.736098 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735540    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-run-netns\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.736098 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735535    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-host-var-lib-kubelet\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.736098 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/778abed0-00de-488b-84e0-26c1497fa68a-env-overrides\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.736098 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735599    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-host\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.736098 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nlv7\" (UniqueName: \"kubernetes.io/projected/7a448609-0972-4f56-953a-e0231a116488-kube-api-access-5nlv7\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.736098 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735647    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.736990 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735664    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-cni-netd\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.736990 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735693    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.736990 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735722    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/778abed0-00de-488b-84e0-26c1497fa68a-ovnkube-config\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.736990 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735752    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlkm9\" (UniqueName: \"kubernetes.io/projected/d8dfffe2-b127-481e-90c8-d0372f409044-kube-api-access-dlkm9\") pod \"network-metrics-daemon-pbp8p\" (UID: \"d8dfffe2-b127-481e-90c8-d0372f409044\") " pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:11.736990 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735779    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/61e408a2-709b-4928-b6b2-018a505d8d4d-dbus\") pod \"global-pull-secret-syncer-lgwp5\" (UID: \"61e408a2-709b-4928-b6b2-018a505d8d4d\") " pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:11.736990 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735807    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/57b0b632-a518-4519-a7f6-3e18e8cabdb4-agent-certs\") pod \"konnectivity-agent-gkmrf\" (UID: \"57b0b632-a518-4519-a7f6-3e18e8cabdb4\") " pod="kube-system/konnectivity-agent-gkmrf"
Jul 16 22:33:11.736990 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735814    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-cni-netd\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.736990 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2vc\" (UniqueName: \"kubernetes.io/projected/8bce50b5-5616-4134-b9ee-f1264bd569dd-kube-api-access-lr2vc\") pod \"node-ca-r7xmd\" (UID: \"8bce50b5-5616-4134-b9ee-f1264bd569dd\") " pod="openshift-image-registry/node-ca-r7xmd"
Jul 16 22:33:11.736990 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735776    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.736990 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-cnibin\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.736990 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735873    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2190bbd8-a2fb-4522-9254-092c9f78d6bc-multus-daemon-config\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.736990 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/778abed0-00de-488b-84e0-26c1497fa68a-ovnkube-script-lib\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.736990 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735890    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7a448609-0972-4f56-953a-e0231a116488-socket-dir\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.736990 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735952    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-cnibin\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.737737 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735969    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-var-lib-openvswitch\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.737737 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735987    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/778abed0-00de-488b-84e0-26c1497fa68a-env-overrides\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.737737 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.735997    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-run-ovn\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.737737 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/57b0b632-a518-4519-a7f6-3e18e8cabdb4-konnectivity-ca\") pod \"konnectivity-agent-gkmrf\" (UID: \"57b0b632-a518-4519-a7f6-3e18e8cabdb4\") " pod="kube-system/konnectivity-agent-gkmrf"
Jul 16 22:33:11.737737 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7a448609-0972-4f56-953a-e0231a116488-socket-dir\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.737737 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736030    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-var-lib-openvswitch\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.737737 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736047    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-run-ovn\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.737737 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736055    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-modprobe-d\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.737737 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736080    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-sysconfig\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.737737 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736106    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.737737 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736132    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7a448609-0972-4f56-953a-e0231a116488-device-dir\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.737737 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736158    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-kubelet\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.737737 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736213    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7a448609-0972-4f56-953a-e0231a116488-device-dir\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.737737 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736226    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-etc-openvswitch\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.738448 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736253    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/778abed0-00de-488b-84e0-26c1497fa68a-ovn-node-metrics-cert\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.738448 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736266    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-kubelet\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.738448 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736274    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/778abed0-00de-488b-84e0-26c1497fa68a-ovnkube-config\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.738448 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736279    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-sysctl-conf\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.738448 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736309    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-host-run-netns\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.738448 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736311    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-etc-openvswitch\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.738448 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-multus-conf-dir\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.738448 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736362    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7a448609-0972-4f56-953a-e0231a116488-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.738448 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736364    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-host-run-netns\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.738448 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736391    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-multus-conf-dir\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.738448 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736390    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-tuned\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.738448 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736425    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sznvk\" (UniqueName: \"kubernetes.io/projected/31f32160-6b80-42d4-b1e6-daf6395f0e13-kube-api-access-sznvk\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.738448 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736444    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-host-run-multus-certs\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.738448 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736460    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7a448609-0972-4f56-953a-e0231a116488-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.739501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736473    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-os-release\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.739501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736497    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.739501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736510    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-host-run-multus-certs\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.739501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736515    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7a448609-0972-4f56-953a-e0231a116488-sys-fs\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.739501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736555    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-os-release\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.739501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736564    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7a448609-0972-4f56-953a-e0231a116488-sys-fs\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.739501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-systemd-units\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.739501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736603    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-systemd-units\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.739501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736602    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.739501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736603    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bba53725-3faf-4d5c-8641-848a055f65df-hosts-file\") pod \"node-resolver-cgrb2\" (UID: \"bba53725-3faf-4d5c-8641-848a055f65df\") " pod="openshift-dns/node-resolver-cgrb2"
Jul 16 22:33:11.739501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2190bbd8-a2fb-4522-9254-092c9f78d6bc-cni-binary-copy\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.739501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736673    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4pm7\" (UniqueName: \"kubernetes.io/projected/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-kube-api-access-p4pm7\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.739501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736660    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 22:33:11.739501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736702    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret\") pod \"global-pull-secret-syncer-lgwp5\" (UID: \"61e408a2-709b-4928-b6b2-018a505d8d4d\") " pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:11.740409 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736807    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bba53725-3faf-4d5c-8641-848a055f65df-tmp-dir\") pod \"node-resolver-cgrb2\" (UID: \"bba53725-3faf-4d5c-8641-848a055f65df\") " pod="openshift-dns/node-resolver-cgrb2"
Jul 16 22:33:11.740409 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7a448609-0972-4f56-953a-e0231a116488-registration-dir\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.740409 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736852    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-run-openvswitch\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.740409 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736870    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4045c278-1bda-4f6c-9db4-80c4c9c3b202-iptables-alerter-script\") pod \"iptables-alerter-b5j8x\" (UID: \"4045c278-1bda-4f6c-9db4-80c4c9c3b202\") " pod="openshift-network-operator/iptables-alerter-b5j8x"
Jul 16 22:33:11.740409 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736896    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8bce50b5-5616-4134-b9ee-f1264bd569dd-serviceca\") pod \"node-ca-r7xmd\" (UID: \"8bce50b5-5616-4134-b9ee-f1264bd569dd\") " pod="openshift-image-registry/node-ca-r7xmd"
Jul 16 22:33:11.740409 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736921    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljpxm\" (UniqueName: \"kubernetes.io/projected/2190bbd8-a2fb-4522-9254-092c9f78d6bc-kube-api-access-ljpxm\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.740409 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736942    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7a448609-0972-4f56-953a-e0231a116488-registration-dir\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.740409 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736960    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-sysctl-d\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.740409 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736965    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.740409 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736998    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-os-release\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.740409 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737028    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-host-var-lib-cni-bin\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.740409 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.736945    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-run-openvswitch\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.740409 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwhdp\" (UniqueName: \"kubernetes.io/projected/778abed0-00de-488b-84e0-26c1497fa68a-kube-api-access-kwhdp\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.740409 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737083    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pbd2\" (UniqueName: \"kubernetes.io/projected/4045c278-1bda-4f6c-9db4-80c4c9c3b202-kube-api-access-8pbd2\") pod \"iptables-alerter-b5j8x\" (UID: \"4045c278-1bda-4f6c-9db4-80c4c9c3b202\") " pod="openshift-network-operator/iptables-alerter-b5j8x"
Jul 16 22:33:11.741298 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737091    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-host-var-lib-cni-bin\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.741298 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737057    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-os-release\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.741298 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737113    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-multus-socket-dir-parent\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.741298 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-etc-kubernetes\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.741298 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737155    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2190bbd8-a2fb-4522-9254-092c9f78d6bc-cni-binary-copy\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.741298 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.741298 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737180    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-multus-socket-dir-parent\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.741298 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737190    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-cni-bin\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.741298 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737213    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-etc-kubernetes\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.741298 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737217    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs\") pod \"network-metrics-daemon-pbp8p\" (UID: \"d8dfffe2-b127-481e-90c8-d0372f409044\") " pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:11.741298 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737247    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-kubernetes\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.741298 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737272    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-host-cni-bin\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.741298 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737278    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-sys\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.741298 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737349    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-cnibin\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.741298 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8bce50b5-5616-4134-b9ee-f1264bd569dd-serviceca\") pod \"node-ca-r7xmd\" (UID: \"8bce50b5-5616-4134-b9ee-f1264bd569dd\") " pod="openshift-image-registry/node-ca-r7xmd"
Jul 16 22:33:11.742436 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737385    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-run-systemd\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.742436 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737417    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-cnibin\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.742436 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4qmh\" (UniqueName: \"kubernetes.io/projected/bba53725-3faf-4d5c-8641-848a055f65df-kube-api-access-z4qmh\") pod \"node-resolver-cgrb2\" (UID: \"bba53725-3faf-4d5c-8641-848a055f65df\") " pod="openshift-dns/node-resolver-cgrb2"
Jul 16 22:33:11.742436 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737451    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/778abed0-00de-488b-84e0-26c1497fa68a-run-systemd\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.742436 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737453    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-systemd\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.742436 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737506    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31f32160-6b80-42d4-b1e6-daf6395f0e13-tmp\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.742436 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737525    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-system-cni-dir\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.742436 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737545    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-host-var-lib-cni-multus\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.742436 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737567    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc49c\" (UniqueName: \"kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c\") pod \"network-check-target-lvsgm\" (UID: \"cca1b284-455e-4795-a955-930aad902e9f\") " pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:11.742436 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737590    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/61e408a2-709b-4928-b6b2-018a505d8d4d-kubelet-config\") pod \"global-pull-secret-syncer-lgwp5\" (UID: \"61e408a2-709b-4928-b6b2-018a505d8d4d\") " pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:11.742436 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737570    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-system-cni-dir\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.742436 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737615    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2190bbd8-a2fb-4522-9254-092c9f78d6bc-host-var-lib-cni-multus\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.742436 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-lib-modules\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.742436 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.737656    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.744171 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.741460    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/778abed0-00de-488b-84e0-26c1497fa68a-ovn-node-metrics-cert\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.744811 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.744783    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nlv7\" (UniqueName: \"kubernetes.io/projected/7a448609-0972-4f56-953a-e0231a116488-kube-api-access-5nlv7\") pod \"aws-ebs-csi-driver-node-ttr5z\" (UID: \"7a448609-0972-4f56-953a-e0231a116488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.745034 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.744969    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2vc\" (UniqueName: \"kubernetes.io/projected/8bce50b5-5616-4134-b9ee-f1264bd569dd-kube-api-access-lr2vc\") pod \"node-ca-r7xmd\" (UID: \"8bce50b5-5616-4134-b9ee-f1264bd569dd\") " pod="openshift-image-registry/node-ca-r7xmd"
Jul 16 22:33:11.746772 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.746748    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4pm7\" (UniqueName: \"kubernetes.io/projected/d0a97cb4-0345-49f0-a7ca-7d35a24a59f5-kube-api-access-p4pm7\") pod \"multus-additional-cni-plugins-z5q2j\" (UID: \"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5\") " pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.747145 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.747110    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljpxm\" (UniqueName: \"kubernetes.io/projected/2190bbd8-a2fb-4522-9254-092c9f78d6bc-kube-api-access-ljpxm\") pod \"multus-8c8p5\" (UID: \"2190bbd8-a2fb-4522-9254-092c9f78d6bc\") " pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.747418 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.747399    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwhdp\" (UniqueName: \"kubernetes.io/projected/778abed0-00de-488b-84e0-26c1497fa68a-kube-api-access-kwhdp\") pod \"ovnkube-node-s4s28\" (UID: \"778abed0-00de-488b-84e0-26c1497fa68a\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.838098 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838058    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4045c278-1bda-4f6c-9db4-80c4c9c3b202-host-slash\") pod \"iptables-alerter-b5j8x\" (UID: \"4045c278-1bda-4f6c-9db4-80c4c9c3b202\") " pod="openshift-network-operator/iptables-alerter-b5j8x"
Jul 16 22:33:11.838098 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838101    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-run\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.838365 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-var-lib-kubelet\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.838365 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838159    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-host\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.838365 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838184    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4045c278-1bda-4f6c-9db4-80c4c9c3b202-host-slash\") pod \"iptables-alerter-b5j8x\" (UID: \"4045c278-1bda-4f6c-9db4-80c4c9c3b202\") " pod="openshift-network-operator/iptables-alerter-b5j8x"
Jul 16 22:33:11.838365 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838191    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlkm9\" (UniqueName: \"kubernetes.io/projected/d8dfffe2-b127-481e-90c8-d0372f409044-kube-api-access-dlkm9\") pod \"network-metrics-daemon-pbp8p\" (UID: \"d8dfffe2-b127-481e-90c8-d0372f409044\") " pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:11.838365 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838214    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-run\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.838365 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838244    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/61e408a2-709b-4928-b6b2-018a505d8d4d-dbus\") pod \"global-pull-secret-syncer-lgwp5\" (UID: \"61e408a2-709b-4928-b6b2-018a505d8d4d\") " pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:11.838365 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838287    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/57b0b632-a518-4519-a7f6-3e18e8cabdb4-agent-certs\") pod \"konnectivity-agent-gkmrf\" (UID: \"57b0b632-a518-4519-a7f6-3e18e8cabdb4\") " pod="kube-system/konnectivity-agent-gkmrf"
Jul 16 22:33:11.838365 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/57b0b632-a518-4519-a7f6-3e18e8cabdb4-konnectivity-ca\") pod \"konnectivity-agent-gkmrf\" (UID: \"57b0b632-a518-4519-a7f6-3e18e8cabdb4\") " pod="kube-system/konnectivity-agent-gkmrf"
Jul 16 22:33:11.838365 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838339    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-var-lib-kubelet\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.838365 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838368    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-modprobe-d\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.838876 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838392    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-sysconfig\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.838876 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838431    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-sysctl-conf\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.838876 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838461    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-host\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.838876 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838474    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-tuned\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.838876 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838508    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sznvk\" (UniqueName: \"kubernetes.io/projected/31f32160-6b80-42d4-b1e6-daf6395f0e13-kube-api-access-sznvk\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.838876 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bba53725-3faf-4d5c-8641-848a055f65df-hosts-file\") pod \"node-resolver-cgrb2\" (UID: \"bba53725-3faf-4d5c-8641-848a055f65df\") " pod="openshift-dns/node-resolver-cgrb2"
Jul 16 22:33:11.838876 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838548    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-sysconfig\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.838876 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838573    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret\") pod \"global-pull-secret-syncer-lgwp5\" (UID: \"61e408a2-709b-4928-b6b2-018a505d8d4d\") " pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:11.838876 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838601    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bba53725-3faf-4d5c-8641-848a055f65df-tmp-dir\") pod \"node-resolver-cgrb2\" (UID: \"bba53725-3faf-4d5c-8641-848a055f65df\") " pod="openshift-dns/node-resolver-cgrb2"
Jul 16 22:33:11.838876 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4045c278-1bda-4f6c-9db4-80c4c9c3b202-iptables-alerter-script\") pod \"iptables-alerter-b5j8x\" (UID: \"4045c278-1bda-4f6c-9db4-80c4c9c3b202\") " pod="openshift-network-operator/iptables-alerter-b5j8x"
Jul 16 22:33:11.838876 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838682    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-sysctl-d\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.838876 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pbd2\" (UniqueName: \"kubernetes.io/projected/4045c278-1bda-4f6c-9db4-80c4c9c3b202-kube-api-access-8pbd2\") pod \"iptables-alerter-b5j8x\" (UID: \"4045c278-1bda-4f6c-9db4-80c4c9c3b202\") " pod="openshift-network-operator/iptables-alerter-b5j8x"
Jul 16 22:33:11.838876 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838746    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs\") pod \"network-metrics-daemon-pbp8p\" (UID: \"d8dfffe2-b127-481e-90c8-d0372f409044\") " pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:11.838876 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:11.838857    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:11.838876 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838874    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-modprobe-d\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.839661 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:11.838939    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs podName:d8dfffe2-b127-481e-90c8-d0372f409044 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:12.338901618 +0000 UTC m=+3.087385691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs") pod "network-metrics-daemon-pbp8p" (UID: "d8dfffe2-b127-481e-90c8-d0372f409044") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:11.839661 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-kubernetes\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.839661 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:11.838975    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:11.839661 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-sys\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.839661 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.839429    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4qmh\" (UniqueName: \"kubernetes.io/projected/bba53725-3faf-4d5c-8641-848a055f65df-kube-api-access-z4qmh\") pod \"node-resolver-cgrb2\" (UID: \"bba53725-3faf-4d5c-8641-848a055f65df\") " pod="openshift-dns/node-resolver-cgrb2"
Jul 16 22:33:11.839661 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.839500    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-systemd\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.839661 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.839530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31f32160-6b80-42d4-b1e6-daf6395f0e13-tmp\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.839661 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.839571    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc49c\" (UniqueName: \"kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c\") pod \"network-check-target-lvsgm\" (UID: \"cca1b284-455e-4795-a955-930aad902e9f\") " pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:11.839661 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.839610    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/61e408a2-709b-4928-b6b2-018a505d8d4d-kubelet-config\") pod \"global-pull-secret-syncer-lgwp5\" (UID: \"61e408a2-709b-4928-b6b2-018a505d8d4d\") " pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:11.839661 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.839645    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-lib-modules\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.840174 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.839703    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/57b0b632-a518-4519-a7f6-3e18e8cabdb4-konnectivity-ca\") pod \"konnectivity-agent-gkmrf\" (UID: \"57b0b632-a518-4519-a7f6-3e18e8cabdb4\") " pod="kube-system/konnectivity-agent-gkmrf"
Jul 16 22:33:11.840174 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.839803    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-sysctl-conf\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.840174 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.839869    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-sysctl-d\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.842668 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.840529    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4045c278-1bda-4f6c-9db4-80c4c9c3b202-iptables-alerter-script\") pod \"iptables-alerter-b5j8x\" (UID: \"4045c278-1bda-4f6c-9db4-80c4c9c3b202\") " pod="openshift-network-operator/iptables-alerter-b5j8x"
Jul 16 22:33:11.842668 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.838512    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/61e408a2-709b-4928-b6b2-018a505d8d4d-dbus\") pod \"global-pull-secret-syncer-lgwp5\" (UID: \"61e408a2-709b-4928-b6b2-018a505d8d4d\") " pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:11.842668 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.841261    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-systemd\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.842668 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:11.841485    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret podName:61e408a2-709b-4928-b6b2-018a505d8d4d nodeName:}" failed. No retries permitted until 2026-07-16 22:33:12.341463358 +0000 UTC m=+3.089947408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret") pod "global-pull-secret-syncer-lgwp5" (UID: "61e408a2-709b-4928-b6b2-018a505d8d4d") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:11.842668 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.841518    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bba53725-3faf-4d5c-8641-848a055f65df-tmp-dir\") pod \"node-resolver-cgrb2\" (UID: \"bba53725-3faf-4d5c-8641-848a055f65df\") " pod="openshift-dns/node-resolver-cgrb2"
Jul 16 22:33:11.842668 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.841660    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-lib-modules\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.842668 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.841738    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/61e408a2-709b-4928-b6b2-018a505d8d4d-kubelet-config\") pod \"global-pull-secret-syncer-lgwp5\" (UID: \"61e408a2-709b-4928-b6b2-018a505d8d4d\") " pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:11.842668 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.841809    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bba53725-3faf-4d5c-8641-848a055f65df-hosts-file\") pod \"node-resolver-cgrb2\" (UID: \"bba53725-3faf-4d5c-8641-848a055f65df\") " pod="openshift-dns/node-resolver-cgrb2"
Jul 16 22:33:11.842668 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.841889    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-kubernetes\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.842668 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.841950    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31f32160-6b80-42d4-b1e6-daf6395f0e13-sys\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.843389 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.843367    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/57b0b632-a518-4519-a7f6-3e18e8cabdb4-agent-certs\") pod \"konnectivity-agent-gkmrf\" (UID: \"57b0b632-a518-4519-a7f6-3e18e8cabdb4\") " pod="kube-system/konnectivity-agent-gkmrf"
Jul 16 22:33:11.845177 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.845152    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/31f32160-6b80-42d4-b1e6-daf6395f0e13-etc-tuned\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.846681 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.846652    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31f32160-6b80-42d4-b1e6-daf6395f0e13-tmp\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.850393 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.850303    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlkm9\" (UniqueName: \"kubernetes.io/projected/d8dfffe2-b127-481e-90c8-d0372f409044-kube-api-access-dlkm9\") pod \"network-metrics-daemon-pbp8p\" (UID: \"d8dfffe2-b127-481e-90c8-d0372f409044\") " pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:11.850551 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:11.850346    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:33:11.850674 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:11.850563    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:33:11.850674 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:11.850580    2565 projected.go:196] Error preparing data for projected volume kube-api-access-xc49c for pod openshift-network-diagnostics/network-check-target-lvsgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:11.850674 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:11.850640    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c podName:cca1b284-455e-4795-a955-930aad902e9f nodeName:}" failed. No retries permitted until 2026-07-16 22:33:12.350622766 +0000 UTC m=+3.099106837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xc49c" (UniqueName: "kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c") pod "network-check-target-lvsgm" (UID: "cca1b284-455e-4795-a955-930aad902e9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:11.851457 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.851413    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pbd2\" (UniqueName: \"kubernetes.io/projected/4045c278-1bda-4f6c-9db4-80c4c9c3b202-kube-api-access-8pbd2\") pod \"iptables-alerter-b5j8x\" (UID: \"4045c278-1bda-4f6c-9db4-80c4c9c3b202\") " pod="openshift-network-operator/iptables-alerter-b5j8x"
Jul 16 22:33:11.851826 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.851802    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sznvk\" (UniqueName: \"kubernetes.io/projected/31f32160-6b80-42d4-b1e6-daf6395f0e13-kube-api-access-sznvk\") pod \"tuned-9nmvs\" (UID: \"31f32160-6b80-42d4-b1e6-daf6395f0e13\") " pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:11.853073 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.853047    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4qmh\" (UniqueName: \"kubernetes.io/projected/bba53725-3faf-4d5c-8641-848a055f65df-kube-api-access-z4qmh\") pod \"node-resolver-cgrb2\" (UID: \"bba53725-3faf-4d5c-8641-848a055f65df\") " pod="openshift-dns/node-resolver-cgrb2"
Jul 16 22:33:11.929533 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.929501    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z"
Jul 16 22:33:11.937537 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.937517    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:11.948197 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.948168    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cgrb2"
Jul 16 22:33:11.955051 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.955028    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r7xmd"
Jul 16 22:33:11.963027 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.963005    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8c8p5"
Jul 16 22:33:11.969914 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.969895    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z5q2j"
Jul 16 22:33:11.977782 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.977698    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b5j8x"
Jul 16 22:33:11.985767 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.985746    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gkmrf"
Jul 16 22:33:11.990695 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:11.990670    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9nmvs"
Jul 16 22:33:12.343009 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:12.342920    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs\") pod \"network-metrics-daemon-pbp8p\" (UID: \"d8dfffe2-b127-481e-90c8-d0372f409044\") " pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:12.343009 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:12.343001    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret\") pod \"global-pull-secret-syncer-lgwp5\" (UID: \"61e408a2-709b-4928-b6b2-018a505d8d4d\") " pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:12.343247 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:12.343105    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:12.343247 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:12.343171    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret podName:61e408a2-709b-4928-b6b2-018a505d8d4d nodeName:}" failed. No retries permitted until 2026-07-16 22:33:13.34315573 +0000 UTC m=+4.091639780 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret") pod "global-pull-secret-syncer-lgwp5" (UID: "61e408a2-709b-4928-b6b2-018a505d8d4d") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:12.343247 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:12.343107    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:12.343424 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:12.343281    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs podName:d8dfffe2-b127-481e-90c8-d0372f409044 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:13.343264757 +0000 UTC m=+4.091748827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs") pod "network-metrics-daemon-pbp8p" (UID: "d8dfffe2-b127-481e-90c8-d0372f409044") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:12.444041 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:12.443991    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc49c\" (UniqueName: \"kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c\") pod \"network-check-target-lvsgm\" (UID: \"cca1b284-455e-4795-a955-930aad902e9f\") " pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:12.444222 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:12.444192    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:33:12.444222 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:12.444223    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:33:12.444443 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:12.444235    2565 projected.go:196] Error preparing data for projected volume kube-api-access-xc49c for pod openshift-network-diagnostics/network-check-target-lvsgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:12.444443 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:12.444305    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c podName:cca1b284-455e-4795-a955-930aad902e9f nodeName:}" failed. No retries permitted until 2026-07-16 22:33:13.444285509 +0000 UTC m=+4.192769561 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xc49c" (UniqueName: "kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c") pod "network-check-target-lvsgm" (UID: "cca1b284-455e-4795-a955-930aad902e9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:12.679852 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:12.679739    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 22:28:12 +0000 UTC" deadline="2028-01-24 22:22:43.112455916 +0000 UTC"
Jul 16 22:33:12.679852 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:12.679798    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13367h49m30.432662314s"
Jul 16 22:33:12.817226 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:12.817178    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a448609_0972_4f56_953a_e0231a116488.slice/crio-f768c9b55a573db870c715bd20f81cfb6a1932a1b52bb409b3d8d78346a5f0ac WatchSource:0}: Error finding container f768c9b55a573db870c715bd20f81cfb6a1932a1b52bb409b3d8d78346a5f0ac: Status 404 returned error can't find the container with id f768c9b55a573db870c715bd20f81cfb6a1932a1b52bb409b3d8d78346a5f0ac
Jul 16 22:33:12.818818 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:12.818787    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2190bbd8_a2fb_4522_9254_092c9f78d6bc.slice/crio-8fbea8c41655eeb161a261e772a38c5f6b18aa0203d620667947926f6fae2c74 WatchSource:0}: Error finding container 8fbea8c41655eeb161a261e772a38c5f6b18aa0203d620667947926f6fae2c74: Status 404 returned error can't find the container with id 8fbea8c41655eeb161a261e772a38c5f6b18aa0203d620667947926f6fae2c74
Jul 16 22:33:12.820302 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:12.820221    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a97cb4_0345_49f0_a7ca_7d35a24a59f5.slice/crio-fe9b36c8c2c08543157ec794d280bea1572b53e67e43022f2cb36c9440c6008e WatchSource:0}: Error finding container fe9b36c8c2c08543157ec794d280bea1572b53e67e43022f2cb36c9440c6008e: Status 404 returned error can't find the container with id fe9b36c8c2c08543157ec794d280bea1572b53e67e43022f2cb36c9440c6008e
Jul 16 22:33:12.822861 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:12.822837    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778abed0_00de_488b_84e0_26c1497fa68a.slice/crio-d955c9e71082249c5b843f2f06605b008c3c72e477ae973102807fa7e6beae3c WatchSource:0}: Error finding container d955c9e71082249c5b843f2f06605b008c3c72e477ae973102807fa7e6beae3c: Status 404 returned error can't find the container with id d955c9e71082249c5b843f2f06605b008c3c72e477ae973102807fa7e6beae3c
Jul 16 22:33:12.824033 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:12.824004    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba53725_3faf_4d5c_8641_848a055f65df.slice/crio-a19f318f4b614a25a6cdc0d3cbc01322c524167eaa796f0a0449106e401b1148 WatchSource:0}: Error finding container a19f318f4b614a25a6cdc0d3cbc01322c524167eaa796f0a0449106e401b1148: Status 404 returned error can't find the container with id a19f318f4b614a25a6cdc0d3cbc01322c524167eaa796f0a0449106e401b1148
Jul 16 22:33:12.825300 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:12.825277    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4045c278_1bda_4f6c_9db4_80c4c9c3b202.slice/crio-3d5542257dcf55312cc394018262ce39de8392d4ba44c5f9cdbe322d42875fed WatchSource:0}: Error finding container 3d5542257dcf55312cc394018262ce39de8392d4ba44c5f9cdbe322d42875fed: Status 404 returned error can't find the container with id 3d5542257dcf55312cc394018262ce39de8392d4ba44c5f9cdbe322d42875fed
Jul 16 22:33:12.825967 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:12.825872    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bce50b5_5616_4134_b9ee_f1264bd569dd.slice/crio-ffbce9e2b4ef03c60322be5a5278a4797e5698536b8bbbb1e6bf6c54e25f958d WatchSource:0}: Error finding container ffbce9e2b4ef03c60322be5a5278a4797e5698536b8bbbb1e6bf6c54e25f958d: Status 404 returned error can't find the container with id ffbce9e2b4ef03c60322be5a5278a4797e5698536b8bbbb1e6bf6c54e25f958d
Jul 16 22:33:12.830085 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:12.830062    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b0b632_a518_4519_a7f6_3e18e8cabdb4.slice/crio-2b07cebeddef59bdc1c6c6b95989bf3b8e12d960810ab0191407524bf7c49ca5 WatchSource:0}: Error finding container 2b07cebeddef59bdc1c6c6b95989bf3b8e12d960810ab0191407524bf7c49ca5: Status 404 returned error can't find the container with id 2b07cebeddef59bdc1c6c6b95989bf3b8e12d960810ab0191407524bf7c49ca5
Jul 16 22:33:12.830431 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:12.830407    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f32160_6b80_42d4_b1e6_daf6395f0e13.slice/crio-e0ecd74b6e551ed73fb18b09453b3fd33697e540577169245aebfd60ccf5ab64 WatchSource:0}: Error finding container e0ecd74b6e551ed73fb18b09453b3fd33697e540577169245aebfd60ccf5ab64: Status 404 returned error can't find the container with id e0ecd74b6e551ed73fb18b09453b3fd33697e540577169245aebfd60ccf5ab64
Jul 16 22:33:13.352376 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.351222    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret\") pod \"global-pull-secret-syncer-lgwp5\" (UID: \"61e408a2-709b-4928-b6b2-018a505d8d4d\") " pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:13.352376 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.351541    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs\") pod \"network-metrics-daemon-pbp8p\" (UID: \"d8dfffe2-b127-481e-90c8-d0372f409044\") " pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:13.352376 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:13.351701    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:13.352376 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:13.351764    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs podName:d8dfffe2-b127-481e-90c8-d0372f409044 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:15.351743816 +0000 UTC m=+6.100227869 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs") pod "network-metrics-daemon-pbp8p" (UID: "d8dfffe2-b127-481e-90c8-d0372f409044") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:13.352376 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:13.352193    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:13.352376 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:13.352253    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret podName:61e408a2-709b-4928-b6b2-018a505d8d4d nodeName:}" failed. No retries permitted until 2026-07-16 22:33:15.35223415 +0000 UTC m=+6.100718201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret") pod "global-pull-secret-syncer-lgwp5" (UID: "61e408a2-709b-4928-b6b2-018a505d8d4d") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:13.452680 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.452611    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc49c\" (UniqueName: \"kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c\") pod \"network-check-target-lvsgm\" (UID: \"cca1b284-455e-4795-a955-930aad902e9f\") " pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:13.452864 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:13.452771    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:33:13.452864 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:13.452800    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:33:13.452864 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:13.452814    2565 projected.go:196] Error preparing data for projected volume kube-api-access-xc49c for pod openshift-network-diagnostics/network-check-target-lvsgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:13.453033 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:13.452879    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c podName:cca1b284-455e-4795-a955-930aad902e9f nodeName:}" failed. No retries permitted until 2026-07-16 22:33:15.452858627 +0000 UTC m=+6.201342679 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xc49c" (UniqueName: "kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c") pod "network-check-target-lvsgm" (UID: "cca1b284-455e-4795-a955-930aad902e9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:13.680979 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.680873    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 22:28:12 +0000 UTC" deadline="2027-12-23 23:43:21.695444977 +0000 UTC"
Jul 16 22:33:13.680979 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.680930    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12601h10m8.014518888s"
Jul 16 22:33:13.723255 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.723021    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:13.723255 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:13.723173    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbp8p" podUID="d8dfffe2-b127-481e-90c8-d0372f409044"
Jul 16 22:33:13.724627 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.723801    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:13.724627 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:13.723900    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvsgm" podUID="cca1b284-455e-4795-a955-930aad902e9f"
Jul 16 22:33:13.724627 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.723966    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:13.724627 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:13.724045    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgwp5" podUID="61e408a2-709b-4928-b6b2-018a505d8d4d"
Jul 16 22:33:13.740399 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.740179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b5j8x" event={"ID":"4045c278-1bda-4f6c-9db4-80c4c9c3b202","Type":"ContainerStarted","Data":"3d5542257dcf55312cc394018262ce39de8392d4ba44c5f9cdbe322d42875fed"}
Jul 16 22:33:13.747467 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.747436    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4s28" event={"ID":"778abed0-00de-488b-84e0-26c1497fa68a","Type":"ContainerStarted","Data":"d955c9e71082249c5b843f2f06605b008c3c72e477ae973102807fa7e6beae3c"}
Jul 16 22:33:13.756034 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.756007    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z5q2j" event={"ID":"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5","Type":"ContainerStarted","Data":"fe9b36c8c2c08543157ec794d280bea1572b53e67e43022f2cb36c9440c6008e"}
Jul 16 22:33:13.765219 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.765161    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8c8p5" event={"ID":"2190bbd8-a2fb-4522-9254-092c9f78d6bc","Type":"ContainerStarted","Data":"8fbea8c41655eeb161a261e772a38c5f6b18aa0203d620667947926f6fae2c74"}
Jul 16 22:33:13.768423 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.767786    2565 generic.go:356] "Generic (PLEG): container finished" podID="8cf190d7ed5bcce5f5086d7c54f3dfe0" containerID="e890e6deb81f506a316c7a7dedfee3df8e28f3f2571378e0e841228e4011d182" exitCode=0
Jul 16 22:33:13.768423 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.767852    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal" event={"ID":"8cf190d7ed5bcce5f5086d7c54f3dfe0","Type":"ContainerDied","Data":"e890e6deb81f506a316c7a7dedfee3df8e28f3f2571378e0e841228e4011d182"}
Jul 16 22:33:13.779001 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.778975    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal" event={"ID":"f6e975acdd98c0499b69a0b9bfc9cbb0","Type":"ContainerStarted","Data":"933699b4a19a1f0012c714d4d4fc25e19bd8291bb22ad118f94ac7ddcc99e198"}
Jul 16 22:33:13.792686 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.792615    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gkmrf" event={"ID":"57b0b632-a518-4519-a7f6-3e18e8cabdb4","Type":"ContainerStarted","Data":"2b07cebeddef59bdc1c6c6b95989bf3b8e12d960810ab0191407524bf7c49ca5"}
Jul 16 22:33:13.802645 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.802291    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-23.ec2.internal" podStartSLOduration=2.802277744 podStartE2EDuration="2.802277744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:13.80211418 +0000 UTC m=+4.550598256" watchObservedRunningTime="2026-07-16 22:33:13.802277744 +0000 UTC m=+4.550761818"
Jul 16 22:33:13.817620 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.817593    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r7xmd" event={"ID":"8bce50b5-5616-4134-b9ee-f1264bd569dd","Type":"ContainerStarted","Data":"ffbce9e2b4ef03c60322be5a5278a4797e5698536b8bbbb1e6bf6c54e25f958d"}
Jul 16 22:33:13.821871 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.821843    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cgrb2" event={"ID":"bba53725-3faf-4d5c-8641-848a055f65df","Type":"ContainerStarted","Data":"a19f318f4b614a25a6cdc0d3cbc01322c524167eaa796f0a0449106e401b1148"}
Jul 16 22:33:13.824172 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.824145    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z" event={"ID":"7a448609-0972-4f56-953a-e0231a116488","Type":"ContainerStarted","Data":"f768c9b55a573db870c715bd20f81cfb6a1932a1b52bb409b3d8d78346a5f0ac"}
Jul 16 22:33:13.838031 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:13.837975    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9nmvs" event={"ID":"31f32160-6b80-42d4-b1e6-daf6395f0e13","Type":"ContainerStarted","Data":"e0ecd74b6e551ed73fb18b09453b3fd33697e540577169245aebfd60ccf5ab64"}
Jul 16 22:33:14.858160 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:14.858018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal" event={"ID":"8cf190d7ed5bcce5f5086d7c54f3dfe0","Type":"ContainerStarted","Data":"c14757c7364f82e369ab34e576774f0cc530831ddbfbc641f5fe7f6ce6996d97"}
Jul 16 22:33:15.371635 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:15.371588    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret\") pod \"global-pull-secret-syncer-lgwp5\" (UID: \"61e408a2-709b-4928-b6b2-018a505d8d4d\") " pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:15.371635 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:15.371640    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs\") pod \"network-metrics-daemon-pbp8p\" (UID: \"d8dfffe2-b127-481e-90c8-d0372f409044\") " pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:15.371877 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:15.371764    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:15.371877 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:15.371823    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs podName:d8dfffe2-b127-481e-90c8-d0372f409044 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:19.371804717 +0000 UTC m=+10.120288798 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs") pod "network-metrics-daemon-pbp8p" (UID: "d8dfffe2-b127-481e-90c8-d0372f409044") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:15.371994 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:15.371913    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:15.371994 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:15.371976    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret podName:61e408a2-709b-4928-b6b2-018a505d8d4d nodeName:}" failed. No retries permitted until 2026-07-16 22:33:19.37195941 +0000 UTC m=+10.120443478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret") pod "global-pull-secret-syncer-lgwp5" (UID: "61e408a2-709b-4928-b6b2-018a505d8d4d") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:15.472918 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:15.472876    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc49c\" (UniqueName: \"kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c\") pod \"network-check-target-lvsgm\" (UID: \"cca1b284-455e-4795-a955-930aad902e9f\") " pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:15.473098 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:15.473085    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:33:15.473160 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:15.473108    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:33:15.473160 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:15.473121    2565 projected.go:196] Error preparing data for projected volume kube-api-access-xc49c for pod openshift-network-diagnostics/network-check-target-lvsgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:15.473264 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:15.473179    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c podName:cca1b284-455e-4795-a955-930aad902e9f nodeName:}" failed. No retries permitted until 2026-07-16 22:33:19.473156239 +0000 UTC m=+10.221640310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xc49c" (UniqueName: "kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c") pod "network-check-target-lvsgm" (UID: "cca1b284-455e-4795-a955-930aad902e9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:15.723702 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:15.722825    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:15.723702 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:15.722970    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbp8p" podUID="d8dfffe2-b127-481e-90c8-d0372f409044"
Jul 16 22:33:15.723702 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:15.723373    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:15.723702 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:15.723470    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvsgm" podUID="cca1b284-455e-4795-a955-930aad902e9f"
Jul 16 22:33:15.723702 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:15.723527    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:15.723702 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:15.723603    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgwp5" podUID="61e408a2-709b-4928-b6b2-018a505d8d4d"
Jul 16 22:33:17.721565 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:17.720971    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:17.721565 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:17.721133    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbp8p" podUID="d8dfffe2-b127-481e-90c8-d0372f409044"
Jul 16 22:33:17.721565 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:17.721358    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:17.721565 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:17.721460    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvsgm" podUID="cca1b284-455e-4795-a955-930aad902e9f"
Jul 16 22:33:17.721565 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:17.721492    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:17.721565 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:17.721570    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgwp5" podUID="61e408a2-709b-4928-b6b2-018a505d8d4d"
Jul 16 22:33:19.407341 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:19.407260    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret\") pod \"global-pull-secret-syncer-lgwp5\" (UID: \"61e408a2-709b-4928-b6b2-018a505d8d4d\") " pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:19.407839 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:19.407381    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs\") pod \"network-metrics-daemon-pbp8p\" (UID: \"d8dfffe2-b127-481e-90c8-d0372f409044\") " pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:19.407839 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:19.407587    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:19.407839 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:19.407657    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs podName:d8dfffe2-b127-481e-90c8-d0372f409044 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:27.407635157 +0000 UTC m=+18.156119218 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs") pod "network-metrics-daemon-pbp8p" (UID: "d8dfffe2-b127-481e-90c8-d0372f409044") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:19.408170 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:19.408098    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:19.408170 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:19.408162    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret podName:61e408a2-709b-4928-b6b2-018a505d8d4d nodeName:}" failed. No retries permitted until 2026-07-16 22:33:27.408145262 +0000 UTC m=+18.156629314 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret") pod "global-pull-secret-syncer-lgwp5" (UID: "61e408a2-709b-4928-b6b2-018a505d8d4d") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:19.508350 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:19.508307    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc49c\" (UniqueName: \"kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c\") pod \"network-check-target-lvsgm\" (UID: \"cca1b284-455e-4795-a955-930aad902e9f\") " pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:19.508543 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:19.508459    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:33:19.508543 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:19.508483    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:33:19.508543 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:19.508495    2565 projected.go:196] Error preparing data for projected volume kube-api-access-xc49c for pod openshift-network-diagnostics/network-check-target-lvsgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:19.508543 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:19.508545    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c podName:cca1b284-455e-4795-a955-930aad902e9f nodeName:}" failed. No retries permitted until 2026-07-16 22:33:27.508530604 +0000 UTC m=+18.257014655 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xc49c" (UniqueName: "kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c") pod "network-check-target-lvsgm" (UID: "cca1b284-455e-4795-a955-930aad902e9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:19.721594 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:19.721508    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:19.721779 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:19.721704    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbp8p" podUID="d8dfffe2-b127-481e-90c8-d0372f409044"
Jul 16 22:33:19.722213 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:19.722181    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:19.722969 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:19.722339    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:19.722969 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:19.722402    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvsgm" podUID="cca1b284-455e-4795-a955-930aad902e9f"
Jul 16 22:33:19.722969 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:19.722511    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgwp5" podUID="61e408a2-709b-4928-b6b2-018a505d8d4d"
Jul 16 22:33:21.721275 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:21.720326    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:21.721275 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:21.720326    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:21.721275 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:21.720468    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbp8p" podUID="d8dfffe2-b127-481e-90c8-d0372f409044"
Jul 16 22:33:21.721275 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:21.720557    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgwp5" podUID="61e408a2-709b-4928-b6b2-018a505d8d4d"
Jul 16 22:33:21.721275 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:21.720326    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:21.721275 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:21.720651    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvsgm" podUID="cca1b284-455e-4795-a955-930aad902e9f"
Jul 16 22:33:23.720713 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:23.720634    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:23.720713 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:23.720659    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:23.721219 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:23.720771    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:23.721219 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:23.720770    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbp8p" podUID="d8dfffe2-b127-481e-90c8-d0372f409044"
Jul 16 22:33:23.721219 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:23.720905    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvsgm" podUID="cca1b284-455e-4795-a955-930aad902e9f"
Jul 16 22:33:23.721219 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:23.721030    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgwp5" podUID="61e408a2-709b-4928-b6b2-018a505d8d4d"
Jul 16 22:33:25.720390 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:25.720355    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:25.720860 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:25.720368    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:25.720860 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:25.720490    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbp8p" podUID="d8dfffe2-b127-481e-90c8-d0372f409044"
Jul 16 22:33:25.720860 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:25.720572    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgwp5" podUID="61e408a2-709b-4928-b6b2-018a505d8d4d"
Jul 16 22:33:25.720860 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:25.720366    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:25.720860 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:25.720685    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvsgm" podUID="cca1b284-455e-4795-a955-930aad902e9f"
Jul 16 22:33:27.470984 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:27.470939    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret\") pod \"global-pull-secret-syncer-lgwp5\" (UID: \"61e408a2-709b-4928-b6b2-018a505d8d4d\") " pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:27.471515 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:27.471001    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs\") pod \"network-metrics-daemon-pbp8p\" (UID: \"d8dfffe2-b127-481e-90c8-d0372f409044\") " pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:27.471515 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:27.471095    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:27.471515 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:27.471138    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:27.471515 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:27.471172    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret podName:61e408a2-709b-4928-b6b2-018a505d8d4d nodeName:}" failed. No retries permitted until 2026-07-16 22:33:43.471151839 +0000 UTC m=+34.219635892 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret") pod "global-pull-secret-syncer-lgwp5" (UID: "61e408a2-709b-4928-b6b2-018a505d8d4d") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:27.471515 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:27.471189    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs podName:d8dfffe2-b127-481e-90c8-d0372f409044 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:43.471180994 +0000 UTC m=+34.219665043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs") pod "network-metrics-daemon-pbp8p" (UID: "d8dfffe2-b127-481e-90c8-d0372f409044") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:27.572075 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:27.572047    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc49c\" (UniqueName: \"kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c\") pod \"network-check-target-lvsgm\" (UID: \"cca1b284-455e-4795-a955-930aad902e9f\") " pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:27.572240 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:27.572196    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:33:27.572240 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:27.572211    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:33:27.572240 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:27.572224    2565 projected.go:196] Error preparing data for projected volume kube-api-access-xc49c for pod openshift-network-diagnostics/network-check-target-lvsgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:27.572407 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:27.572294    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c podName:cca1b284-455e-4795-a955-930aad902e9f nodeName:}" failed. No retries permitted until 2026-07-16 22:33:43.572277748 +0000 UTC m=+34.320761797 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xc49c" (UniqueName: "kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c") pod "network-check-target-lvsgm" (UID: "cca1b284-455e-4795-a955-930aad902e9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:27.719931 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:27.719906    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:27.719931 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:27.719923    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:27.720169 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:27.719911    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:27.720169 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:27.720023    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbp8p" podUID="d8dfffe2-b127-481e-90c8-d0372f409044"
Jul 16 22:33:27.720169 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:27.720099    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgwp5" podUID="61e408a2-709b-4928-b6b2-018a505d8d4d"
Jul 16 22:33:27.720169 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:27.720160    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvsgm" podUID="cca1b284-455e-4795-a955-930aad902e9f"
Jul 16 22:33:29.721440 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:29.721400    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:29.721909 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:29.721511    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:29.721909 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:29.721544    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbp8p" podUID="d8dfffe2-b127-481e-90c8-d0372f409044"
Jul 16 22:33:29.721909 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:29.721583    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvsgm" podUID="cca1b284-455e-4795-a955-930aad902e9f"
Jul 16 22:33:29.721909 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:29.721617    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:29.721909 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:29.721686    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgwp5" podUID="61e408a2-709b-4928-b6b2-018a505d8d4d"
Jul 16 22:33:31.720374 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:31.720338    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:31.720837 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:31.720344    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:31.720837 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:31.720476    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgwp5" podUID="61e408a2-709b-4928-b6b2-018a505d8d4d"
Jul 16 22:33:31.720837 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:31.720567    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbp8p" podUID="d8dfffe2-b127-481e-90c8-d0372f409044"
Jul 16 22:33:31.720837 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:31.720352    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:31.720837 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:31.720674    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvsgm" podUID="cca1b284-455e-4795-a955-930aad902e9f"
Jul 16 22:33:32.898000 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.897675    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gkmrf" event={"ID":"57b0b632-a518-4519-a7f6-3e18e8cabdb4","Type":"ContainerStarted","Data":"e471670c6ef32a5fea7712eb3944ebb3f2661abb41aee5a7a142e6904176c80c"}
Jul 16 22:33:32.899406 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.899378    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r7xmd" event={"ID":"8bce50b5-5616-4134-b9ee-f1264bd569dd","Type":"ContainerStarted","Data":"731c25c08666c59c5ff9f4818a187499cd745f5d412afb0ff37cea54b037be13"}
Jul 16 22:33:32.900994 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.900967    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cgrb2" event={"ID":"bba53725-3faf-4d5c-8641-848a055f65df","Type":"ContainerStarted","Data":"82b3fca27eb988e54fb56ac9d34428e26963cfc94b6442ae96f1ce82f35021d8"}
Jul 16 22:33:32.902540 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.902520    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z" event={"ID":"7a448609-0972-4f56-953a-e0231a116488","Type":"ContainerStarted","Data":"30b2abb14d477afd664287f0e902fadc021558219e1a6e6a611cd6594d5b0616"}
Jul 16 22:33:32.904070 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.904046    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9nmvs" event={"ID":"31f32160-6b80-42d4-b1e6-daf6395f0e13","Type":"ContainerStarted","Data":"5aa8b3e3d8736b209e37cedc3c8f59cd85e2bdd7bea18348683711eb82fe7989"}
Jul 16 22:33:32.907362 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.907342    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4s28" event={"ID":"778abed0-00de-488b-84e0-26c1497fa68a","Type":"ContainerStarted","Data":"a450232334ba66e30e49a5916987519120ee294c230b159adeb01d12130bcef2"}
Jul 16 22:33:32.907457 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.907398    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4s28" event={"ID":"778abed0-00de-488b-84e0-26c1497fa68a","Type":"ContainerStarted","Data":"f4c6f26d2272eeb261cb8ba2e88fe133e64ff3027e6e17df3135c02fddf2f4ef"}
Jul 16 22:33:32.907457 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.907441    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4s28" event={"ID":"778abed0-00de-488b-84e0-26c1497fa68a","Type":"ContainerStarted","Data":"292689ab6ca145cb10daf00bf5a0d4a2b58051f3bd23117a1a106fd031b5290f"}
Jul 16 22:33:32.907566 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.907496    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4s28" event={"ID":"778abed0-00de-488b-84e0-26c1497fa68a","Type":"ContainerStarted","Data":"cc67a118cc13f98bdf05976da7633e05b4b07158fb266c347f25b28765ff2fc3"}
Jul 16 22:33:32.907566 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.907545    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4s28" event={"ID":"778abed0-00de-488b-84e0-26c1497fa68a","Type":"ContainerStarted","Data":"aedeafd6627fc1ecfe626f45aef47ddf289f2ebb86955720b3cc669cad77d6af"}
Jul 16 22:33:32.907668 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.907590    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4s28" event={"ID":"778abed0-00de-488b-84e0-26c1497fa68a","Type":"ContainerStarted","Data":"21fea16b85363b32a14b89016e7d2762d2d68768e8488a6a3afa2ebac3e0dc23"}
Jul 16 22:33:32.908976 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.908953    2565 generic.go:356] "Generic (PLEG): container finished" podID="d0a97cb4-0345-49f0-a7ca-7d35a24a59f5" containerID="e4d664e65033f866d74c5eff478a9b05c7a41fba1ca1c541def723221e9fb869" exitCode=0
Jul 16 22:33:32.909078 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.909029    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z5q2j" event={"ID":"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5","Type":"ContainerDied","Data":"e4d664e65033f866d74c5eff478a9b05c7a41fba1ca1c541def723221e9fb869"}
Jul 16 22:33:32.910728 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.910692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8c8p5" event={"ID":"2190bbd8-a2fb-4522-9254-092c9f78d6bc","Type":"ContainerStarted","Data":"082c32e376555fae9d6e6b4186a0ca2028c007e72fcfa9b5f831b90c51e35114"}
Jul 16 22:33:32.912824 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.912785    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-23.ec2.internal" podStartSLOduration=21.912772591 podStartE2EDuration="21.912772591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:14.888375953 +0000 UTC m=+5.636860031" watchObservedRunningTime="2026-07-16 22:33:32.912772591 +0000 UTC m=+23.661256662"
Jul 16 22:33:32.913174 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.913147    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gkmrf" podStartSLOduration=7.965676139 podStartE2EDuration="22.913137237s" totalImagesPullingTime="14.947461098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:32.912466923 +0000 UTC m=+23.660950995" watchObservedRunningTime="2026-07-16 22:33:32.913137237 +0000 UTC m=+23.661621310"
Jul 16 22:33:32.945803 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.945762    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r7xmd" podStartSLOduration=3.730829566 podStartE2EDuration="22.945750672s" totalImagesPullingTime="19.214921106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:32.945504798 +0000 UTC m=+23.693988871" watchObservedRunningTime="2026-07-16 22:33:32.945750672 +0000 UTC m=+23.694234742"
Jul 16 22:33:32.963603 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.963564    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8c8p5" podStartSLOduration=3.690553479 podStartE2EDuration="22.963553044s" totalImagesPullingTime="19.272999565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:32.963425002 +0000 UTC m=+23.711909071" watchObservedRunningTime="2026-07-16 22:33:32.963553044 +0000 UTC m=+23.712037114"
Jul 16 22:33:32.981135 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.981099    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cgrb2" podStartSLOduration=3.763165703 podStartE2EDuration="22.981089009s" totalImagesPullingTime="19.217923306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:32.980817532 +0000 UTC m=+23.729301605" watchObservedRunningTime="2026-07-16 22:33:32.981089009 +0000 UTC m=+23.729573079"
Jul 16 22:33:32.999335 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:32.999287    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9nmvs" podStartSLOduration=3.786817928 podStartE2EDuration="22.999271547s" totalImagesPullingTime="19.212453619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:32.998653867 +0000 UTC m=+23.747137938" watchObservedRunningTime="2026-07-16 22:33:32.999271547 +0000 UTC m=+23.747755617"
Jul 16 22:33:33.029364 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:33.029346    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gkmrf"
Jul 16 22:33:33.029854 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:33.029841    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gkmrf"
Jul 16 22:33:33.707899 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:33.707876    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 22:33:33.720682 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:33.720661    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:33.720682 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:33.720672    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:33.720843 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:33.720672    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:33.720843 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:33.720766    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvsgm" podUID="cca1b284-455e-4795-a955-930aad902e9f"
Jul 16 22:33:33.720906 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:33.720861    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbp8p" podUID="d8dfffe2-b127-481e-90c8-d0372f409044"
Jul 16 22:33:33.720947 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:33.720932    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgwp5" podUID="61e408a2-709b-4928-b6b2-018a505d8d4d"
Jul 16 22:33:33.914843 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:33.914768    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z" event={"ID":"7a448609-0972-4f56-953a-e0231a116488","Type":"ContainerStarted","Data":"5b2cc027972dd7b94a021f06fffbab602542f7b9c48477855a24fabaa64822d3"}
Jul 16 22:33:33.916606 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:33.916576    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b5j8x" event={"ID":"4045c278-1bda-4f6c-9db4-80c4c9c3b202","Type":"ContainerStarted","Data":"543792f28f24e00ed6747d223e455e884bdcde3c0c43a1304ee8e93aa6c87916"}
Jul 16 22:33:33.917298 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:33.917277    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gkmrf"
Jul 16 22:33:33.917644 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:33.917625    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gkmrf"
Jul 16 22:33:33.930596 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:33.930558    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b5j8x" podStartSLOduration=4.714088727 podStartE2EDuration="23.930545302s" totalImagesPullingTime="19.216456575s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:33.930072627 +0000 UTC m=+24.678556699" watchObservedRunningTime="2026-07-16 22:33:33.930545302 +0000 UTC m=+24.679029372"
Jul 16 22:33:34.687638 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:34.687542    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T22:33:33.707894626Z","UUID":"3fafb8d2-1d0e-4417-ba7f-635aa44d71b5","Handler":null,"Name":"","Endpoint":""}
Jul 16 22:33:34.688915 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:34.688887    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 22:33:34.689033 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:34.688923    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 22:33:35.721128 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:35.720867    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:35.721798 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:35.720867    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:35.721798 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:35.721191    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbp8p" podUID="d8dfffe2-b127-481e-90c8-d0372f409044"
Jul 16 22:33:35.721798 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:35.721230    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvsgm" podUID="cca1b284-455e-4795-a955-930aad902e9f"
Jul 16 22:33:35.721798 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:35.720871    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:35.721798 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:35.721330    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgwp5" podUID="61e408a2-709b-4928-b6b2-018a505d8d4d"
Jul 16 22:33:35.927442 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:35.927407    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z" event={"ID":"7a448609-0972-4f56-953a-e0231a116488","Type":"ContainerStarted","Data":"0c788c3fd325866fbfead0d658e260056158832b282c629540e9c6e23950c4bd"}
Jul 16 22:33:35.933727 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:35.933694    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4s28" event={"ID":"778abed0-00de-488b-84e0-26c1497fa68a","Type":"ContainerStarted","Data":"3f8f781341431a8cf57de2978ce1528421eedde12e08fd96b14af0037346b3cf"}
Jul 16 22:33:35.948416 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:35.948373    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttr5z" podStartSLOduration=3.813587118 podStartE2EDuration="25.948360033s" totalImagesPullingTime="22.134772915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:35.948031306 +0000 UTC m=+26.696515389" watchObservedRunningTime="2026-07-16 22:33:35.948360033 +0000 UTC m=+26.696844140"
Jul 16 22:33:37.720384 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:37.720347    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:37.720384 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:37.720377    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:37.720880 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:37.720482    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgwp5" podUID="61e408a2-709b-4928-b6b2-018a505d8d4d"
Jul 16 22:33:37.720880 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:37.720598    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbp8p" podUID="d8dfffe2-b127-481e-90c8-d0372f409044"
Jul 16 22:33:37.720880 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:37.720693    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:37.720880 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:37.720784    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvsgm" podUID="cca1b284-455e-4795-a955-930aad902e9f"
Jul 16 22:33:38.944519 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:38.944304    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4s28" event={"ID":"778abed0-00de-488b-84e0-26c1497fa68a","Type":"ContainerStarted","Data":"59a2e235e053af11e2d06d15feec0d2a5cb172482f349c221a7386883ee3a59b"}
Jul 16 22:33:38.945396 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:38.944584    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:38.945396 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:38.944638    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:38.946229 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:38.946200    2565 generic.go:356] "Generic (PLEG): container finished" podID="d0a97cb4-0345-49f0-a7ca-7d35a24a59f5" containerID="3dd2b186b237206662236f4074877f8668e880d3fbc18b5c558c23b123d5b5fd" exitCode=0
Jul 16 22:33:38.946351 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:38.946269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z5q2j" event={"ID":"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5","Type":"ContainerDied","Data":"3dd2b186b237206662236f4074877f8668e880d3fbc18b5c558c23b123d5b5fd"}
Jul 16 22:33:38.960167 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:38.960147    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:38.979590 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:38.979553    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s4s28" podStartSLOduration=9.69012507 podStartE2EDuration="28.97954001s" totalImagesPullingTime="19.28941494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:38.978564811 +0000 UTC m=+29.727048872" watchObservedRunningTime="2026-07-16 22:33:38.97954001 +0000 UTC m=+29.728024081"
Jul 16 22:33:39.721421 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:39.721394    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:39.721790 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:39.721549    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:39.721884 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:39.721584    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:39.722270 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:39.722129    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvsgm" podUID="cca1b284-455e-4795-a955-930aad902e9f"
Jul 16 22:33:39.722270 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:39.722215    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgwp5" podUID="61e408a2-709b-4928-b6b2-018a505d8d4d"
Jul 16 22:33:39.722467 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:39.722039    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbp8p" podUID="d8dfffe2-b127-481e-90c8-d0372f409044"
Jul 16 22:33:39.948766 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:39.948725    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:39.963752 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:39.963730    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:33:40.308505 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:40.308478    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lgwp5"]
Jul 16 22:33:40.308634 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:40.308572    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:40.308700 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:40.308652    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgwp5" podUID="61e408a2-709b-4928-b6b2-018a505d8d4d"
Jul 16 22:33:40.310810 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:40.310788    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lvsgm"]
Jul 16 22:33:40.310893 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:40.310874    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:40.310986 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:40.310967    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvsgm" podUID="cca1b284-455e-4795-a955-930aad902e9f"
Jul 16 22:33:40.314593 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:40.314572    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pbp8p"]
Jul 16 22:33:40.314683 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:40.314662    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:40.314805 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:40.314767    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbp8p" podUID="d8dfffe2-b127-481e-90c8-d0372f409044"
Jul 16 22:33:40.951594 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:40.951562    2565 generic.go:356] "Generic (PLEG): container finished" podID="d0a97cb4-0345-49f0-a7ca-7d35a24a59f5" containerID="a8fba960170dff509541b237e428e99ece6d821bb7abf69c4e6b3543d49199ef" exitCode=0
Jul 16 22:33:40.952001 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:40.951640    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z5q2j" event={"ID":"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5","Type":"ContainerDied","Data":"a8fba960170dff509541b237e428e99ece6d821bb7abf69c4e6b3543d49199ef"}
Jul 16 22:33:41.720634 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:41.720560    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:41.720634 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:41.720582    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:41.720808 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:41.720694    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbp8p" podUID="d8dfffe2-b127-481e-90c8-d0372f409044"
Jul 16 22:33:41.720808 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:41.720745    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:41.720963 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:41.720831    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvsgm" podUID="cca1b284-455e-4795-a955-930aad902e9f"
Jul 16 22:33:41.720963 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:41.720919    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgwp5" podUID="61e408a2-709b-4928-b6b2-018a505d8d4d"
Jul 16 22:33:42.958722 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:42.958544    2565 generic.go:356] "Generic (PLEG): container finished" podID="d0a97cb4-0345-49f0-a7ca-7d35a24a59f5" containerID="268bcfe6a39258284353291094f3abf71925fde878a1c83ac9e44b4d7b0ccff2" exitCode=0
Jul 16 22:33:42.958722 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:42.958620    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z5q2j" event={"ID":"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5","Type":"ContainerDied","Data":"268bcfe6a39258284353291094f3abf71925fde878a1c83ac9e44b4d7b0ccff2"}
Jul 16 22:33:43.495898 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:43.495857    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret\") pod \"global-pull-secret-syncer-lgwp5\" (UID: \"61e408a2-709b-4928-b6b2-018a505d8d4d\") " pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:43.496081 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:43.495916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs\") pod \"network-metrics-daemon-pbp8p\" (UID: \"d8dfffe2-b127-481e-90c8-d0372f409044\") " pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:43.496081 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:43.496022    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:43.496081 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:43.496048    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:43.496206 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:43.496090    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret podName:61e408a2-709b-4928-b6b2-018a505d8d4d nodeName:}" failed. No retries permitted until 2026-07-16 22:34:15.496071023 +0000 UTC m=+66.244555077 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret") pod "global-pull-secret-syncer-lgwp5" (UID: "61e408a2-709b-4928-b6b2-018a505d8d4d") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:33:43.496206 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:43.496111    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs podName:d8dfffe2-b127-481e-90c8-d0372f409044 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:15.496100741 +0000 UTC m=+66.244584796 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs") pod "network-metrics-daemon-pbp8p" (UID: "d8dfffe2-b127-481e-90c8-d0372f409044") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:33:43.597071 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:43.597037    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc49c\" (UniqueName: \"kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c\") pod \"network-check-target-lvsgm\" (UID: \"cca1b284-455e-4795-a955-930aad902e9f\") " pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:43.597225 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:43.597184    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:33:43.597225 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:43.597209    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:33:43.597225 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:43.597220    2565 projected.go:196] Error preparing data for projected volume kube-api-access-xc49c for pod openshift-network-diagnostics/network-check-target-lvsgm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:43.597389 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:43.597293    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c podName:cca1b284-455e-4795-a955-930aad902e9f nodeName:}" failed. No retries permitted until 2026-07-16 22:34:15.597276232 +0000 UTC m=+66.345760303 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xc49c" (UniqueName: "kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c") pod "network-check-target-lvsgm" (UID: "cca1b284-455e-4795-a955-930aad902e9f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:33:43.720831 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:43.720749    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:43.720986 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:43.720851    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lvsgm" podUID="cca1b284-455e-4795-a955-930aad902e9f"
Jul 16 22:33:43.720986 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:43.720749    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:43.720986 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:43.720917    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbp8p" podUID="d8dfffe2-b127-481e-90c8-d0372f409044"
Jul 16 22:33:43.720986 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:43.720868    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:43.720986 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:43.720972    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgwp5" podUID="61e408a2-709b-4928-b6b2-018a505d8d4d"
Jul 16 22:33:45.074021 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.073996    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-23.ec2.internal" event="NodeReady"
Jul 16 22:33:45.074579 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.074157    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 22:33:45.113718 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.113683    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b99bcf697-55fzz"]
Jul 16 22:33:45.117114 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.117095    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.120399 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.120371    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 16 22:33:45.120502 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.120370    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 16 22:33:45.120502 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.120432    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 16 22:33:45.120918 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.120896    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9sg5r\""
Jul 16 22:33:45.128286 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.128232    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 16 22:33:45.132586 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.132560    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b99bcf697-55fzz"]
Jul 16 22:33:45.137370 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.137349    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bnnqb"]
Jul 16 22:33:45.141050 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.141032    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-frld8"]
Jul 16 22:33:45.141199 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.141177    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:33:45.144090 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.144072    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-frld8"
Jul 16 22:33:45.144678 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.144627    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 22:33:45.144753 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.144688    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 22:33:45.144816 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.144786    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-57zhl\""
Jul 16 22:33:45.146002 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.145984    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-m9krj\""
Jul 16 22:33:45.146153 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.146132    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 22:33:45.146644 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.146625    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 22:33:45.146802 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.146768    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 22:33:45.158845 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.158823    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bnnqb"]
Jul 16 22:33:45.159467 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.159446    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-frld8"]
Jul 16 22:33:45.207862 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.207834    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-certificates\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.207988 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.207894    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-bound-sa-token\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.207988 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.207920    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41bf030b-380e-4614-a98a-a471cc1fd0e8-ca-trust-extracted\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.207988 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.207941    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41bf030b-380e-4614-a98a-a471cc1fd0e8-installation-pull-secrets\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.207988 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.207964    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkrw5\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-kube-api-access-jkrw5\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.208183 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.208022    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41bf030b-380e-4614-a98a-a471cc1fd0e8-image-registry-private-configuration\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.208183 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.208044    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.208183 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.208069    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41bf030b-380e-4614-a98a-a471cc1fd0e8-trusted-ca\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.308908 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.308879    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6mrm\" (UniqueName: \"kubernetes.io/projected/2e1f153e-c668-492c-99df-6e50a746cbfb-kube-api-access-r6mrm\") pod \"ingress-canary-frld8\" (UID: \"2e1f153e-c668-492c-99df-6e50a746cbfb\") " pod="openshift-ingress-canary/ingress-canary-frld8"
Jul 16 22:33:45.309081 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.308918    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:33:45.309081 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.308946    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-bound-sa-token\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.309081 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.308964    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41bf030b-380e-4614-a98a-a471cc1fd0e8-ca-trust-extracted\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.309081 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.309017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41bf030b-380e-4614-a98a-a471cc1fd0e8-installation-pull-secrets\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.309294 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.309077    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkrw5\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-kube-api-access-jkrw5\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.309294 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.309265    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr64d\" (UniqueName: \"kubernetes.io/projected/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-kube-api-access-qr64d\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:33:45.309400 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.309319    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-config-volume\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:33:45.309400 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.309382    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41bf030b-380e-4614-a98a-a471cc1fd0e8-image-registry-private-configuration\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.309500 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.309411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.309500 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.309444    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41bf030b-380e-4614-a98a-a471cc1fd0e8-trusted-ca\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.309500 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.309467    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e1f153e-c668-492c-99df-6e50a746cbfb-cert\") pod \"ingress-canary-frld8\" (UID: \"2e1f153e-c668-492c-99df-6e50a746cbfb\") " pod="openshift-ingress-canary/ingress-canary-frld8"
Jul 16 22:33:45.309500 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.309488    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-tmp-dir\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:33:45.309701 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.309558    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41bf030b-380e-4614-a98a-a471cc1fd0e8-ca-trust-extracted\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.309701 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:45.309568    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 22:33:45.309701 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:45.309583    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b99bcf697-55fzz: secret "image-registry-tls" not found
Jul 16 22:33:45.309701 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.309684    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-certificates\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.309897 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:45.309838    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls podName:41bf030b-380e-4614-a98a-a471cc1fd0e8 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:45.809814424 +0000 UTC m=+36.558298475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls") pod "image-registry-6b99bcf697-55fzz" (UID: "41bf030b-380e-4614-a98a-a471cc1fd0e8") : secret "image-registry-tls" not found
Jul 16 22:33:45.310227 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.310204    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-certificates\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.310460 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.310437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41bf030b-380e-4614-a98a-a471cc1fd0e8-trusted-ca\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.313892 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.313868    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41bf030b-380e-4614-a98a-a471cc1fd0e8-installation-pull-secrets\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.313987 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.313874    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41bf030b-380e-4614-a98a-a471cc1fd0e8-image-registry-private-configuration\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.318835 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.318811    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-bound-sa-token\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.319225 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.319200    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkrw5\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-kube-api-access-jkrw5\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.410228 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.410152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:33:45.410228 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.410202    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr64d\" (UniqueName: \"kubernetes.io/projected/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-kube-api-access-qr64d\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:33:45.410228 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.410231    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-config-volume\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:33:45.410495 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.410285    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e1f153e-c668-492c-99df-6e50a746cbfb-cert\") pod \"ingress-canary-frld8\" (UID: \"2e1f153e-c668-492c-99df-6e50a746cbfb\") " pod="openshift-ingress-canary/ingress-canary-frld8"
Jul 16 22:33:45.410495 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.410303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-tmp-dir\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:33:45.410495 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:45.410319    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 22:33:45.410495 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.410339    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6mrm\" (UniqueName: \"kubernetes.io/projected/2e1f153e-c668-492c-99df-6e50a746cbfb-kube-api-access-r6mrm\") pod \"ingress-canary-frld8\" (UID: \"2e1f153e-c668-492c-99df-6e50a746cbfb\") " pod="openshift-ingress-canary/ingress-canary-frld8"
Jul 16 22:33:45.410495 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:45.410400    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls podName:25c597a0-b7b8-4ba5-96d5-e6155af0a3fd nodeName:}" failed. No retries permitted until 2026-07-16 22:33:45.910380493 +0000 UTC m=+36.658864557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls") pod "dns-default-bnnqb" (UID: "25c597a0-b7b8-4ba5-96d5-e6155af0a3fd") : secret "dns-default-metrics-tls" not found
Jul 16 22:33:45.410692 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.410627    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-tmp-dir\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:33:45.410858 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.410838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-config-volume\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:33:45.412949 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.412925    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e1f153e-c668-492c-99df-6e50a746cbfb-cert\") pod \"ingress-canary-frld8\" (UID: \"2e1f153e-c668-492c-99df-6e50a746cbfb\") " pod="openshift-ingress-canary/ingress-canary-frld8"
Jul 16 22:33:45.421663 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.421641    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6mrm\" (UniqueName: \"kubernetes.io/projected/2e1f153e-c668-492c-99df-6e50a746cbfb-kube-api-access-r6mrm\") pod \"ingress-canary-frld8\" (UID: \"2e1f153e-c668-492c-99df-6e50a746cbfb\") " pod="openshift-ingress-canary/ingress-canary-frld8"
Jul 16 22:33:45.421750 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.421677    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr64d\" (UniqueName: \"kubernetes.io/projected/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-kube-api-access-qr64d\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:33:45.460979 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.460954    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-frld8"
Jul 16 22:33:45.617655 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.617453    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-frld8"]
Jul 16 22:33:45.622747 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:33:45.622717    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1f153e_c668_492c_99df_6e50a746cbfb.slice/crio-bbc6694ed33f6e291889cc4fc541746bf0bed0ca38c4a2defa1fdefccde5d1ed WatchSource:0}: Error finding container bbc6694ed33f6e291889cc4fc541746bf0bed0ca38c4a2defa1fdefccde5d1ed: Status 404 returned error can't find the container with id bbc6694ed33f6e291889cc4fc541746bf0bed0ca38c4a2defa1fdefccde5d1ed
Jul 16 22:33:45.720414 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.720322    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:33:45.720414 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.720322    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:33:45.720648 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.720619    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:33:45.723186 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.722895    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 22:33:45.723186 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.723039    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 22:33:45.723186 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.723057    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 22:33:45.723186 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.723074    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jdr86\""
Jul 16 22:33:45.723186 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.723059    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9dnx4\""
Jul 16 22:33:45.723186 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.723079    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 22:33:45.813297 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.813244    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:45.813460 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:45.813389    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 22:33:45.813460 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:45.813407    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b99bcf697-55fzz: secret "image-registry-tls" not found
Jul 16 22:33:45.813560 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:45.813477    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls podName:41bf030b-380e-4614-a98a-a471cc1fd0e8 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:46.813457216 +0000 UTC m=+37.561941281 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls") pod "image-registry-6b99bcf697-55fzz" (UID: "41bf030b-380e-4614-a98a-a471cc1fd0e8") : secret "image-registry-tls" not found
Jul 16 22:33:45.914040 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.914003    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:33:45.914204 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:45.914144    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 22:33:45.914297 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:45.914224    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls podName:25c597a0-b7b8-4ba5-96d5-e6155af0a3fd nodeName:}" failed. No retries permitted until 2026-07-16 22:33:46.914199307 +0000 UTC m=+37.662683360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls") pod "dns-default-bnnqb" (UID: "25c597a0-b7b8-4ba5-96d5-e6155af0a3fd") : secret "dns-default-metrics-tls" not found
Jul 16 22:33:45.966548 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:45.966507    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-frld8" event={"ID":"2e1f153e-c668-492c-99df-6e50a746cbfb","Type":"ContainerStarted","Data":"bbc6694ed33f6e291889cc4fc541746bf0bed0ca38c4a2defa1fdefccde5d1ed"}
Jul 16 22:33:46.813650 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:46.813579    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkmrf_57b0b632-a518-4519-a7f6-3e18e8cabdb4/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:46.823491 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:46.823459    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:46.823646 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:46.823626    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 22:33:46.823697 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:46.823649    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b99bcf697-55fzz: secret "image-registry-tls" not found
Jul 16 22:33:46.823738 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:46.823703    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls podName:41bf030b-380e-4614-a98a-a471cc1fd0e8 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:48.82369013 +0000 UTC m=+39.572174179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls") pod "image-registry-6b99bcf697-55fzz" (UID: "41bf030b-380e-4614-a98a-a471cc1fd0e8") : secret "image-registry-tls" not found
Jul 16 22:33:46.923682 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:46.923650    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkmrf_57b0b632-a518-4519-a7f6-3e18e8cabdb4/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:46.923935 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:46.923911    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:33:46.924062 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:46.924046    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 22:33:46.924120 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:46.924107    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls podName:25c597a0-b7b8-4ba5-96d5-e6155af0a3fd nodeName:}" failed. No retries permitted until 2026-07-16 22:33:48.924090088 +0000 UTC m=+39.672574138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls") pod "dns-default-bnnqb" (UID: "25c597a0-b7b8-4ba5-96d5-e6155af0a3fd") : secret "dns-default-metrics-tls" not found
Jul 16 22:33:47.068864 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:47.068776    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkmrf_57b0b632-a518-4519-a7f6-3e18e8cabdb4/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:47.240128 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:47.240095    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkmrf_57b0b632-a518-4519-a7f6-3e18e8cabdb4/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:47.363943 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:47.363866    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkmrf_57b0b632-a518-4519-a7f6-3e18e8cabdb4/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:47.717423 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:47.717339    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkmrf_57b0b632-a518-4519-a7f6-3e18e8cabdb4/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:47.780955 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:47.780917    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkmrf_57b0b632-a518-4519-a7f6-3e18e8cabdb4/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:47.976534 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:47.976450    2565 generic.go:356] "Generic (PLEG): container finished" podID="d0a97cb4-0345-49f0-a7ca-7d35a24a59f5" containerID="1ce4a034a49d4ac1b896e1ebe4fa60a51aa2929e829de2f4066c501278aa20c6" exitCode=0
Jul 16 22:33:47.976534 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:47.976516    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z5q2j" event={"ID":"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5","Type":"ContainerDied","Data":"1ce4a034a49d4ac1b896e1ebe4fa60a51aa2929e829de2f4066c501278aa20c6"}
Jul 16 22:33:48.124564 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:48.124531    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkmrf_57b0b632-a518-4519-a7f6-3e18e8cabdb4/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:48.221585 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:48.221554    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkmrf_57b0b632-a518-4519-a7f6-3e18e8cabdb4/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:48.839755 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:48.839711    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:48.839942 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:48.839866    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 22:33:48.839942 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:48.839885    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b99bcf697-55fzz: secret "image-registry-tls" not found
Jul 16 22:33:48.839942 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:48.839941    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls podName:41bf030b-380e-4614-a98a-a471cc1fd0e8 nodeName:}" failed. No retries permitted until 2026-07-16 22:33:52.839925819 +0000 UTC m=+43.588409868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls") pod "image-registry-6b99bcf697-55fzz" (UID: "41bf030b-380e-4614-a98a-a471cc1fd0e8") : secret "image-registry-tls" not found
Jul 16 22:33:48.940375 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:48.940329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:33:48.940563 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:48.940513    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 22:33:48.940618 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:48.940590    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls podName:25c597a0-b7b8-4ba5-96d5-e6155af0a3fd nodeName:}" failed. No retries permitted until 2026-07-16 22:33:52.94057097 +0000 UTC m=+43.689055045 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls") pod "dns-default-bnnqb" (UID: "25c597a0-b7b8-4ba5-96d5-e6155af0a3fd") : secret "dns-default-metrics-tls" not found
Jul 16 22:33:48.982623 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:48.982587    2565 generic.go:356] "Generic (PLEG): container finished" podID="d0a97cb4-0345-49f0-a7ca-7d35a24a59f5" containerID="cb01a4dc4d47671cdb1bce909158e4c485a184668def201629e513bfa01f5ea6" exitCode=0
Jul 16 22:33:48.983066 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:48.982638    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z5q2j" event={"ID":"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5","Type":"ContainerDied","Data":"cb01a4dc4d47671cdb1bce909158e4c485a184668def201629e513bfa01f5ea6"}
Jul 16 22:33:49.489461 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:49.489228    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkmrf_57b0b632-a518-4519-a7f6-3e18e8cabdb4/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:49.590579 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:49.590500    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkmrf_57b0b632-a518-4519-a7f6-3e18e8cabdb4/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:49.986219 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:49.986179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-frld8" event={"ID":"2e1f153e-c668-492c-99df-6e50a746cbfb","Type":"ContainerStarted","Data":"8653a04fe28d264f042fd0cf6435502627a9f88e4aeb00b8eac8d67888752a29"}
Jul 16 22:33:49.989285 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:49.989239    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z5q2j" event={"ID":"d0a97cb4-0345-49f0-a7ca-7d35a24a59f5","Type":"ContainerStarted","Data":"f67fe5cff83c738066e187fe28220104ba11e2cf03ab57b0b52cd7f0d22a5635"}
Jul 16 22:33:50.009115 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:50.009069    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-frld8" podStartSLOduration=1.210166036 podStartE2EDuration="5.009053488s" totalImagesPullingTime="3.798887452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:50.008107404 +0000 UTC m=+40.756591475" watchObservedRunningTime="2026-07-16 22:33:50.009053488 +0000 UTC m=+40.757537559"
Jul 16 22:33:50.038465 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:50.038413    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-z5q2j" podStartSLOduration=9.123735481 podStartE2EDuration="40.038399435s" totalImagesPullingTime="30.914663954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:50.036364733 +0000 UTC m=+40.784848804" watchObservedRunningTime="2026-07-16 22:33:50.038399435 +0000 UTC m=+40.786883536"
Jul 16 22:33:50.178472 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:50.178384    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkmrf_57b0b632-a518-4519-a7f6-3e18e8cabdb4/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:50.279711 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:50.279676    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkmrf_57b0b632-a518-4519-a7f6-3e18e8cabdb4/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:52.870348 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:52.870311    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:33:52.870803 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:52.870495    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 22:33:52.870803 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:52.870523    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b99bcf697-55fzz: secret "image-registry-tls" not found
Jul 16 22:33:52.870803 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:52.870595    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls podName:41bf030b-380e-4614-a98a-a471cc1fd0e8 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:00.870572652 +0000 UTC m=+51.619056705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls") pod "image-registry-6b99bcf697-55fzz" (UID: "41bf030b-380e-4614-a98a-a471cc1fd0e8") : secret "image-registry-tls" not found
Jul 16 22:33:52.971277 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:52.971226    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:33:52.971439 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:52.971374    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 22:33:52.971439 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:33:52.971434    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls podName:25c597a0-b7b8-4ba5-96d5-e6155af0a3fd nodeName:}" failed. No retries permitted until 2026-07-16 22:34:00.971419258 +0000 UTC m=+51.719903306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls") pod "dns-default-bnnqb" (UID: "25c597a0-b7b8-4ba5-96d5-e6155af0a3fd") : secret "dns-default-metrics-tls" not found
Jul 16 22:33:56.520038 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:56.520006    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkmrf_57b0b632-a518-4519-a7f6-3e18e8cabdb4/konnectivity-agent/0.log" limit=1024
Jul 16 22:33:56.620149 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:33:56.620113    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkmrf_57b0b632-a518-4519-a7f6-3e18e8cabdb4/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:00.922168 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:00.922133    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:34:00.922569 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:00.922299    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 22:34:00.922569 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:00.922318    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b99bcf697-55fzz: secret "image-registry-tls" not found
Jul 16 22:34:00.922569 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:00.922379    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls podName:41bf030b-380e-4614-a98a-a471cc1fd0e8 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:16.922364699 +0000 UTC m=+67.670848752 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls") pod "image-registry-6b99bcf697-55fzz" (UID: "41bf030b-380e-4614-a98a-a471cc1fd0e8") : secret "image-registry-tls" not found
Jul 16 22:34:01.023400 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:01.023367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:34:01.023559 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:01.023507    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 22:34:01.023559 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:01.023556    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls podName:25c597a0-b7b8-4ba5-96d5-e6155af0a3fd nodeName:}" failed. No retries permitted until 2026-07-16 22:34:17.023542758 +0000 UTC m=+67.772026810 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls") pod "dns-default-bnnqb" (UID: "25c597a0-b7b8-4ba5-96d5-e6155af0a3fd") : secret "dns-default-metrics-tls" not found
Jul 16 22:34:02.004768 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:02.004729    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkmrf_57b0b632-a518-4519-a7f6-3e18e8cabdb4/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:02.071014 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:02.070984    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gkmrf_57b0b632-a518-4519-a7f6-3e18e8cabdb4/konnectivity-agent/0.log" limit=1024
Jul 16 22:34:11.972881 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:11.972851    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s4s28"
Jul 16 22:34:15.524005 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:15.523959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret\") pod \"global-pull-secret-syncer-lgwp5\" (UID: \"61e408a2-709b-4928-b6b2-018a505d8d4d\") " pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:34:15.524426 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:15.524017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs\") pod \"network-metrics-daemon-pbp8p\" (UID: \"d8dfffe2-b127-481e-90c8-d0372f409044\") " pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:34:15.526789 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:15.526774    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 22:34:15.526831 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:15.526794    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 22:34:15.534786 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:15.534765    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 16 22:34:15.534859 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:15.534848    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs podName:d8dfffe2-b127-481e-90c8-d0372f409044 nodeName:}" failed. No retries permitted until 2026-07-16 22:35:19.534831275 +0000 UTC m=+130.283315334 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs") pod "network-metrics-daemon-pbp8p" (UID: "d8dfffe2-b127-481e-90c8-d0372f409044") : secret "metrics-daemon-secret" not found
Jul 16 22:34:15.538440 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:15.538411    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/61e408a2-709b-4928-b6b2-018a505d8d4d-original-pull-secret\") pod \"global-pull-secret-syncer-lgwp5\" (UID: \"61e408a2-709b-4928-b6b2-018a505d8d4d\") " pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:34:15.625079 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:15.625042    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc49c\" (UniqueName: \"kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c\") pod \"network-check-target-lvsgm\" (UID: \"cca1b284-455e-4795-a955-930aad902e9f\") " pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:34:15.628166 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:15.628137    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 22:34:15.638658 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:15.638635    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 22:34:15.648533 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:15.648507    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc49c\" (UniqueName: \"kubernetes.io/projected/cca1b284-455e-4795-a955-930aad902e9f-kube-api-access-xc49c\") pod \"network-check-target-lvsgm\" (UID: \"cca1b284-455e-4795-a955-930aad902e9f\") " pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:34:15.735467 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:15.735428    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jdr86\""
Jul 16 22:34:15.743622 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:15.743602    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:34:15.744767 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:15.744751    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgwp5"
Jul 16 22:34:15.876764 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:15.876664    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lvsgm"]
Jul 16 22:34:15.879285 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:34:15.879239    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca1b284_455e_4795_a955_930aad902e9f.slice/crio-1dcf6e9d28f3485b73f88ed2079e516531b8706a6185ab3b38c638e39d3d2527 WatchSource:0}: Error finding container 1dcf6e9d28f3485b73f88ed2079e516531b8706a6185ab3b38c638e39d3d2527: Status 404 returned error can't find the container with id 1dcf6e9d28f3485b73f88ed2079e516531b8706a6185ab3b38c638e39d3d2527
Jul 16 22:34:15.893698 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:15.893673    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lgwp5"]
Jul 16 22:34:15.897229 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:34:15.897207    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e408a2_709b_4928_b6b2_018a505d8d4d.slice/crio-4ef18ed19a3db65eb62ec10febc7bbfee3729bc6b8fbe7303226c57995a76850 WatchSource:0}: Error finding container 4ef18ed19a3db65eb62ec10febc7bbfee3729bc6b8fbe7303226c57995a76850: Status 404 returned error can't find the container with id 4ef18ed19a3db65eb62ec10febc7bbfee3729bc6b8fbe7303226c57995a76850
Jul 16 22:34:16.046379 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:16.046286    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lvsgm" event={"ID":"cca1b284-455e-4795-a955-930aad902e9f","Type":"ContainerStarted","Data":"1dcf6e9d28f3485b73f88ed2079e516531b8706a6185ab3b38c638e39d3d2527"}
Jul 16 22:34:16.047299 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:16.047269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lgwp5" event={"ID":"61e408a2-709b-4928-b6b2-018a505d8d4d","Type":"ContainerStarted","Data":"4ef18ed19a3db65eb62ec10febc7bbfee3729bc6b8fbe7303226c57995a76850"}
Jul 16 22:34:16.934058 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:16.934020    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:34:16.934480 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:16.934200    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 22:34:16.934480 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:16.934224    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b99bcf697-55fzz: secret "image-registry-tls" not found
Jul 16 22:34:16.934480 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:16.934312    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls podName:41bf030b-380e-4614-a98a-a471cc1fd0e8 nodeName:}" failed. No retries permitted until 2026-07-16 22:34:48.934292509 +0000 UTC m=+99.682776560 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls") pod "image-registry-6b99bcf697-55fzz" (UID: "41bf030b-380e-4614-a98a-a471cc1fd0e8") : secret "image-registry-tls" not found
Jul 16 22:34:17.034926 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:17.034885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:34:17.035122 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:17.035064    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 22:34:17.035182 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:17.035133    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls podName:25c597a0-b7b8-4ba5-96d5-e6155af0a3fd nodeName:}" failed. No retries permitted until 2026-07-16 22:34:49.035114612 +0000 UTC m=+99.783598675 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls") pod "dns-default-bnnqb" (UID: "25c597a0-b7b8-4ba5-96d5-e6155af0a3fd") : secret "dns-default-metrics-tls" not found
Jul 16 22:34:22.065204 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:22.065153    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lgwp5" event={"ID":"61e408a2-709b-4928-b6b2-018a505d8d4d","Type":"ContainerStarted","Data":"50610f0078366bbb2a063cee39f6acfd87a1209864cd5f1627bf562be4b3fd82"}
Jul 16 22:34:22.066677 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:22.066650    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lvsgm" event={"ID":"cca1b284-455e-4795-a955-930aad902e9f","Type":"ContainerStarted","Data":"36737920737beee9eef0496ec102af372baebecedaeef36e7617b8576b25a618"}
Jul 16 22:34:22.066810 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:22.066798    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:34:22.081280 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:22.081209    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lgwp5" podStartSLOduration=65.925419203 podStartE2EDuration="1m11.081190817s" totalImagesPullingTime="5.155771614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:22.080745207 +0000 UTC m=+72.829229283" watchObservedRunningTime="2026-07-16 22:34:22.081190817 +0000 UTC m=+72.829674908"
Jul 16 22:34:22.098859 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:22.096279    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lvsgm" podStartSLOduration=65.93587529 podStartE2EDuration="1m11.096237443s" totalImagesPullingTime="5.160362153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:22.095098883 +0000 UTC m=+72.843582955" watchObservedRunningTime="2026-07-16 22:34:22.096237443 +0000 UTC m=+72.844721515"
Jul 16 22:34:42.994988 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:42.994869    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-cc9pp"]
Jul 16 22:34:42.998980 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:42.998961    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-cc9pp"
Jul 16 22:34:43.001056 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.001037    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-sht7s\""
Jul 16 22:34:43.007570 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.007547    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"]
Jul 16 22:34:43.010358 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.010341    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-cc9pp"]
Jul 16 22:34:43.010444 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.010426    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"
Jul 16 22:34:43.012789 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.012763    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6fbfl\""
Jul 16 22:34:43.013051 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.013033    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 22:34:43.013116 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.013044    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 22:34:43.013116 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.013056    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 16 22:34:43.013376 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.013359    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 16 22:34:43.022675 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.022656    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"]
Jul 16 22:34:43.113834 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.113795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-44bjw\" (UID: \"156c60da-5d5c-4d2e-9d91-05d6e4a31cba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"
Jul 16 22:34:43.113834 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.113843    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ngsx\" (UniqueName: \"kubernetes.io/projected/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-kube-api-access-6ngsx\") pod \"cluster-monitoring-operator-76cf8966f8-44bjw\" (UID: \"156c60da-5d5c-4d2e-9d91-05d6e4a31cba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"
Jul 16 22:34:43.114061 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.113865    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kv45\" (UniqueName: \"kubernetes.io/projected/6126f770-c080-4d63-bc09-86841dffec97-kube-api-access-9kv45\") pod \"network-check-source-7bc5b6bf85-cc9pp\" (UID: \"6126f770-c080-4d63-bc09-86841dffec97\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-cc9pp"
Jul 16 22:34:43.114061 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.113922    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-44bjw\" (UID: \"156c60da-5d5c-4d2e-9d91-05d6e4a31cba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"
Jul 16 22:34:43.215349 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.215299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-44bjw\" (UID: \"156c60da-5d5c-4d2e-9d91-05d6e4a31cba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"
Jul 16 22:34:43.215349 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.215358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ngsx\" (UniqueName: \"kubernetes.io/projected/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-kube-api-access-6ngsx\") pod \"cluster-monitoring-operator-76cf8966f8-44bjw\" (UID: \"156c60da-5d5c-4d2e-9d91-05d6e4a31cba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"
Jul 16 22:34:43.215349 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.215378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kv45\" (UniqueName: \"kubernetes.io/projected/6126f770-c080-4d63-bc09-86841dffec97-kube-api-access-9kv45\") pod \"network-check-source-7bc5b6bf85-cc9pp\" (UID: \"6126f770-c080-4d63-bc09-86841dffec97\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-cc9pp"
Jul 16 22:34:43.215610 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.215452    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-44bjw\" (UID: \"156c60da-5d5c-4d2e-9d91-05d6e4a31cba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"
Jul 16 22:34:43.215610 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:43.215590    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 22:34:43.215676 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:43.215662    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-cluster-monitoring-operator-tls podName:156c60da-5d5c-4d2e-9d91-05d6e4a31cba nodeName:}" failed. No retries permitted until 2026-07-16 22:34:43.715642846 +0000 UTC m=+94.464126897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-44bjw" (UID: "156c60da-5d5c-4d2e-9d91-05d6e4a31cba") : secret "cluster-monitoring-operator-tls" not found
Jul 16 22:34:43.216267 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.216223    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-44bjw\" (UID: \"156c60da-5d5c-4d2e-9d91-05d6e4a31cba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"
Jul 16 22:34:43.224256 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.224204    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kv45\" (UniqueName: \"kubernetes.io/projected/6126f770-c080-4d63-bc09-86841dffec97-kube-api-access-9kv45\") pod \"network-check-source-7bc5b6bf85-cc9pp\" (UID: \"6126f770-c080-4d63-bc09-86841dffec97\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-cc9pp"
Jul 16 22:34:43.224375 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.224281    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ngsx\" (UniqueName: \"kubernetes.io/projected/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-kube-api-access-6ngsx\") pod \"cluster-monitoring-operator-76cf8966f8-44bjw\" (UID: \"156c60da-5d5c-4d2e-9d91-05d6e4a31cba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"
Jul 16 22:34:43.306899 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.306792    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-cc9pp"
Jul 16 22:34:43.423749 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.423716    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-cc9pp"]
Jul 16 22:34:43.427274 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:34:43.427226    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6126f770_c080_4d63_bc09_86841dffec97.slice/crio-d8ae0f8549aa58319cafb19e055323d5a39e25c39b79ede6e1da894518c09f13 WatchSource:0}: Error finding container d8ae0f8549aa58319cafb19e055323d5a39e25c39b79ede6e1da894518c09f13: Status 404 returned error can't find the container with id d8ae0f8549aa58319cafb19e055323d5a39e25c39b79ede6e1da894518c09f13
Jul 16 22:34:43.719366 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:43.719329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-44bjw\" (UID: \"156c60da-5d5c-4d2e-9d91-05d6e4a31cba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"
Jul 16 22:34:43.719549 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:43.719480    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 22:34:43.719549 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:43.719545    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-cluster-monitoring-operator-tls podName:156c60da-5d5c-4d2e-9d91-05d6e4a31cba nodeName:}" failed. No retries permitted until 2026-07-16 22:34:44.719528985 +0000 UTC m=+95.468013039 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-44bjw" (UID: "156c60da-5d5c-4d2e-9d91-05d6e4a31cba") : secret "cluster-monitoring-operator-tls" not found
Jul 16 22:34:44.125898 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:44.125867    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-cc9pp" event={"ID":"6126f770-c080-4d63-bc09-86841dffec97","Type":"ContainerStarted","Data":"ae13d9946b9ab23bfd330ac44fd788353059dd71807badc4b21357d96d97d094"}
Jul 16 22:34:44.126294 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:44.125939    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-cc9pp" event={"ID":"6126f770-c080-4d63-bc09-86841dffec97","Type":"ContainerStarted","Data":"d8ae0f8549aa58319cafb19e055323d5a39e25c39b79ede6e1da894518c09f13"}
Jul 16 22:34:44.143725 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:44.143680    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-cc9pp" podStartSLOduration=2.143665752 podStartE2EDuration="2.143665752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:44.142742125 +0000 UTC m=+94.891226199" watchObservedRunningTime="2026-07-16 22:34:44.143665752 +0000 UTC m=+94.892149822"
Jul 16 22:34:44.728798 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:44.728761    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-44bjw\" (UID: \"156c60da-5d5c-4d2e-9d91-05d6e4a31cba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"
Jul 16 22:34:44.728991 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:44.728921    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 22:34:44.729052 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:44.728994    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-cluster-monitoring-operator-tls podName:156c60da-5d5c-4d2e-9d91-05d6e4a31cba nodeName:}" failed. No retries permitted until 2026-07-16 22:34:46.728977774 +0000 UTC m=+97.477461826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-44bjw" (UID: "156c60da-5d5c-4d2e-9d91-05d6e4a31cba") : secret "cluster-monitoring-operator-tls" not found
Jul 16 22:34:46.740671 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:46.740610    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-44bjw\" (UID: \"156c60da-5d5c-4d2e-9d91-05d6e4a31cba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"
Jul 16 22:34:46.741190 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:46.740794    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 22:34:46.741190 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:46.740885    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-cluster-monitoring-operator-tls podName:156c60da-5d5c-4d2e-9d91-05d6e4a31cba nodeName:}" failed. No retries permitted until 2026-07-16 22:34:50.740862518 +0000 UTC m=+101.489346581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-44bjw" (UID: "156c60da-5d5c-4d2e-9d91-05d6e4a31cba") : secret "cluster-monitoring-operator-tls" not found
Jul 16 22:34:47.670430 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:47.670392    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc"]
Jul 16 22:34:47.673410 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:47.673391    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc"
Jul 16 22:34:47.678295 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:47.678270    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 16 22:34:47.678917 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:47.678883    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 16 22:34:47.679020 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:47.678974    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 16 22:34:47.679020 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:47.678981    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 16 22:34:47.691800 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:47.691775    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc"]
Jul 16 22:34:47.747969 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:47.747937    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a-klusterlet-config\") pod \"klusterlet-addon-workmgr-c677d8f4d-zd5sc\" (UID: \"f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc"
Jul 16 22:34:47.748340 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:47.748028    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppt9m\" (UniqueName: \"kubernetes.io/projected/f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a-kube-api-access-ppt9m\") pod \"klusterlet-addon-workmgr-c677d8f4d-zd5sc\" (UID: \"f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc"
Jul 16 22:34:47.748340 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:47.748086    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a-tmp\") pod \"klusterlet-addon-workmgr-c677d8f4d-zd5sc\" (UID: \"f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc"
Jul 16 22:34:47.849141 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:47.849108    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a-tmp\") pod \"klusterlet-addon-workmgr-c677d8f4d-zd5sc\" (UID: \"f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc"
Jul 16 22:34:47.849329 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:47.849165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a-klusterlet-config\") pod \"klusterlet-addon-workmgr-c677d8f4d-zd5sc\" (UID: \"f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc"
Jul 16 22:34:47.849329 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:47.849209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppt9m\" (UniqueName: \"kubernetes.io/projected/f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a-kube-api-access-ppt9m\") pod \"klusterlet-addon-workmgr-c677d8f4d-zd5sc\" (UID: \"f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc"
Jul 16 22:34:47.850054 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:47.850027    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a-tmp\") pod \"klusterlet-addon-workmgr-c677d8f4d-zd5sc\" (UID: \"f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc"
Jul 16 22:34:47.851767 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:47.851742    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a-klusterlet-config\") pod \"klusterlet-addon-workmgr-c677d8f4d-zd5sc\" (UID: \"f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc"
Jul 16 22:34:47.858705 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:47.858683    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppt9m\" (UniqueName: \"kubernetes.io/projected/f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a-kube-api-access-ppt9m\") pod \"klusterlet-addon-workmgr-c677d8f4d-zd5sc\" (UID: \"f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc"
Jul 16 22:34:47.982593 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:47.982503    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc"
Jul 16 22:34:48.106420 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:48.106390    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc"]
Jul 16 22:34:48.109730 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:34:48.109706    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a2c0e7_f8d6_4c22_b5c9_95e29f0ba30a.slice/crio-ce5ec670d979360cc0f9c7a88f4c11160c2d6a36a4ea347ff913c1ad3e43cf9d WatchSource:0}: Error finding container ce5ec670d979360cc0f9c7a88f4c11160c2d6a36a4ea347ff913c1ad3e43cf9d: Status 404 returned error can't find the container with id ce5ec670d979360cc0f9c7a88f4c11160c2d6a36a4ea347ff913c1ad3e43cf9d
Jul 16 22:34:48.136578 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:48.136549    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc" event={"ID":"f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a","Type":"ContainerStarted","Data":"ce5ec670d979360cc0f9c7a88f4c11160c2d6a36a4ea347ff913c1ad3e43cf9d"}
Jul 16 22:34:48.956792 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:48.956755    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls\") pod \"image-registry-6b99bcf697-55fzz\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") " pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:34:48.957332 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:48.956928    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 22:34:48.957332 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:48.956947    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b99bcf697-55fzz: secret "image-registry-tls" not found
Jul 16 22:34:48.957332 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:48.957015    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls podName:41bf030b-380e-4614-a98a-a471cc1fd0e8 nodeName:}" failed. No retries permitted until 2026-07-16 22:35:52.956994565 +0000 UTC m=+163.705478636 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls") pod "image-registry-6b99bcf697-55fzz" (UID: "41bf030b-380e-4614-a98a-a471cc1fd0e8") : secret "image-registry-tls" not found
Jul 16 22:34:49.058111 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:49.058078    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:34:49.058317 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:49.058263    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 22:34:49.058416 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:49.058341    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls podName:25c597a0-b7b8-4ba5-96d5-e6155af0a3fd nodeName:}" failed. No retries permitted until 2026-07-16 22:35:53.058320931 +0000 UTC m=+163.806804989 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls") pod "dns-default-bnnqb" (UID: "25c597a0-b7b8-4ba5-96d5-e6155af0a3fd") : secret "dns-default-metrics-tls" not found
Jul 16 22:34:50.775294 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:50.775236    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-44bjw\" (UID: \"156c60da-5d5c-4d2e-9d91-05d6e4a31cba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"
Jul 16 22:34:50.775750 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:50.775385    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 22:34:50.775750 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:34:50.775473    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-cluster-monitoring-operator-tls podName:156c60da-5d5c-4d2e-9d91-05d6e4a31cba nodeName:}" failed. No retries permitted until 2026-07-16 22:34:58.775450324 +0000 UTC m=+109.523934376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-44bjw" (UID: "156c60da-5d5c-4d2e-9d91-05d6e4a31cba") : secret "cluster-monitoring-operator-tls" not found
Jul 16 22:34:53.072105 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:53.072073    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lvsgm"
Jul 16 22:34:53.088819 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:53.088795    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cgrb2_bba53725-3faf-4d5c-8641-848a055f65df/dns-node-resolver/0.log"
Jul 16 22:34:53.151735 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:53.151707    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc" event={"ID":"f4a2c0e7-f8d6-4c22-b5c9-95e29f0ba30a","Type":"ContainerStarted","Data":"ab27f9b1493e3e4d2b2845c0922038e95429bbcca04106ab85dd8dcbaad18b58"}
Jul 16 22:34:53.151954 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:53.151935    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc"
Jul 16 22:34:53.153438 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:53.153416    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc"
Jul 16 22:34:53.169572 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:53.169531    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c677d8f4d-zd5sc" podStartSLOduration=1.9021092259999999 podStartE2EDuration="6.169515209s" totalImagesPullingTime="4.267405983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:53.168081779 +0000 UTC m=+103.916565849" watchObservedRunningTime="2026-07-16 22:34:53.169515209 +0000 UTC m=+103.917999279"
Jul 16 22:34:54.089926 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:54.089895    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r7xmd_8bce50b5-5616-4134-b9ee-f1264bd569dd/node-ca/0.log"
Jul 16 22:34:54.887915 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:54.887882    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-frld8_2e1f153e-c668-492c-99df-6e50a746cbfb/serve-healthcheck-canary/0.log"
Jul 16 22:34:58.838565 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:58.838534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-44bjw\" (UID: \"156c60da-5d5c-4d2e-9d91-05d6e4a31cba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"
Jul 16 22:34:58.840919 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:58.840900    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/156c60da-5d5c-4d2e-9d91-05d6e4a31cba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-44bjw\" (UID: \"156c60da-5d5c-4d2e-9d91-05d6e4a31cba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"
Jul 16 22:34:58.918955 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:58.918920    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"
Jul 16 22:34:59.037918 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:59.037885    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw"]
Jul 16 22:34:59.041550 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:34:59.041524    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156c60da_5d5c_4d2e_9d91_05d6e4a31cba.slice/crio-1739d114a784d4a11b53ece12aaaf7be13b778e1904b4d32e75e7e2d870254d7 WatchSource:0}: Error finding container 1739d114a784d4a11b53ece12aaaf7be13b778e1904b4d32e75e7e2d870254d7: Status 404 returned error can't find the container with id 1739d114a784d4a11b53ece12aaaf7be13b778e1904b4d32e75e7e2d870254d7
Jul 16 22:34:59.167644 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:34:59.167555    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw" event={"ID":"156c60da-5d5c-4d2e-9d91-05d6e4a31cba","Type":"ContainerStarted","Data":"1739d114a784d4a11b53ece12aaaf7be13b778e1904b4d32e75e7e2d870254d7"}
Jul 16 22:35:01.175051 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:01.175015    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw" event={"ID":"156c60da-5d5c-4d2e-9d91-05d6e4a31cba","Type":"ContainerStarted","Data":"514ac354eca917e9d947cc3e3aeddd8369e3b29a3b2f54980b6274a59047918d"}
Jul 16 22:35:01.194231 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:01.194179    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-44bjw" podStartSLOduration=17.59270964 podStartE2EDuration="19.194163653s" totalImagesPullingTime="1.601454013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:01.193856002 +0000 UTC m=+111.942340096" watchObservedRunningTime="2026-07-16 22:35:01.194163653 +0000 UTC m=+111.942647726"
Jul 16 22:35:11.065835 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.065801    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dlbxx"]
Jul 16 22:35:11.068760 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.068741    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dlbxx"
Jul 16 22:35:11.072809 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.072787    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 22:35:11.072932 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.072787    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 22:35:11.073455 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.073408    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 22:35:11.073540 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.073459    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lklv7\""
Jul 16 22:35:11.073540 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.073440    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 22:35:11.077530 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.077510    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r4kck"]
Jul 16 22:35:11.079897 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.079883    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r4kck"
Jul 16 22:35:11.085623 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.085603    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9gz74\""
Jul 16 22:35:11.085849 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.085832    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 16 22:35:11.094159 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.094136    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dlbxx"]
Jul 16 22:35:11.098282 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.098243    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r4kck"]
Jul 16 22:35:11.132865 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.132842    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pplph"]
Jul 16 22:35:11.135549 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.135532    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pplph"
Jul 16 22:35:11.137802 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.137785    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 16 22:35:11.137894 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.137818    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 16 22:35:11.138098 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.138083    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-x2gj6\""
Jul 16 22:35:11.160666 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.160648    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pplph"]
Jul 16 22:35:11.228522 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.228497    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86e283ba-586c-45f8-886e-80a3f0a56aa6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlbxx\" (UID: \"86e283ba-586c-45f8-886e-80a3f0a56aa6\") " pod="openshift-insights/insights-runtime-extractor-dlbxx"
Jul 16 22:35:11.228624 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.228530    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86e283ba-586c-45f8-886e-80a3f0a56aa6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlbxx\" (UID: \"86e283ba-586c-45f8-886e-80a3f0a56aa6\") " pod="openshift-insights/insights-runtime-extractor-dlbxx"
Jul 16 22:35:11.228624 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.228553    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z7bg\" (UniqueName: \"kubernetes.io/projected/86e283ba-586c-45f8-886e-80a3f0a56aa6-kube-api-access-4z7bg\") pod \"insights-runtime-extractor-dlbxx\" (UID: \"86e283ba-586c-45f8-886e-80a3f0a56aa6\") " pod="openshift-insights/insights-runtime-extractor-dlbxx"
Jul 16 22:35:11.228692 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.228618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86e283ba-586c-45f8-886e-80a3f0a56aa6-crio-socket\") pod \"insights-runtime-extractor-dlbxx\" (UID: \"86e283ba-586c-45f8-886e-80a3f0a56aa6\") " pod="openshift-insights/insights-runtime-extractor-dlbxx"
Jul 16 22:35:11.228692 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.228683    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86e283ba-586c-45f8-886e-80a3f0a56aa6-data-volume\") pod \"insights-runtime-extractor-dlbxx\" (UID: \"86e283ba-586c-45f8-886e-80a3f0a56aa6\") " pod="openshift-insights/insights-runtime-extractor-dlbxx"
Jul 16 22:35:11.228756 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.228731    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v24dg\" (UniqueName: \"kubernetes.io/projected/76d9a002-345d-4324-a645-71675213c52d-kube-api-access-v24dg\") pod \"migrator-7cc9c7ffd7-pplph\" (UID: \"76d9a002-345d-4324-a645-71675213c52d\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pplph"
Jul 16 22:35:11.228788 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.228767    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a17fd63f-8198-4623-a983-6577ad6983f4-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-r4kck\" (UID: \"a17fd63f-8198-4623-a983-6577ad6983f4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r4kck"
Jul 16 22:35:11.329948 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.329922    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86e283ba-586c-45f8-886e-80a3f0a56aa6-data-volume\") pod \"insights-runtime-extractor-dlbxx\" (UID: \"86e283ba-586c-45f8-886e-80a3f0a56aa6\") " pod="openshift-insights/insights-runtime-extractor-dlbxx"
Jul 16 22:35:11.330091 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.329973    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v24dg\" (UniqueName: \"kubernetes.io/projected/76d9a002-345d-4324-a645-71675213c52d-kube-api-access-v24dg\") pod \"migrator-7cc9c7ffd7-pplph\" (UID: \"76d9a002-345d-4324-a645-71675213c52d\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pplph"
Jul 16 22:35:11.330091 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.330001    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a17fd63f-8198-4623-a983-6577ad6983f4-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-r4kck\" (UID: \"a17fd63f-8198-4623-a983-6577ad6983f4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r4kck"
Jul 16 22:35:11.330091 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.330030    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86e283ba-586c-45f8-886e-80a3f0a56aa6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlbxx\" (UID: \"86e283ba-586c-45f8-886e-80a3f0a56aa6\") " pod="openshift-insights/insights-runtime-extractor-dlbxx"
Jul 16 22:35:11.330091 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.330049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86e283ba-586c-45f8-886e-80a3f0a56aa6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlbxx\" (UID: \"86e283ba-586c-45f8-886e-80a3f0a56aa6\") " pod="openshift-insights/insights-runtime-extractor-dlbxx"
Jul 16 22:35:11.330091 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.330066    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z7bg\" (UniqueName: \"kubernetes.io/projected/86e283ba-586c-45f8-886e-80a3f0a56aa6-kube-api-access-4z7bg\") pod \"insights-runtime-extractor-dlbxx\" (UID: \"86e283ba-586c-45f8-886e-80a3f0a56aa6\") " pod="openshift-insights/insights-runtime-extractor-dlbxx"
Jul 16 22:35:11.330374 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.330106    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86e283ba-586c-45f8-886e-80a3f0a56aa6-crio-socket\") pod \"insights-runtime-extractor-dlbxx\" (UID: \"86e283ba-586c-45f8-886e-80a3f0a56aa6\") " pod="openshift-insights/insights-runtime-extractor-dlbxx"
Jul 16 22:35:11.330374 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.330184    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86e283ba-586c-45f8-886e-80a3f0a56aa6-crio-socket\") pod \"insights-runtime-extractor-dlbxx\" (UID: \"86e283ba-586c-45f8-886e-80a3f0a56aa6\") " pod="openshift-insights/insights-runtime-extractor-dlbxx"
Jul 16 22:35:11.330374 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.330355    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86e283ba-586c-45f8-886e-80a3f0a56aa6-data-volume\") pod \"insights-runtime-extractor-dlbxx\" (UID: \"86e283ba-586c-45f8-886e-80a3f0a56aa6\") " pod="openshift-insights/insights-runtime-extractor-dlbxx"
Jul 16 22:35:11.330628 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.330598    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86e283ba-586c-45f8-886e-80a3f0a56aa6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dlbxx\" (UID: \"86e283ba-586c-45f8-886e-80a3f0a56aa6\") " pod="openshift-insights/insights-runtime-extractor-dlbxx"
Jul 16 22:35:11.332476 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.332446    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86e283ba-586c-45f8-886e-80a3f0a56aa6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dlbxx\" (UID: \"86e283ba-586c-45f8-886e-80a3f0a56aa6\") " pod="openshift-insights/insights-runtime-extractor-dlbxx"
Jul 16 22:35:11.332565 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.332533    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a17fd63f-8198-4623-a983-6577ad6983f4-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-r4kck\" (UID: \"a17fd63f-8198-4623-a983-6577ad6983f4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r4kck"
Jul 16 22:35:11.338094 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:35:11.338075    2565 projected.go:291] Couldn't get configMap openshift-kube-storage-version-migrator/openshift-service-ca.crt: configmap "openshift-service-ca.crt" not found
Jul 16 22:35:11.338094 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:35:11.338094    2565 projected.go:196] Error preparing data for projected volume kube-api-access-v24dg for pod openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pplph: configmap "openshift-service-ca.crt" not found
Jul 16 22:35:11.338259 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:35:11.338157    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76d9a002-345d-4324-a645-71675213c52d-kube-api-access-v24dg podName:76d9a002-345d-4324-a645-71675213c52d nodeName:}" failed. No retries permitted until 2026-07-16 22:35:11.838142566 +0000 UTC m=+122.586626615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v24dg" (UniqueName: "kubernetes.io/projected/76d9a002-345d-4324-a645-71675213c52d-kube-api-access-v24dg") pod "migrator-7cc9c7ffd7-pplph" (UID: "76d9a002-345d-4324-a645-71675213c52d") : configmap "openshift-service-ca.crt" not found
Jul 16 22:35:11.372912 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.372888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z7bg\" (UniqueName: \"kubernetes.io/projected/86e283ba-586c-45f8-886e-80a3f0a56aa6-kube-api-access-4z7bg\") pod \"insights-runtime-extractor-dlbxx\" (UID: \"86e283ba-586c-45f8-886e-80a3f0a56aa6\") " pod="openshift-insights/insights-runtime-extractor-dlbxx"
Jul 16 22:35:11.377790 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.377771    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dlbxx"
Jul 16 22:35:11.387687 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.387666    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r4kck"
Jul 16 22:35:11.560088 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.560060    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dlbxx"]
Jul 16 22:35:11.563067 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:35:11.563041    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e283ba_586c_45f8_886e_80a3f0a56aa6.slice/crio-d8ec598f664849013562fa88b77a040fd1ba1233946610ed3770d0f839f80bae WatchSource:0}: Error finding container d8ec598f664849013562fa88b77a040fd1ba1233946610ed3770d0f839f80bae: Status 404 returned error can't find the container with id d8ec598f664849013562fa88b77a040fd1ba1233946610ed3770d0f839f80bae
Jul 16 22:35:11.571710 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:35:11.571686    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17fd63f_8198_4623_a983_6577ad6983f4.slice/crio-1582d8b7de36dfcb3812cebf1351dca4af26ca725a282cc29ce34a834d4bed25 WatchSource:0}: Error finding container 1582d8b7de36dfcb3812cebf1351dca4af26ca725a282cc29ce34a834d4bed25: Status 404 returned error can't find the container with id 1582d8b7de36dfcb3812cebf1351dca4af26ca725a282cc29ce34a834d4bed25
Jul 16 22:35:11.574078 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.574058    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r4kck"]
Jul 16 22:35:11.935212 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.935179    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v24dg\" (UniqueName: \"kubernetes.io/projected/76d9a002-345d-4324-a645-71675213c52d-kube-api-access-v24dg\") pod \"migrator-7cc9c7ffd7-pplph\" (UID: \"76d9a002-345d-4324-a645-71675213c52d\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pplph"
Jul 16 22:35:11.937442 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:11.937424    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v24dg\" (UniqueName: \"kubernetes.io/projected/76d9a002-345d-4324-a645-71675213c52d-kube-api-access-v24dg\") pod \"migrator-7cc9c7ffd7-pplph\" (UID: \"76d9a002-345d-4324-a645-71675213c52d\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pplph"
Jul 16 22:35:12.043650 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:12.043611    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pplph"
Jul 16 22:35:12.204936 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:12.204755    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pplph"]
Jul 16 22:35:12.208799 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:35:12.208766    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d9a002_345d_4324_a645_71675213c52d.slice/crio-57b5ba38929101e930dd77eda36c724ca2a7e91eb06537372ad1dd390b0f5edb WatchSource:0}: Error finding container 57b5ba38929101e930dd77eda36c724ca2a7e91eb06537372ad1dd390b0f5edb: Status 404 returned error can't find the container with id 57b5ba38929101e930dd77eda36c724ca2a7e91eb06537372ad1dd390b0f5edb
Jul 16 22:35:12.210586 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:12.210510    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dlbxx" event={"ID":"86e283ba-586c-45f8-886e-80a3f0a56aa6","Type":"ContainerStarted","Data":"d8ec598f664849013562fa88b77a040fd1ba1233946610ed3770d0f839f80bae"}
Jul 16 22:35:12.214098 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:12.214044    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r4kck" event={"ID":"a17fd63f-8198-4623-a983-6577ad6983f4","Type":"ContainerStarted","Data":"1582d8b7de36dfcb3812cebf1351dca4af26ca725a282cc29ce34a834d4bed25"}
Jul 16 22:35:13.219243 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:13.219207    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pplph" event={"ID":"76d9a002-345d-4324-a645-71675213c52d","Type":"ContainerStarted","Data":"57b5ba38929101e930dd77eda36c724ca2a7e91eb06537372ad1dd390b0f5edb"}
Jul 16 22:35:14.224596 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:14.224561    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r4kck" event={"ID":"a17fd63f-8198-4623-a983-6577ad6983f4","Type":"ContainerStarted","Data":"d872b19b028ff5ca4137a0c6b4a1208dbe104a636d56a418b9dd55fa0fe94684"}
Jul 16 22:35:14.225063 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:14.224757    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r4kck"
Jul 16 22:35:14.230814 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:14.230789    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r4kck"
Jul 16 22:35:14.260855 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:14.260791    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r4kck" podStartSLOduration=1.3847679529999999 podStartE2EDuration="3.260773996s" totalImagesPullingTime="1.876006043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:14.259357672 +0000 UTC m=+125.007841744" watchObservedRunningTime="2026-07-16 22:35:14.260773996 +0000 UTC m=+125.009258071"
Jul 16 22:35:15.219711 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.219682    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv"]
Jul 16 22:35:15.240346 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.240318    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv"
Jul 16 22:35:15.245819 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.245790    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv"]
Jul 16 22:35:15.246799 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.246765    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 16 22:35:15.246927 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.246887    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 16 22:35:15.246999 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.246986    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 22:35:15.246999 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.246987    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rcc4f\""
Jul 16 22:35:15.367599 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.367542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb078957-0631-4ff9-af8a-78ab0f668f4d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-9dmjv\" (UID: \"fb078957-0631-4ff9-af8a-78ab0f668f4d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv"
Jul 16 22:35:15.367759 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.367631    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb078957-0631-4ff9-af8a-78ab0f668f4d-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-9dmjv\" (UID: \"fb078957-0631-4ff9-af8a-78ab0f668f4d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv"
Jul 16 22:35:15.367759 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.367719    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5hn9\" (UniqueName: \"kubernetes.io/projected/fb078957-0631-4ff9-af8a-78ab0f668f4d-kube-api-access-j5hn9\") pod \"prometheus-operator-5945b5cfd-9dmjv\" (UID: \"fb078957-0631-4ff9-af8a-78ab0f668f4d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv"
Jul 16 22:35:15.367883 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.367795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb078957-0631-4ff9-af8a-78ab0f668f4d-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-9dmjv\" (UID: \"fb078957-0631-4ff9-af8a-78ab0f668f4d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv"
Jul 16 22:35:15.468971 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.468930    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb078957-0631-4ff9-af8a-78ab0f668f4d-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-9dmjv\" (UID: \"fb078957-0631-4ff9-af8a-78ab0f668f4d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv"
Jul 16 22:35:15.469134 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.469043    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb078957-0631-4ff9-af8a-78ab0f668f4d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-9dmjv\" (UID: \"fb078957-0631-4ff9-af8a-78ab0f668f4d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv"
Jul 16 22:35:15.469134 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.469081    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb078957-0631-4ff9-af8a-78ab0f668f4d-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-9dmjv\" (UID: \"fb078957-0631-4ff9-af8a-78ab0f668f4d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv"
Jul 16 22:35:15.469134 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.469110    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5hn9\" (UniqueName: \"kubernetes.io/projected/fb078957-0631-4ff9-af8a-78ab0f668f4d-kube-api-access-j5hn9\") pod \"prometheus-operator-5945b5cfd-9dmjv\" (UID: \"fb078957-0631-4ff9-af8a-78ab0f668f4d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv"
Jul 16 22:35:15.470588 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.470529    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb078957-0631-4ff9-af8a-78ab0f668f4d-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-9dmjv\" (UID: \"fb078957-0631-4ff9-af8a-78ab0f668f4d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv"
Jul 16 22:35:15.471997 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.471975    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb078957-0631-4ff9-af8a-78ab0f668f4d-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-9dmjv\" (UID: \"fb078957-0631-4ff9-af8a-78ab0f668f4d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv"
Jul 16 22:35:15.472119 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.472097    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fb078957-0631-4ff9-af8a-78ab0f668f4d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-9dmjv\" (UID: \"fb078957-0631-4ff9-af8a-78ab0f668f4d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv"
Jul 16 22:35:15.482065 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.482040    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5hn9\" (UniqueName: \"kubernetes.io/projected/fb078957-0631-4ff9-af8a-78ab0f668f4d-kube-api-access-j5hn9\") pod \"prometheus-operator-5945b5cfd-9dmjv\" (UID: \"fb078957-0631-4ff9-af8a-78ab0f668f4d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv"
Jul 16 22:35:15.552864 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:15.552831    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv"
Jul 16 22:35:16.862363 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:16.862334    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv"]
Jul 16 22:35:16.868614 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:35:16.868593    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb078957_0631_4ff9_af8a_78ab0f668f4d.slice/crio-a65a537d4f635bb3e0ffe83edede703d660876a6274f62a0ec555a0e81c2faa3 WatchSource:0}: Error finding container a65a537d4f635bb3e0ffe83edede703d660876a6274f62a0ec555a0e81c2faa3: Status 404 returned error can't find the container with id a65a537d4f635bb3e0ffe83edede703d660876a6274f62a0ec555a0e81c2faa3
Jul 16 22:35:17.236596 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:17.236561    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dlbxx" event={"ID":"86e283ba-586c-45f8-886e-80a3f0a56aa6","Type":"ContainerStarted","Data":"a034f44b5e4ab4b4b1c116d9d27d0479d5a19e3e9891f2d4751391e88912e6c4"}
Jul 16 22:35:17.238192 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:17.238167    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pplph" event={"ID":"76d9a002-345d-4324-a645-71675213c52d","Type":"ContainerStarted","Data":"77180cbdad34b005a053b4b41ffff42df2267d2b58e12b0c87e46fe6e71c84e7"}
Jul 16 22:35:17.238342 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:17.238216    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pplph" event={"ID":"76d9a002-345d-4324-a645-71675213c52d","Type":"ContainerStarted","Data":"5750ca943831db31444646e5563cbdd38e944db7b6b4495f83a4e8e5287b2d4d"}
Jul 16 22:35:17.239272 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:17.239234    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv" event={"ID":"fb078957-0631-4ff9-af8a-78ab0f668f4d","Type":"ContainerStarted","Data":"a65a537d4f635bb3e0ffe83edede703d660876a6274f62a0ec555a0e81c2faa3"}
Jul 16 22:35:17.263021 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:17.262972    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-pplph" podStartSLOduration=1.761581496 podStartE2EDuration="6.262953548s" totalImagesPullingTime="4.501372052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:17.262782475 +0000 UTC m=+128.011266547" watchObservedRunningTime="2026-07-16 22:35:17.262953548 +0000 UTC m=+128.011437618"
Jul 16 22:35:19.249203 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:19.249159    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv" event={"ID":"fb078957-0631-4ff9-af8a-78ab0f668f4d","Type":"ContainerStarted","Data":"9ae1469ea51c09bad647380b43dc8d0bf555c77454865f6557576dd0f929597a"}
Jul 16 22:35:19.249662 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:19.249283    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv" event={"ID":"fb078957-0631-4ff9-af8a-78ab0f668f4d","Type":"ContainerStarted","Data":"e6aabe3f14c601d782972d2bae0fef84cde4f249dd46e08bce40762a5fcb9e17"}
Jul 16 22:35:19.274068 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:19.274011    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-9dmjv" podStartSLOduration=2.88482591 podStartE2EDuration="4.273992445s" totalImagesPullingTime="1.389166535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:19.272340487 +0000 UTC m=+130.020824558" watchObservedRunningTime="2026-07-16 22:35:19.273992445 +0000 UTC m=+130.022476517"
Jul 16 22:35:19.603949 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:19.603879    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs\") pod \"network-metrics-daemon-pbp8p\" (UID: \"d8dfffe2-b127-481e-90c8-d0372f409044\") " pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:35:19.606471 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:19.606440    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8dfffe2-b127-481e-90c8-d0372f409044-metrics-certs\") pod \"network-metrics-daemon-pbp8p\" (UID: \"d8dfffe2-b127-481e-90c8-d0372f409044\") " pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:35:19.641027 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:19.640996    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9dnx4\""
Jul 16 22:35:19.649104 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:19.649073    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbp8p"
Jul 16 22:35:20.842179 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:20.842152    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pbp8p"]
Jul 16 22:35:20.845108 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:35:20.845085    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8dfffe2_b127_481e_90c8_d0372f409044.slice/crio-85377abd28bd65a31cf2661b420a711c29e2cf03da3e5d1c037ff486f7060ecb WatchSource:0}: Error finding container 85377abd28bd65a31cf2661b420a711c29e2cf03da3e5d1c037ff486f7060ecb: Status 404 returned error can't find the container with id 85377abd28bd65a31cf2661b420a711c29e2cf03da3e5d1c037ff486f7060ecb
Jul 16 22:35:21.261816 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.261768    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dlbxx" event={"ID":"86e283ba-586c-45f8-886e-80a3f0a56aa6","Type":"ContainerStarted","Data":"540573bcc05bbebd82d18124af4d48e8ad4dd7f7cd954cd5593b16945eceb011"}
Jul 16 22:35:21.263162 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.263126    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pbp8p" event={"ID":"d8dfffe2-b127-481e-90c8-d0372f409044","Type":"ContainerStarted","Data":"85377abd28bd65a31cf2661b420a711c29e2cf03da3e5d1c037ff486f7060ecb"}
Jul 16 22:35:21.792750 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.792708    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mhpdq"]
Jul 16 22:35:21.796454 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.796424    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.798687 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.798527    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 22:35:21.798687 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.798602    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xww28\""
Jul 16 22:35:21.798876 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.798856    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 22:35:21.803090 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.803069    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 22:35:21.824546 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.824442    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-sys\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.824546 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.824560    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-metrics-client-ca\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.825095 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.824601    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.825095 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.824632    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.825095 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.824733    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-node-exporter-tls\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.825095 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.824779    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgnpr\" (UniqueName: \"kubernetes.io/projected/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-kube-api-access-xgnpr\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.825095 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.824835    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-node-exporter-textfile\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.825095 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.824891    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-root\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.825095 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.824938    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-node-exporter-wtmp\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.925408 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.925370    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-node-exporter-tls\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.925877 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.925422    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnpr\" (UniqueName: \"kubernetes.io/projected/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-kube-api-access-xgnpr\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.925877 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.925461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-node-exporter-textfile\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.925877 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:35:21.925552    2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 16 22:35:21.925877 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:35:21.925635    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-node-exporter-tls podName:bbb767dc-1b1a-4aad-b781-bdd9dcf75e41 nodeName:}" failed. No retries permitted until 2026-07-16 22:35:22.425610635 +0000 UTC m=+133.174094707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-node-exporter-tls") pod "node-exporter-mhpdq" (UID: "bbb767dc-1b1a-4aad-b781-bdd9dcf75e41") : secret "node-exporter-tls" not found
Jul 16 22:35:21.925877 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.925695    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-root\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.925877 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.925749    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-node-exporter-wtmp\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.925877 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.925790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-sys\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.925877 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.925808    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-node-exporter-textfile\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.925877 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.925810    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-root\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.925877 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.925859    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-sys\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.925877 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.925864    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-metrics-client-ca\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.926432 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.925898    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.926432 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.925927    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.926432 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.925933    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-node-exporter-wtmp\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.926626 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.926438    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-node-exporter-accelerators-collector-config\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.926626 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.926441    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-metrics-client-ca\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.928981 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.928949    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:21.938617 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:21.938589    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgnpr\" (UniqueName: \"kubernetes.io/projected/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-kube-api-access-xgnpr\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:22.272328 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:22.272290    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pbp8p" event={"ID":"d8dfffe2-b127-481e-90c8-d0372f409044","Type":"ContainerStarted","Data":"c3a2d1da36a2226821aa6adf85a9449796431504b83418776859445297c31229"}
Jul 16 22:35:22.430408 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:22.430314    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-node-exporter-tls\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:22.432895 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:22.432872    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bbb767dc-1b1a-4aad-b781-bdd9dcf75e41-node-exporter-tls\") pod \"node-exporter-mhpdq\" (UID: \"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41\") " pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:22.709151 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:22.709062    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mhpdq"
Jul 16 22:35:23.280294 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:23.279828    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pbp8p" event={"ID":"d8dfffe2-b127-481e-90c8-d0372f409044","Type":"ContainerStarted","Data":"37149d3d1e8598385ca83c96d685f83a2c50c3b4a3d4a8d502f176c490b70aab"}
Jul 16 22:35:24.472025 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:35:24.471992    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb767dc_1b1a_4aad_b781_bdd9dcf75e41.slice/crio-5b8811d76f708297d94f30e76574ba575fce411c9185f5dedccbee20f12fc9e3 WatchSource:0}: Error finding container 5b8811d76f708297d94f30e76574ba575fce411c9185f5dedccbee20f12fc9e3: Status 404 returned error can't find the container with id 5b8811d76f708297d94f30e76574ba575fce411c9185f5dedccbee20f12fc9e3
Jul 16 22:35:25.290068 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.290031    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dlbxx" event={"ID":"86e283ba-586c-45f8-886e-80a3f0a56aa6","Type":"ContainerStarted","Data":"701ff55a177c6c98bf94db5da5ecb02a2a8bc1cef784ce9975e508f43e8f90e1"}
Jul 16 22:35:25.291156 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.291133    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhpdq" event={"ID":"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41","Type":"ContainerStarted","Data":"5b8811d76f708297d94f30e76574ba575fce411c9185f5dedccbee20f12fc9e3"}
Jul 16 22:35:25.316417 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.316377    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dlbxx" podStartSLOduration=1.529466464 podStartE2EDuration="14.316360921s" totalImagesPullingTime="12.786894457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:35:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:25.315448663 +0000 UTC m=+136.063932733" watchObservedRunningTime="2026-07-16 22:35:25.316360921 +0000 UTC m=+136.064845171"
Jul 16 22:35:25.316584 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.316548    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pbp8p" podStartSLOduration=134.029829488 podStartE2EDuration="2m15.316542436s" totalImagesPullingTime="1.286712948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:23.305353722 +0000 UTC m=+134.053837795" watchObservedRunningTime="2026-07-16 22:35:25.316542436 +0000 UTC m=+136.065026508"
Jul 16 22:35:25.706266 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.706220    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-75c84d948b-hw986"]
Jul 16 22:35:25.709475 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.709459    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.713670 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.712422    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 16 22:35:25.713670 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.712463    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 16 22:35:25.713670 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.712730    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 16 22:35:25.713670 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.712878    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 16 22:35:25.713670 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.713021    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-eko0dnvi5r3bg\""
Jul 16 22:35:25.713670 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.713330    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 16 22:35:25.713670 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.713566    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4rwgx\""
Jul 16 22:35:25.724777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.724750    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75c84d948b-hw986"]
Jul 16 22:35:25.760784 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.760762    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3bf1d80d-3994-4c03-a50a-496478abf16e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.760894 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.760801    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3bf1d80d-3994-4c03-a50a-496478abf16e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.760894 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.760820    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3bf1d80d-3994-4c03-a50a-496478abf16e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.760894 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.760846    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54kr5\" (UniqueName: \"kubernetes.io/projected/3bf1d80d-3994-4c03-a50a-496478abf16e-kube-api-access-54kr5\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.761004 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.760915    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3bf1d80d-3994-4c03-a50a-496478abf16e-secret-thanos-querier-tls\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.761004 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.760934    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3bf1d80d-3994-4c03-a50a-496478abf16e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.761004 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.760978    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3bf1d80d-3994-4c03-a50a-496478abf16e-secret-grpc-tls\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.761004 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.760993    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bf1d80d-3994-4c03-a50a-496478abf16e-metrics-client-ca\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.862062 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.862030    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3bf1d80d-3994-4c03-a50a-496478abf16e-secret-grpc-tls\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.862062 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.862069    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bf1d80d-3994-4c03-a50a-496478abf16e-metrics-client-ca\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.862197 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.862117    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3bf1d80d-3994-4c03-a50a-496478abf16e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.862197 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.862164    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3bf1d80d-3994-4c03-a50a-496478abf16e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.862197 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.862191    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3bf1d80d-3994-4c03-a50a-496478abf16e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.862417 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.862219    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54kr5\" (UniqueName: \"kubernetes.io/projected/3bf1d80d-3994-4c03-a50a-496478abf16e-kube-api-access-54kr5\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.862473 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.862457    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3bf1d80d-3994-4c03-a50a-496478abf16e-secret-thanos-querier-tls\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.862528 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.862491    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3bf1d80d-3994-4c03-a50a-496478abf16e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.862968 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.862944    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bf1d80d-3994-4c03-a50a-496478abf16e-metrics-client-ca\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.864789 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.864769    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3bf1d80d-3994-4c03-a50a-496478abf16e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.864789 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.864782    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3bf1d80d-3994-4c03-a50a-496478abf16e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.865088 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.865072    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3bf1d80d-3994-4c03-a50a-496478abf16e-secret-grpc-tls\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.865179 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.865157    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3bf1d80d-3994-4c03-a50a-496478abf16e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.865215 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.865179    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3bf1d80d-3994-4c03-a50a-496478abf16e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.865317 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.865298    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3bf1d80d-3994-4c03-a50a-496478abf16e-secret-thanos-querier-tls\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:25.870169 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:25.870151    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54kr5\" (UniqueName: \"kubernetes.io/projected/3bf1d80d-3994-4c03-a50a-496478abf16e-kube-api-access-54kr5\") pod \"thanos-querier-75c84d948b-hw986\" (UID: \"3bf1d80d-3994-4c03-a50a-496478abf16e\") " pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:26.022668 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:26.022603    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:26.159119 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:26.159093    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75c84d948b-hw986"]
Jul 16 22:35:26.161154 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:35:26.161125    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf1d80d_3994_4c03_a50a_496478abf16e.slice/crio-e5055145435a5e5061d2f6fd4e8d307db393e0cc416feb3a50ad9441201bfaba WatchSource:0}: Error finding container e5055145435a5e5061d2f6fd4e8d307db393e0cc416feb3a50ad9441201bfaba: Status 404 returned error can't find the container with id e5055145435a5e5061d2f6fd4e8d307db393e0cc416feb3a50ad9441201bfaba
Jul 16 22:35:26.295540 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:26.295470    2565 generic.go:356] "Generic (PLEG): container finished" podID="bbb767dc-1b1a-4aad-b781-bdd9dcf75e41" containerID="f78fa10d1986a527017edc470d8ec82f6f28c254070c904168782f77489b17c1" exitCode=0
Jul 16 22:35:26.295660 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:26.295540    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhpdq" event={"ID":"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41","Type":"ContainerDied","Data":"f78fa10d1986a527017edc470d8ec82f6f28c254070c904168782f77489b17c1"}
Jul 16 22:35:26.296809 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:26.296781    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75c84d948b-hw986" event={"ID":"3bf1d80d-3994-4c03-a50a-496478abf16e","Type":"ContainerStarted","Data":"e5055145435a5e5061d2f6fd4e8d307db393e0cc416feb3a50ad9441201bfaba"}
Jul 16 22:35:26.456632 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:26.456604    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-7d6x7"]
Jul 16 22:35:26.459640 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:26.459625    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-7d6x7"
Jul 16 22:35:26.461836 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:26.461817    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-fgp8d\""
Jul 16 22:35:26.462047 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:26.462033    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 16 22:35:26.478997 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:26.478970    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-7d6x7"]
Jul 16 22:35:26.569010 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:26.568939    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9wr5\" (UniqueName: \"kubernetes.io/projected/83289e15-a6aa-453a-bac4-355fdadd007e-kube-api-access-r9wr5\") pod \"monitoring-plugin-7b7b7bc8bf-7d6x7\" (UID: \"83289e15-a6aa-453a-bac4-355fdadd007e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-7d6x7"
Jul 16 22:35:26.569010 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:26.568985    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83289e15-a6aa-453a-bac4-355fdadd007e-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-7d6x7\" (UID: \"83289e15-a6aa-453a-bac4-355fdadd007e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-7d6x7"
Jul 16 22:35:26.669732 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:26.669697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9wr5\" (UniqueName: \"kubernetes.io/projected/83289e15-a6aa-453a-bac4-355fdadd007e-kube-api-access-r9wr5\") pod \"monitoring-plugin-7b7b7bc8bf-7d6x7\" (UID: \"83289e15-a6aa-453a-bac4-355fdadd007e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-7d6x7"
Jul 16 22:35:26.669882 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:26.669747    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83289e15-a6aa-453a-bac4-355fdadd007e-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-7d6x7\" (UID: \"83289e15-a6aa-453a-bac4-355fdadd007e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-7d6x7"
Jul 16 22:35:26.672243 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:26.672215    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/83289e15-a6aa-453a-bac4-355fdadd007e-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-7d6x7\" (UID: \"83289e15-a6aa-453a-bac4-355fdadd007e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-7d6x7"
Jul 16 22:35:26.681574 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:26.681553    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9wr5\" (UniqueName: \"kubernetes.io/projected/83289e15-a6aa-453a-bac4-355fdadd007e-kube-api-access-r9wr5\") pod \"monitoring-plugin-7b7b7bc8bf-7d6x7\" (UID: \"83289e15-a6aa-453a-bac4-355fdadd007e\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-7d6x7"
Jul 16 22:35:26.768387 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:26.768355    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-7d6x7"
Jul 16 22:35:26.905488 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:26.905455    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-7d6x7"]
Jul 16 22:35:26.908702 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:35:26.908672    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83289e15_a6aa_453a_bac4_355fdadd007e.slice/crio-641a7771f73df5e8b7a81900dfc2120094f75c0698b088405f90d1fced1457bc WatchSource:0}: Error finding container 641a7771f73df5e8b7a81900dfc2120094f75c0698b088405f90d1fced1457bc: Status 404 returned error can't find the container with id 641a7771f73df5e8b7a81900dfc2120094f75c0698b088405f90d1fced1457bc
Jul 16 22:35:27.303012 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.302965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhpdq" event={"ID":"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41","Type":"ContainerStarted","Data":"30ba7ead9e7de2408d933a398b8d2e4e5723d0582d0d6d00fb5e6a90cb5d61a2"}
Jul 16 22:35:27.303267 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.303028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mhpdq" event={"ID":"bbb767dc-1b1a-4aad-b781-bdd9dcf75e41","Type":"ContainerStarted","Data":"a1e9b86fd0a63125f9e32e82cd356951abe99ed3c47b122145bfa8e8bfc554d8"}
Jul 16 22:35:27.304422 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.304386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-7d6x7" event={"ID":"83289e15-a6aa-453a-bac4-355fdadd007e","Type":"ContainerStarted","Data":"641a7771f73df5e8b7a81900dfc2120094f75c0698b088405f90d1fced1457bc"}
Jul 16 22:35:27.331173 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.331133    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mhpdq" podStartSLOduration=5.486810275 podStartE2EDuration="6.331118276s" totalImagesPullingTime="780.127286ms" totalInitContainerRuntime="64.180715ms" isStatefulPod=true podCreationTimestamp="2026-07-16 22:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:27.329048158 +0000 UTC m=+138.077532238" watchObservedRunningTime="2026-07-16 22:35:27.331118276 +0000 UTC m=+138.079602348"
Jul 16 22:35:27.937184 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.935824    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 22:35:27.944693 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.944667    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.950304 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.950281    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 16 22:35:27.950490 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.950394    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 16 22:35:27.950573 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.950439    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 16 22:35:27.950636 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.950588    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 16 22:35:27.950707 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.950682    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 16 22:35:27.950760 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.950708    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 16 22:35:27.950852 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.950832    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 16 22:35:27.950971 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.950937    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 16 22:35:27.951158 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.951124    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 16 22:35:27.951261 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.951209    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 16 22:35:27.951319 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.951276    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 16 22:35:27.951736 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.951541    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 16 22:35:27.951736 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.951559    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 16 22:35:27.951736 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.951577    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-99t6n7vk14vv0\""
Jul 16 22:35:27.951736 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.951579    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 16 22:35:27.951736 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.951615    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dtms6\""
Jul 16 22:35:27.953282 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.953198    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 16 22:35:27.962173 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.962154    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 22:35:27.983232 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983190    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.983232 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983237    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.983501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/422deed3-a980-48b1-bbdd-21e309d6c0ed-config-out\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.983501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983350    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.983501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.983501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.983501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983475    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.983501 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.983796 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983543    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/422deed3-a980-48b1-bbdd-21e309d6c0ed-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.983796 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983571    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.983796 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983620    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.983796 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983663    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4fhh\" (UniqueName: \"kubernetes.io/projected/422deed3-a980-48b1-bbdd-21e309d6c0ed-kube-api-access-t4fhh\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.983796 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983720    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-config\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.983796 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983774    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.984082 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983809    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.984082 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983854    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.984082 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.984082 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983929    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.984082 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.983983    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:27.984082 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:27.984009    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-web-config\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.085976 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.084742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.085976 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.084789    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.085976 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.084827    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/422deed3-a980-48b1-bbdd-21e309d6c0ed-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.085976 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.084853    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.085976 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.084877    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.085976 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.084898    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4fhh\" (UniqueName: \"kubernetes.io/projected/422deed3-a980-48b1-bbdd-21e309d6c0ed-kube-api-access-t4fhh\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.085976 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.084928    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-config\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.085976 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.084966    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.085976 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.084992    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.085976 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.085019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.085976 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.085056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.085976 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.085088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.085976 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.085131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.085976 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.085155    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-web-config\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.086900 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.085186    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.086900 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.085209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.086900 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.085236    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/422deed3-a980-48b1-bbdd-21e309d6c0ed-config-out\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.086900 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.085279    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.086900 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.085305    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.086900 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.085327    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.086900 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.086294    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.086900 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.086578    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.087317 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.086996    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.088132 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.088075    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.089162 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.088466    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.089162 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.088819    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.089162 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.089154    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.090079 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.089584    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-config\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.090079 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.089744    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/422deed3-a980-48b1-bbdd-21e309d6c0ed-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.090079 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.089836    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.090079 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.090057    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.090879 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.090834    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.090992 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.090968    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.091274 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.091219    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/422deed3-a980-48b1-bbdd-21e309d6c0ed-config-out\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.091630 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.091582    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.092312 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.092231    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.092837 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.092814    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.092960 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.092940    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-web-config\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.093026 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.093005    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.099660 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.099643    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4fhh\" (UniqueName: \"kubernetes.io/projected/422deed3-a980-48b1-bbdd-21e309d6c0ed-kube-api-access-t4fhh\") pod \"prometheus-k8s-0\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.258865 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.258783    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:28.655619 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:28.655588    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 22:35:28.667575 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:35:28.667546    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422deed3_a980_48b1_bbdd_21e309d6c0ed.slice/crio-8a30fc64876d8706f2a60a695a7c647fdd612ec1a25d166206289553b3d64d8b WatchSource:0}: Error finding container 8a30fc64876d8706f2a60a695a7c647fdd612ec1a25d166206289553b3d64d8b: Status 404 returned error can't find the container with id 8a30fc64876d8706f2a60a695a7c647fdd612ec1a25d166206289553b3d64d8b
Jul 16 22:35:29.314146 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:29.314107    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75c84d948b-hw986" event={"ID":"3bf1d80d-3994-4c03-a50a-496478abf16e","Type":"ContainerStarted","Data":"faf627132791275dc07e15783566abab5e96411342803eb214592114d4579316"}
Jul 16 22:35:29.314580 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:29.314177    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75c84d948b-hw986" event={"ID":"3bf1d80d-3994-4c03-a50a-496478abf16e","Type":"ContainerStarted","Data":"ae67df30e5d7ec09b2ad94bcf176f93bbd44028bbf157fcde56dc525e9b6128a"}
Jul 16 22:35:29.314580 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:29.314204    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75c84d948b-hw986" event={"ID":"3bf1d80d-3994-4c03-a50a-496478abf16e","Type":"ContainerStarted","Data":"c29b8d2ec9f0a3dd3ade5bb0e73ba5705489e3864e1d4ba51d39b5bcf0a78034"}
Jul 16 22:35:29.315572 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:29.315546    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-7d6x7" event={"ID":"83289e15-a6aa-453a-bac4-355fdadd007e","Type":"ContainerStarted","Data":"28108c799eac2c9621f3d532e822ff6122803d1cf5072f42b0d76f9b2c38e921"}
Jul 16 22:35:29.315781 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:29.315767    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-7d6x7"
Jul 16 22:35:29.316946 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:29.316925    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"422deed3-a980-48b1-bbdd-21e309d6c0ed","Type":"ContainerStarted","Data":"8a30fc64876d8706f2a60a695a7c647fdd612ec1a25d166206289553b3d64d8b"}
Jul 16 22:35:29.320898 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:29.320874    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-7d6x7"
Jul 16 22:35:29.340523 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:29.340485    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-7d6x7" podStartSLOduration=1.741865533 podStartE2EDuration="3.340471143s" totalImagesPullingTime="1.59860561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:29.338778999 +0000 UTC m=+140.087263070" watchObservedRunningTime="2026-07-16 22:35:29.340471143 +0000 UTC m=+140.088955214"
Jul 16 22:35:30.323288 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:30.323172    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75c84d948b-hw986" event={"ID":"3bf1d80d-3994-4c03-a50a-496478abf16e","Type":"ContainerStarted","Data":"a132601f72764cc9439dd62d69f340d8f631311d68980472b26883264ac7fb80"}
Jul 16 22:35:30.323288 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:30.323276    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75c84d948b-hw986" event={"ID":"3bf1d80d-3994-4c03-a50a-496478abf16e","Type":"ContainerStarted","Data":"c8bebfcf7f1198fa533d1bb565e790946a1c051e191d1d43e9c87db36525e4a9"}
Jul 16 22:35:30.323703 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:30.323332    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:30.323703 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:30.323374    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75c84d948b-hw986" event={"ID":"3bf1d80d-3994-4c03-a50a-496478abf16e","Type":"ContainerStarted","Data":"65c05ae9b6701bd900926a71078fae4e601b792078d0223534ac03a511dba287"}
Jul 16 22:35:30.324725 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:30.324700    2565 generic.go:356] "Generic (PLEG): container finished" podID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerID="3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8" exitCode=0
Jul 16 22:35:30.324834 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:30.324782    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"422deed3-a980-48b1-bbdd-21e309d6c0ed","Type":"ContainerDied","Data":"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8"}
Jul 16 22:35:30.347613 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:30.347555    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-75c84d948b-hw986" podStartSLOduration=1.8387385840000001 podStartE2EDuration="5.347535657s" totalImagesPullingTime="3.508797073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:30.346622561 +0000 UTC m=+141.095106634" watchObservedRunningTime="2026-07-16 22:35:30.347535657 +0000 UTC m=+141.096019733"
Jul 16 22:35:34.351647 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:34.351609    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"422deed3-a980-48b1-bbdd-21e309d6c0ed","Type":"ContainerStarted","Data":"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a"}
Jul 16 22:35:34.352031 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:34.351699    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"422deed3-a980-48b1-bbdd-21e309d6c0ed","Type":"ContainerStarted","Data":"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95"}
Jul 16 22:35:34.352031 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:34.351729    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"422deed3-a980-48b1-bbdd-21e309d6c0ed","Type":"ContainerStarted","Data":"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b"}
Jul 16 22:35:34.352031 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:34.351775    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"422deed3-a980-48b1-bbdd-21e309d6c0ed","Type":"ContainerStarted","Data":"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d"}
Jul 16 22:35:34.352031 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:34.351798    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"422deed3-a980-48b1-bbdd-21e309d6c0ed","Type":"ContainerStarted","Data":"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b"}
Jul 16 22:35:34.352031 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:34.351821    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"422deed3-a980-48b1-bbdd-21e309d6c0ed","Type":"ContainerStarted","Data":"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332"}
Jul 16 22:35:34.381669 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:34.381584    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.98867256 podStartE2EDuration="7.381563691s" totalImagesPullingTime="4.37346767s" totalInitContainerRuntime="19.423461ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:34.38068067 +0000 UTC m=+145.129164740" watchObservedRunningTime="2026-07-16 22:35:34.381563691 +0000 UTC m=+145.130047766"
Jul 16 22:35:36.335565 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:36.335538    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-75c84d948b-hw986"
Jul 16 22:35:38.259307 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:38.259238    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:35:43.149349 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.149312    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b99bcf697-55fzz"]
Jul 16 22:35:43.149795 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:35:43.149677    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6b99bcf697-55fzz" podUID="41bf030b-380e-4614-a98a-a471cc1fd0e8"
Jul 16 22:35:43.385621 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.385583    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:35:43.394135 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.394110    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:35:43.520666 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.520582    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41bf030b-380e-4614-a98a-a471cc1fd0e8-image-registry-private-configuration\") pod \"41bf030b-380e-4614-a98a-a471cc1fd0e8\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") "
Jul 16 22:35:43.520666 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.520642    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41bf030b-380e-4614-a98a-a471cc1fd0e8-ca-trust-extracted\") pod \"41bf030b-380e-4614-a98a-a471cc1fd0e8\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") "
Jul 16 22:35:43.520885 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.520683    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41bf030b-380e-4614-a98a-a471cc1fd0e8-trusted-ca\") pod \"41bf030b-380e-4614-a98a-a471cc1fd0e8\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") "
Jul 16 22:35:43.520885 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.520727    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-certificates\") pod \"41bf030b-380e-4614-a98a-a471cc1fd0e8\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") "
Jul 16 22:35:43.520885 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.520755    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41bf030b-380e-4614-a98a-a471cc1fd0e8-installation-pull-secrets\") pod \"41bf030b-380e-4614-a98a-a471cc1fd0e8\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") "
Jul 16 22:35:43.521044 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.520887    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkrw5\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-kube-api-access-jkrw5\") pod \"41bf030b-380e-4614-a98a-a471cc1fd0e8\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") "
Jul 16 22:35:43.521044 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.520941    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-bound-sa-token\") pod \"41bf030b-380e-4614-a98a-a471cc1fd0e8\" (UID: \"41bf030b-380e-4614-a98a-a471cc1fd0e8\") "
Jul 16 22:35:43.521044 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.520978    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41bf030b-380e-4614-a98a-a471cc1fd0e8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "41bf030b-380e-4614-a98a-a471cc1fd0e8" (UID: "41bf030b-380e-4614-a98a-a471cc1fd0e8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:35:43.521205 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.521129    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41bf030b-380e-4614-a98a-a471cc1fd0e8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "41bf030b-380e-4614-a98a-a471cc1fd0e8" (UID: "41bf030b-380e-4614-a98a-a471cc1fd0e8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:43.521205 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.521148    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "41bf030b-380e-4614-a98a-a471cc1fd0e8" (UID: "41bf030b-380e-4614-a98a-a471cc1fd0e8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:43.521342 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.521242    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-certificates\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:35:43.521342 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.521283    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41bf030b-380e-4614-a98a-a471cc1fd0e8-ca-trust-extracted\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:35:43.521342 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.521292    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41bf030b-380e-4614-a98a-a471cc1fd0e8-trusted-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:35:43.523008 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.522984    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41bf030b-380e-4614-a98a-a471cc1fd0e8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "41bf030b-380e-4614-a98a-a471cc1fd0e8" (UID: "41bf030b-380e-4614-a98a-a471cc1fd0e8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:35:43.523128 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.523022    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-kube-api-access-jkrw5" (OuterVolumeSpecName: "kube-api-access-jkrw5") pod "41bf030b-380e-4614-a98a-a471cc1fd0e8" (UID: "41bf030b-380e-4614-a98a-a471cc1fd0e8"). InnerVolumeSpecName "kube-api-access-jkrw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:35:43.523128 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.523066    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41bf030b-380e-4614-a98a-a471cc1fd0e8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "41bf030b-380e-4614-a98a-a471cc1fd0e8" (UID: "41bf030b-380e-4614-a98a-a471cc1fd0e8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:35:43.523128 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.523099    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "41bf030b-380e-4614-a98a-a471cc1fd0e8" (UID: "41bf030b-380e-4614-a98a-a471cc1fd0e8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:35:43.621882 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.621841    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41bf030b-380e-4614-a98a-a471cc1fd0e8-image-registry-private-configuration\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:35:43.621882 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.621874    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41bf030b-380e-4614-a98a-a471cc1fd0e8-installation-pull-secrets\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:35:43.621882 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.621887    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkrw5\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-kube-api-access-jkrw5\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:35:43.622111 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:43.621895    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-bound-sa-token\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:35:44.389366 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:44.389337    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b99bcf697-55fzz"
Jul 16 22:35:44.421934 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:44.421900    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b99bcf697-55fzz"]
Jul 16 22:35:44.425949 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:44.425923    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b99bcf697-55fzz"]
Jul 16 22:35:44.527738 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:44.527703    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41bf030b-380e-4614-a98a-a471cc1fd0e8-registry-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:35:45.725154 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:45.725118    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41bf030b-380e-4614-a98a-a471cc1fd0e8" path="/var/lib/kubelet/pods/41bf030b-380e-4614-a98a-a471cc1fd0e8/volumes"
Jul 16 22:35:48.156457 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:35:48.156413    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bnnqb" podUID="25c597a0-b7b8-4ba5-96d5-e6155af0a3fd"
Jul 16 22:35:48.403746 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:48.403715    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:35:53.107692 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:53.107653    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:35:53.109914 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:53.109887    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/25c597a0-b7b8-4ba5-96d5-e6155af0a3fd-metrics-tls\") pod \"dns-default-bnnqb\" (UID: \"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd\") " pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:35:53.207700 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:53.207666    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-57zhl\""
Jul 16 22:35:53.215450 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:53.215425    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:35:53.364678 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:53.364590    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bnnqb"]
Jul 16 22:35:53.368943 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:35:53.368912    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c597a0_b7b8_4ba5_96d5_e6155af0a3fd.slice/crio-623ff272b06bda6e3dbb163c9b7f695316955bbae0567f8ae34860e5bffa97f1 WatchSource:0}: Error finding container 623ff272b06bda6e3dbb163c9b7f695316955bbae0567f8ae34860e5bffa97f1: Status 404 returned error can't find the container with id 623ff272b06bda6e3dbb163c9b7f695316955bbae0567f8ae34860e5bffa97f1
Jul 16 22:35:53.425189 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:53.425154    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bnnqb" event={"ID":"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd","Type":"ContainerStarted","Data":"623ff272b06bda6e3dbb163c9b7f695316955bbae0567f8ae34860e5bffa97f1"}
Jul 16 22:35:56.438692 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:56.438654    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bnnqb" event={"ID":"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd","Type":"ContainerStarted","Data":"4a635349c35acb296abfdc4d0eaa2c2b02223d9938a5cbd4bbf6e2cc91930aa8"}
Jul 16 22:35:56.439084 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:56.438719    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bnnqb" event={"ID":"25c597a0-b7b8-4ba5-96d5-e6155af0a3fd","Type":"ContainerStarted","Data":"2029e6b7e8e88e71b55dc604a774474ee82acb3d691245e7f0ba9e3114ff464f"}
Jul 16 22:35:56.439084 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:56.438823    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:35:56.460208 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:35:56.460153    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bnnqb" podStartSLOduration=129.045715107 podStartE2EDuration="2m11.460137379s" totalImagesPullingTime="2.414422272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:56.458906634 +0000 UTC m=+167.207390705" watchObservedRunningTime="2026-07-16 22:35:56.460137379 +0000 UTC m=+167.208621450"
Jul 16 22:36:03.451347 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:03.451263    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-44bjw_156c60da-5d5c-4d2e-9d91-05d6e4a31cba/cluster-monitoring-operator/0.log"
Jul 16 22:36:04.445037 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:04.445009    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-7d6x7_83289e15-a6aa-453a-bac4-355fdadd007e/monitoring-plugin/0.log"
Jul 16 22:36:05.844552 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:05.844520    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhpdq_bbb767dc-1b1a-4aad-b781-bdd9dcf75e41/init-textfile/0.log"
Jul 16 22:36:06.046112 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:06.046082    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhpdq_bbb767dc-1b1a-4aad-b781-bdd9dcf75e41/node-exporter/0.log"
Jul 16 22:36:06.244849 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:06.244771    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhpdq_bbb767dc-1b1a-4aad-b781-bdd9dcf75e41/kube-rbac-proxy/0.log"
Jul 16 22:36:06.444451 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:06.444426    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bnnqb"
Jul 16 22:36:07.044438 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:07.044410    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_422deed3-a980-48b1-bbdd-21e309d6c0ed/init-config-reloader/0.log"
Jul 16 22:36:07.247839 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:07.247808    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_422deed3-a980-48b1-bbdd-21e309d6c0ed/prometheus/0.log"
Jul 16 22:36:07.447782 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:07.447745    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_422deed3-a980-48b1-bbdd-21e309d6c0ed/config-reloader/0.log"
Jul 16 22:36:07.645990 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:07.645960    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_422deed3-a980-48b1-bbdd-21e309d6c0ed/thanos-sidecar/0.log"
Jul 16 22:36:07.845150 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:07.845120    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_422deed3-a980-48b1-bbdd-21e309d6c0ed/kube-rbac-proxy-web/0.log"
Jul 16 22:36:08.044572 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:08.044545    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_422deed3-a980-48b1-bbdd-21e309d6c0ed/kube-rbac-proxy/0.log"
Jul 16 22:36:08.245601 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:08.245519    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_422deed3-a980-48b1-bbdd-21e309d6c0ed/kube-rbac-proxy-thanos/0.log"
Jul 16 22:36:08.447834 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:08.447798    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-9dmjv_fb078957-0631-4ff9-af8a-78ab0f668f4d/prometheus-operator/0.log"
Jul 16 22:36:08.648649 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:08.648625    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-9dmjv_fb078957-0631-4ff9-af8a-78ab0f668f4d/kube-rbac-proxy/0.log"
Jul 16 22:36:08.844987 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:08.844958    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-r4kck_a17fd63f-8198-4623-a983-6577ad6983f4/prometheus-operator-admission-webhook/0.log"
Jul 16 22:36:09.045166 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:09.045088    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75c84d948b-hw986_3bf1d80d-3994-4c03-a50a-496478abf16e/thanos-query/0.log"
Jul 16 22:36:09.244230 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:09.244197    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75c84d948b-hw986_3bf1d80d-3994-4c03-a50a-496478abf16e/kube-rbac-proxy-web/0.log"
Jul 16 22:36:09.444977 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:09.444949    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75c84d948b-hw986_3bf1d80d-3994-4c03-a50a-496478abf16e/kube-rbac-proxy/0.log"
Jul 16 22:36:09.645919 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:09.645889    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75c84d948b-hw986_3bf1d80d-3994-4c03-a50a-496478abf16e/prom-label-proxy/0.log"
Jul 16 22:36:09.844487 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:09.844462    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75c84d948b-hw986_3bf1d80d-3994-4c03-a50a-496478abf16e/kube-rbac-proxy-rules/0.log"
Jul 16 22:36:10.045089 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:10.045059    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75c84d948b-hw986_3bf1d80d-3994-4c03-a50a-496478abf16e/kube-rbac-proxy-metrics/0.log"
Jul 16 22:36:28.259755 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:28.259714    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:28.278612 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:28.278584    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:28.576846 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:28.576823    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:46.330713 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.330671    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 22:36:46.331323 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.330964    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="prometheus" containerID="cri-o://cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332" gracePeriod=600
Jul 16 22:36:46.331323 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.330985    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="kube-rbac-proxy" containerID="cri-o://433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95" gracePeriod=600
Jul 16 22:36:46.331323 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.331012    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="thanos-sidecar" containerID="cri-o://a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d" gracePeriod=600
Jul 16 22:36:46.331323 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.331070    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a" gracePeriod=600
Jul 16 22:36:46.331323 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.331097    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="config-reloader" containerID="cri-o://5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b" gracePeriod=600
Jul 16 22:36:46.331323 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.331032    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="kube-rbac-proxy-web" containerID="cri-o://940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b" gracePeriod=600
Jul 16 22:36:46.582499 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.582436    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:46.638649 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.638614    2565 generic.go:356] "Generic (PLEG): container finished" podID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerID="f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a" exitCode=0
Jul 16 22:36:46.638649 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.638646    2565 generic.go:356] "Generic (PLEG): container finished" podID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerID="433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95" exitCode=0
Jul 16 22:36:46.638649 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.638657    2565 generic.go:356] "Generic (PLEG): container finished" podID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerID="940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b" exitCode=0
Jul 16 22:36:46.638952 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.638666    2565 generic.go:356] "Generic (PLEG): container finished" podID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerID="a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d" exitCode=0
Jul 16 22:36:46.638952 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.638655    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"422deed3-a980-48b1-bbdd-21e309d6c0ed","Type":"ContainerDied","Data":"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a"}
Jul 16 22:36:46.638952 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.638725    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:46.638952 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.638786    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"422deed3-a980-48b1-bbdd-21e309d6c0ed","Type":"ContainerDied","Data":"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95"}
Jul 16 22:36:46.638952 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.638831    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"422deed3-a980-48b1-bbdd-21e309d6c0ed","Type":"ContainerDied","Data":"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b"}
Jul 16 22:36:46.638952 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.638870    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"422deed3-a980-48b1-bbdd-21e309d6c0ed","Type":"ContainerDied","Data":"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d"}
Jul 16 22:36:46.638952 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.638911    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"422deed3-a980-48b1-bbdd-21e309d6c0ed","Type":"ContainerDied","Data":"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b"}
Jul 16 22:36:46.639420 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.638963    2565 scope.go:117] "RemoveContainer" containerID="f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a"
Jul 16 22:36:46.639420 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.638671    2565 generic.go:356] "Generic (PLEG): container finished" podID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerID="5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b" exitCode=0
Jul 16 22:36:46.639420 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.639106    2565 generic.go:356] "Generic (PLEG): container finished" podID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerID="cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332" exitCode=0
Jul 16 22:36:46.639420 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.639169    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"422deed3-a980-48b1-bbdd-21e309d6c0ed","Type":"ContainerDied","Data":"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332"}
Jul 16 22:36:46.639420 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.639212    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"422deed3-a980-48b1-bbdd-21e309d6c0ed","Type":"ContainerDied","Data":"8a30fc64876d8706f2a60a695a7c647fdd612ec1a25d166206289553b3d64d8b"}
Jul 16 22:36:46.649259 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.649219    2565 scope.go:117] "RemoveContainer" containerID="433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95"
Jul 16 22:36:46.651484 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.651455    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-web-config\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.651910 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.651506    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-config\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.651910 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.651539    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-rulefiles-0\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.651910 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.651568    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-db\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.651910 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.651602    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/422deed3-a980-48b1-bbdd-21e309d6c0ed-tls-assets\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.651910 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.651643    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-thanos-prometheus-http-client-file\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.651910 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.651685    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-trusted-ca-bundle\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.651910 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.651714    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-prometheus-k8s-tls\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.651910 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.651741    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-configmap-serving-certs-ca-bundle\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.651910 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.651772    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.651910 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.651796    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/422deed3-a980-48b1-bbdd-21e309d6c0ed-config-out\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.651910 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.651859    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4fhh\" (UniqueName: \"kubernetes.io/projected/422deed3-a980-48b1-bbdd-21e309d6c0ed-kube-api-access-t4fhh\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.651910 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.651898    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-rulefiles-2\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.652563 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.651953    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-configmap-metrics-client-ca\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.652563 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.651978    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-grpc-tls\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.652563 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.652006    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-metrics-client-certs\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.652563 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.652031    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-rulefiles-1\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.652563 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.652058    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.652563 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.652096    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-configmap-kubelet-serving-ca-bundle\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.652563 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.652121    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-kube-rbac-proxy\") pod \"422deed3-a980-48b1-bbdd-21e309d6c0ed\" (UID: \"422deed3-a980-48b1-bbdd-21e309d6c0ed\") "
Jul 16 22:36:46.652563 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.652340    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:46.652563 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.652395    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:46.653891 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.652708    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:46.655315 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.655290    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:46.655560 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.655469    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:46.656530 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.656502    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:46.657013 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.656988    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:46.657687 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.657659    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:36:46.657790 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.657742    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:36:46.658208 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.658169    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:36:46.658701 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.658574    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422deed3-a980-48b1-bbdd-21e309d6c0ed-config-out" (OuterVolumeSpecName: "config-out") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:36:46.658701 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.658606    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:36:46.658701 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.658656    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422deed3-a980-48b1-bbdd-21e309d6c0ed-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:36:46.659145 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.659065    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:36:46.659340 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.659307    2565 scope.go:117] "RemoveContainer" containerID="940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b"
Jul 16 22:36:46.659708 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.659686    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422deed3-a980-48b1-bbdd-21e309d6c0ed-kube-api-access-t4fhh" (OuterVolumeSpecName: "kube-api-access-t4fhh") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "kube-api-access-t4fhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:36:46.660277 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.659935    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:36:46.660868 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.660775    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:36:46.661067 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.661025    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:36:46.663755 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.663726    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-config" (OuterVolumeSpecName: "config") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:36:46.669525 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.669419    2565 scope.go:117] "RemoveContainer" containerID="a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d"
Jul 16 22:36:46.672543 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.672519    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-web-config" (OuterVolumeSpecName: "web-config") pod "422deed3-a980-48b1-bbdd-21e309d6c0ed" (UID: "422deed3-a980-48b1-bbdd-21e309d6c0ed"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:36:46.681169 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.681140    2565 scope.go:117] "RemoveContainer" containerID="5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b"
Jul 16 22:36:46.688839 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.688817    2565 scope.go:117] "RemoveContainer" containerID="cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332"
Jul 16 22:36:46.696505 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.696482    2565 scope.go:117] "RemoveContainer" containerID="3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8"
Jul 16 22:36:46.703527 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.703506    2565 scope.go:117] "RemoveContainer" containerID="f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a"
Jul 16 22:36:46.703806 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:36:46.703780    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a\": container with ID starting with f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a not found: ID does not exist" containerID="f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a"
Jul 16 22:36:46.703884 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.703822    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a"} err="failed to get container status \"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a\": rpc error: code = NotFound desc = could not find container \"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a\": container with ID starting with f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a not found: ID does not exist"
Jul 16 22:36:46.703884 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.703872    2565 scope.go:117] "RemoveContainer" containerID="433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95"
Jul 16 22:36:46.704138 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:36:46.704118    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95\": container with ID starting with 433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95 not found: ID does not exist" containerID="433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95"
Jul 16 22:36:46.704180 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.704144    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95"} err="failed to get container status \"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95\": rpc error: code = NotFound desc = could not find container \"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95\": container with ID starting with 433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95 not found: ID does not exist"
Jul 16 22:36:46.704180 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.704160    2565 scope.go:117] "RemoveContainer" containerID="940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b"
Jul 16 22:36:46.704404 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:36:46.704386    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b\": container with ID starting with 940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b not found: ID does not exist" containerID="940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b"
Jul 16 22:36:46.704482 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.704411    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b"} err="failed to get container status \"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b\": rpc error: code = NotFound desc = could not find container \"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b\": container with ID starting with 940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b not found: ID does not exist"
Jul 16 22:36:46.704482 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.704431    2565 scope.go:117] "RemoveContainer" containerID="a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d"
Jul 16 22:36:46.704678 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:36:46.704659    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d\": container with ID starting with a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d not found: ID does not exist" containerID="a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d"
Jul 16 22:36:46.704716 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.704684    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d"} err="failed to get container status \"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d\": rpc error: code = NotFound desc = could not find container \"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d\": container with ID starting with a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d not found: ID does not exist"
Jul 16 22:36:46.704716 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.704699    2565 scope.go:117] "RemoveContainer" containerID="5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b"
Jul 16 22:36:46.704902 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:36:46.704885    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b\": container with ID starting with 5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b not found: ID does not exist" containerID="5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b"
Jul 16 22:36:46.704962 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.704908    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b"} err="failed to get container status \"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b\": rpc error: code = NotFound desc = could not find container \"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b\": container with ID starting with 5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b not found: ID does not exist"
Jul 16 22:36:46.704962 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.704926    2565 scope.go:117] "RemoveContainer" containerID="cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332"
Jul 16 22:36:46.705130 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:36:46.705108    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332\": container with ID starting with cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332 not found: ID does not exist" containerID="cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332"
Jul 16 22:36:46.705167 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.705133    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332"} err="failed to get container status \"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332\": rpc error: code = NotFound desc = could not find container \"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332\": container with ID starting with cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332 not found: ID does not exist"
Jul 16 22:36:46.705167 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.705145    2565 scope.go:117] "RemoveContainer" containerID="3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8"
Jul 16 22:36:46.705342 ip-10-0-139-23 kubenswrapper[2565]: E0716 22:36:46.705326    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8\": container with ID starting with 3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8 not found: ID does not exist" containerID="3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8"
Jul 16 22:36:46.705415 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.705348    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8"} err="failed to get container status \"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8\": rpc error: code = NotFound desc = could not find container \"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8\": container with ID starting with 3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8 not found: ID does not exist"
Jul 16 22:36:46.705415 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.705366    2565 scope.go:117] "RemoveContainer" containerID="f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a"
Jul 16 22:36:46.705589 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.705569    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a"} err="failed to get container status \"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a\": rpc error: code = NotFound desc = could not find container \"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a\": container with ID starting with f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a not found: ID does not exist"
Jul 16 22:36:46.705632 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.705591    2565 scope.go:117] "RemoveContainer" containerID="433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95"
Jul 16 22:36:46.705809 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.705785    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95"} err="failed to get container status \"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95\": rpc error: code = NotFound desc = could not find container \"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95\": container with ID starting with 433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95 not found: ID does not exist"
Jul 16 22:36:46.705852 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.705809    2565 scope.go:117] "RemoveContainer" containerID="940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b"
Jul 16 22:36:46.706026 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.706009    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b"} err="failed to get container status \"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b\": rpc error: code = NotFound desc = could not find container \"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b\": container with ID starting with 940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b not found: ID does not exist"
Jul 16 22:36:46.706079 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.706026    2565 scope.go:117] "RemoveContainer" containerID="a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d"
Jul 16 22:36:46.706199 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.706183    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d"} err="failed to get container status \"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d\": rpc error: code = NotFound desc = could not find container \"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d\": container with ID starting with a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d not found: ID does not exist"
Jul 16 22:36:46.706242 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.706200    2565 scope.go:117] "RemoveContainer" containerID="5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b"
Jul 16 22:36:46.706468 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.706449    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b"} err="failed to get container status \"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b\": rpc error: code = NotFound desc = could not find container \"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b\": container with ID starting with 5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b not found: ID does not exist"
Jul 16 22:36:46.706508 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.706469    2565 scope.go:117] "RemoveContainer" containerID="cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332"
Jul 16 22:36:46.706699 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.706679    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332"} err="failed to get container status \"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332\": rpc error: code = NotFound desc = could not find container \"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332\": container with ID starting with cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332 not found: ID does not exist"
Jul 16 22:36:46.706777 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.706699    2565 scope.go:117] "RemoveContainer" containerID="3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8"
Jul 16 22:36:46.706935 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.706917    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8"} err="failed to get container status \"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8\": rpc error: code = NotFound desc = could not find container \"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8\": container with ID starting with 3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8 not found: ID does not exist"
Jul 16 22:36:46.706986 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.706937    2565 scope.go:117] "RemoveContainer" containerID="f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a"
Jul 16 22:36:46.707171 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.707154    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a"} err="failed to get container status \"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a\": rpc error: code = NotFound desc = could not find container \"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a\": container with ID starting with f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a not found: ID does not exist"
Jul 16 22:36:46.707211 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.707171    2565 scope.go:117] "RemoveContainer" containerID="433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95"
Jul 16 22:36:46.707406 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.707387    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95"} err="failed to get container status \"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95\": rpc error: code = NotFound desc = could not find container \"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95\": container with ID starting with 433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95 not found: ID does not exist"
Jul 16 22:36:46.707453 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.707408    2565 scope.go:117] "RemoveContainer" containerID="940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b"
Jul 16 22:36:46.707612 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.707591    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b"} err="failed to get container status \"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b\": rpc error: code = NotFound desc = could not find container \"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b\": container with ID starting with 940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b not found: ID does not exist"
Jul 16 22:36:46.707659 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.707613    2565 scope.go:117] "RemoveContainer" containerID="a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d"
Jul 16 22:36:46.707829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.707811    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d"} err="failed to get container status \"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d\": rpc error: code = NotFound desc = could not find container \"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d\": container with ID starting with a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d not found: ID does not exist"
Jul 16 22:36:46.707894 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.707830    2565 scope.go:117] "RemoveContainer" containerID="5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b"
Jul 16 22:36:46.708022 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.708007    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b"} err="failed to get container status \"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b\": rpc error: code = NotFound desc = could not find container \"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b\": container with ID starting with 5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b not found: ID does not exist"
Jul 16 22:36:46.708069 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.708022    2565 scope.go:117] "RemoveContainer" containerID="cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332"
Jul 16 22:36:46.708199 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.708181    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332"} err="failed to get container status \"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332\": rpc error: code = NotFound desc = could not find container \"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332\": container with ID starting with cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332 not found: ID does not exist"
Jul 16 22:36:46.708282 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.708199    2565 scope.go:117] "RemoveContainer" containerID="3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8"
Jul 16 22:36:46.708438 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.708420    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8"} err="failed to get container status \"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8\": rpc error: code = NotFound desc = could not find container \"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8\": container with ID starting with 3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8 not found: ID does not exist"
Jul 16 22:36:46.708474 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.708438    2565 scope.go:117] "RemoveContainer" containerID="f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a"
Jul 16 22:36:46.708682 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.708667    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a"} err="failed to get container status \"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a\": rpc error: code = NotFound desc = could not find container \"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a\": container with ID starting with f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a not found: ID does not exist"
Jul 16 22:36:46.708740 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.708682    2565 scope.go:117] "RemoveContainer" containerID="433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95"
Jul 16 22:36:46.708886 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.708869    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95"} err="failed to get container status \"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95\": rpc error: code = NotFound desc = could not find container \"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95\": container with ID starting with 433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95 not found: ID does not exist"
Jul 16 22:36:46.708939 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.708887    2565 scope.go:117] "RemoveContainer" containerID="940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b"
Jul 16 22:36:46.709092 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.709074    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b"} err="failed to get container status \"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b\": rpc error: code = NotFound desc = could not find container \"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b\": container with ID starting with 940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b not found: ID does not exist"
Jul 16 22:36:46.709190 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.709094    2565 scope.go:117] "RemoveContainer" containerID="a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d"
Jul 16 22:36:46.709351 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.709333    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d"} err="failed to get container status \"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d\": rpc error: code = NotFound desc = could not find container \"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d\": container with ID starting with a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d not found: ID does not exist"
Jul 16 22:36:46.709351 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.709350    2565 scope.go:117] "RemoveContainer" containerID="5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b"
Jul 16 22:36:46.709578 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.709559    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b"} err="failed to get container status \"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b\": rpc error: code = NotFound desc = could not find container \"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b\": container with ID starting with 5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b not found: ID does not exist"
Jul 16 22:36:46.709578 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.709577    2565 scope.go:117] "RemoveContainer" containerID="cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332"
Jul 16 22:36:46.709823 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.709802    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332"} err="failed to get container status \"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332\": rpc error: code = NotFound desc = could not find container \"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332\": container with ID starting with cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332 not found: ID does not exist"
Jul 16 22:36:46.709876 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.709824    2565 scope.go:117] "RemoveContainer" containerID="3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8"
Jul 16 22:36:46.710040 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.710022    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8"} err="failed to get container status \"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8\": rpc error: code = NotFound desc = could not find container \"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8\": container with ID starting with 3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8 not found: ID does not exist"
Jul 16 22:36:46.710085 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.710041    2565 scope.go:117] "RemoveContainer" containerID="f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a"
Jul 16 22:36:46.710217 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.710198    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a"} err="failed to get container status \"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a\": rpc error: code = NotFound desc = could not find container \"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a\": container with ID starting with f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a not found: ID does not exist"
Jul 16 22:36:46.710333 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.710218    2565 scope.go:117] "RemoveContainer" containerID="433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95"
Jul 16 22:36:46.710452 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.710433    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95"} err="failed to get container status \"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95\": rpc error: code = NotFound desc = could not find container \"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95\": container with ID starting with 433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95 not found: ID does not exist"
Jul 16 22:36:46.710505 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.710452    2565 scope.go:117] "RemoveContainer" containerID="940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b"
Jul 16 22:36:46.710669 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.710650    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b"} err="failed to get container status \"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b\": rpc error: code = NotFound desc = could not find container \"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b\": container with ID starting with 940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b not found: ID does not exist"
Jul 16 22:36:46.710712 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.710670    2565 scope.go:117] "RemoveContainer" containerID="a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d"
Jul 16 22:36:46.710879 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.710859    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d"} err="failed to get container status \"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d\": rpc error: code = NotFound desc = could not find container \"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d\": container with ID starting with a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d not found: ID does not exist"
Jul 16 22:36:46.710926 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.710879    2565 scope.go:117] "RemoveContainer" containerID="5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b"
Jul 16 22:36:46.711106 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.711088    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b"} err="failed to get container status \"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b\": rpc error: code = NotFound desc = could not find container \"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b\": container with ID starting with 5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b not found: ID does not exist"
Jul 16 22:36:46.711151 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.711108    2565 scope.go:117] "RemoveContainer" containerID="cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332"
Jul 16 22:36:46.711314 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.711294    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332"} err="failed to get container status \"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332\": rpc error: code = NotFound desc = could not find container \"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332\": container with ID starting with cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332 not found: ID does not exist"
Jul 16 22:36:46.711363 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.711315    2565 scope.go:117] "RemoveContainer" containerID="3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8"
Jul 16 22:36:46.711517 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.711499    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8"} err="failed to get container status \"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8\": rpc error: code = NotFound desc = could not find container \"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8\": container with ID starting with 3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8 not found: ID does not exist"
Jul 16 22:36:46.711558 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.711517    2565 scope.go:117] "RemoveContainer" containerID="f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a"
Jul 16 22:36:46.711721 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.711705    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a"} err="failed to get container status \"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a\": rpc error: code = NotFound desc = could not find container \"f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a\": container with ID starting with f7e1c454a5642f5541ef11e4dddb6a6a923ab4717b0300bd4843ba3d5905264a not found: ID does not exist"
Jul 16 22:36:46.711766 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.711721    2565 scope.go:117] "RemoveContainer" containerID="433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95"
Jul 16 22:36:46.711915 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.711898    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95"} err="failed to get container status \"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95\": rpc error: code = NotFound desc = could not find container \"433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95\": container with ID starting with 433dbf6ddbce5aea7ab28a768c31251efc3f7249301a18179f2eeaef2d88ec95 not found: ID does not exist"
Jul 16 22:36:46.711956 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.711915    2565 scope.go:117] "RemoveContainer" containerID="940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b"
Jul 16 22:36:46.712119 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.712102    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b"} err="failed to get container status \"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b\": rpc error: code = NotFound desc = could not find container \"940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b\": container with ID starting with 940df03b576c4b5adc6e8df8ab9947675a3695fd2946da7c9d820ce223be187b not found: ID does not exist"
Jul 16 22:36:46.712171 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.712119    2565 scope.go:117] "RemoveContainer" containerID="a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d"
Jul 16 22:36:46.712359 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.712343    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d"} err="failed to get container status \"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d\": rpc error: code = NotFound desc = could not find container \"a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d\": container with ID starting with a6d64923cca3bfe6bc7e7c0db6fe79288accbc623446796b346725aa5e109e7d not found: ID does not exist"
Jul 16 22:36:46.712414 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.712359    2565 scope.go:117] "RemoveContainer" containerID="5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b"
Jul 16 22:36:46.712571 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.712554    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b"} err="failed to get container status \"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b\": rpc error: code = NotFound desc = could not find container \"5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b\": container with ID starting with 5959d1cea5cdb8ad0339d4c259f47119e9b2163a1b9962ad9c72d96d85436f8b not found: ID does not exist"
Jul 16 22:36:46.712616 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.712574    2565 scope.go:117] "RemoveContainer" containerID="cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332"
Jul 16 22:36:46.712781 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.712765    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332"} err="failed to get container status \"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332\": rpc error: code = NotFound desc = could not find container \"cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332\": container with ID starting with cd6e98491f115b56f4097688da16b84fe62fae7504e43e1d87992c0ce7e25332 not found: ID does not exist"
Jul 16 22:36:46.712841 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.712780    2565 scope.go:117] "RemoveContainer" containerID="3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8"
Jul 16 22:36:46.712958 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.712942    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8"} err="failed to get container status \"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8\": rpc error: code = NotFound desc = could not find container \"3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8\": container with ID starting with 3db0209c66b8b19a237a3655bf329e6d74310b2f8ea5365917eb8a4a25d823a8 not found: ID does not exist"
Jul 16 22:36:46.753562 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753528    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-configmap-metrics-client-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753562 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753560    2565 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-grpc-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753562 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753570    2565 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-metrics-client-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753776 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753580    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753776 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753592    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753776 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753601    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753776 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753610    2565 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-kube-rbac-proxy\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753776 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753619    2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-web-config\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753776 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753627    2565 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-config\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753776 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753634    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753776 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753642    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-db\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753776 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753650    2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/422deed3-a980-48b1-bbdd-21e309d6c0ed-tls-assets\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753776 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753658    2565 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753776 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753666    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753776 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753674    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753776 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753682    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753776 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753690    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/422deed3-a980-48b1-bbdd-21e309d6c0ed-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753776 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753698    2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/422deed3-a980-48b1-bbdd-21e309d6c0ed-config-out\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753776 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753706    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4fhh\" (UniqueName: \"kubernetes.io/projected/422deed3-a980-48b1-bbdd-21e309d6c0ed-kube-api-access-t4fhh\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.753776 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.753714    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/422deed3-a980-48b1-bbdd-21e309d6c0ed-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 22:36:46.968557 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.968463    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 22:36:46.970275 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:46.970233    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 22:36:47.001488 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001452    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 22:36:47.001729 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001716    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="init-config-reloader"
Jul 16 22:36:47.001781 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001731    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="init-config-reloader"
Jul 16 22:36:47.001781 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001738    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="config-reloader"
Jul 16 22:36:47.001781 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001744    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="config-reloader"
Jul 16 22:36:47.001781 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001756    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="kube-rbac-proxy"
Jul 16 22:36:47.001781 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001761    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="kube-rbac-proxy"
Jul 16 22:36:47.001781 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001767    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="kube-rbac-proxy-thanos"
Jul 16 22:36:47.001781 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001772    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="kube-rbac-proxy-thanos"
Jul 16 22:36:47.001781 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001779    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="prometheus"
Jul 16 22:36:47.001781 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001784    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="prometheus"
Jul 16 22:36:47.002026 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001795    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="thanos-sidecar"
Jul 16 22:36:47.002026 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001799    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="thanos-sidecar"
Jul 16 22:36:47.002026 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001807    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="kube-rbac-proxy-web"
Jul 16 22:36:47.002026 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001813    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="kube-rbac-proxy-web"
Jul 16 22:36:47.002026 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001854    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="kube-rbac-proxy-web"
Jul 16 22:36:47.002026 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001861    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="thanos-sidecar"
Jul 16 22:36:47.002026 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001869    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="kube-rbac-proxy-thanos"
Jul 16 22:36:47.002026 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001874    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="config-reloader"
Jul 16 22:36:47.002026 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001879    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="prometheus"
Jul 16 22:36:47.002026 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.001884    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" containerName="kube-rbac-proxy"
Jul 16 22:36:47.005339 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.005316    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.008275 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.008237    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 16 22:36:47.008275 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.008270    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 16 22:36:47.010761 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.010733    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 16 22:36:47.010899 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.010795    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dtms6\""
Jul 16 22:36:47.010899 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.010883    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-99t6n7vk14vv0\""
Jul 16 22:36:47.011027 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.010928    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 16 22:36:47.012271 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.011382    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 16 22:36:47.012271 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.011416    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 16 22:36:47.012271 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.011439    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 16 22:36:47.012271 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.011463    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 16 22:36:47.012271 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.011416    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 16 22:36:47.012271 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.011568    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 16 22:36:47.012271 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.011779    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 16 22:36:47.012271 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.011936    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 16 22:36:47.012271 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.012266    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 16 22:36:47.016490 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.016468    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 16 22:36:47.018489 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.018471    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 16 22:36:47.035604 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.035572    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 22:36:47.156232 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156197    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156232 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156455 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156283    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156455 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156317    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156455 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156343    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156455 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156398    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/911afef5-abb0-4869-b331-aed0f5420d2b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156455 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156416    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156455 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156447    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156638 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-config\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156638 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156524    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/911afef5-abb0-4869-b331-aed0f5420d2b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156638 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156552    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156638 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156572    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lmc7\" (UniqueName: \"kubernetes.io/projected/911afef5-abb0-4869-b331-aed0f5420d2b-kube-api-access-9lmc7\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156638 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156595    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-web-config\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156815 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156652    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156815 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156681    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156815 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156698    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156815 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156718    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156815 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156743    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156815 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156759    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.156815 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.156780    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/911afef5-abb0-4869-b331-aed0f5420d2b-config-out\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.257341 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257228    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.257341 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257284    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.257341 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257308    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.257341 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.257341 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257346    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.257693 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/911afef5-abb0-4869-b331-aed0f5420d2b-config-out\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.257693 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257390    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.257693 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257514    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.257693 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257552    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.257693 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.257693 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257622    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.257693 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/911afef5-abb0-4869-b331-aed0f5420d2b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.257693 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257679    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.258079 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257713    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.258079 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-config\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.258079 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257764    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/911afef5-abb0-4869-b331-aed0f5420d2b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.258079 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257794    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.258079 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257824    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lmc7\" (UniqueName: \"kubernetes.io/projected/911afef5-abb0-4869-b331-aed0f5420d2b-kube-api-access-9lmc7\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.258079 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257864    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-web-config\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.258079 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.257905    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.258447 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.258147    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.258447 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.258176    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.258560 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.258461    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.259462 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.259414    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/911afef5-abb0-4869-b331-aed0f5420d2b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.260490 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.260424    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.261966 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.260964    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/911afef5-abb0-4869-b331-aed0f5420d2b-config-out\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.261966 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.261028    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.261966 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.261524    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.261966 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.261578    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.261966 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.261666    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.261966 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.261785    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.261966 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.261927    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.262386 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.262011    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/911afef5-abb0-4869-b331-aed0f5420d2b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.262463 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.262442    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-config\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.262665 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.262644    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.262784 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.262760    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-web-config\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.262852 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.262837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.263509 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.263490    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/911afef5-abb0-4869-b331-aed0f5420d2b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.263735 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.263718    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/911afef5-abb0-4869-b331-aed0f5420d2b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.271509 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.271488    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lmc7\" (UniqueName: \"kubernetes.io/projected/911afef5-abb0-4869-b331-aed0f5420d2b-kube-api-access-9lmc7\") pod \"prometheus-k8s-0\" (UID: \"911afef5-abb0-4869-b331-aed0f5420d2b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.315625 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.315587    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:36:47.469764 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.469729    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 22:36:47.472998 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:36:47.472970    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911afef5_abb0_4869_b331_aed0f5420d2b.slice/crio-eb1ec43f8604ae2bd9b26b79cabd7f664589cd07708cfbd628612baac5f3e4aa WatchSource:0}: Error finding container eb1ec43f8604ae2bd9b26b79cabd7f664589cd07708cfbd628612baac5f3e4aa: Status 404 returned error can't find the container with id eb1ec43f8604ae2bd9b26b79cabd7f664589cd07708cfbd628612baac5f3e4aa
Jul 16 22:36:47.644514 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.644477    2565 generic.go:356] "Generic (PLEG): container finished" podID="911afef5-abb0-4869-b331-aed0f5420d2b" containerID="3e99445af51da872d6a84b721dba894d3b7342d5198a3fa3076f006bc983a638" exitCode=0
Jul 16 22:36:47.644683 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.644560    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"911afef5-abb0-4869-b331-aed0f5420d2b","Type":"ContainerDied","Data":"3e99445af51da872d6a84b721dba894d3b7342d5198a3fa3076f006bc983a638"}
Jul 16 22:36:47.644683 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.644634    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"911afef5-abb0-4869-b331-aed0f5420d2b","Type":"ContainerStarted","Data":"eb1ec43f8604ae2bd9b26b79cabd7f664589cd07708cfbd628612baac5f3e4aa"}
Jul 16 22:36:47.726309 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:47.726274    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="422deed3-a980-48b1-bbdd-21e309d6c0ed" path="/var/lib/kubelet/pods/422deed3-a980-48b1-bbdd-21e309d6c0ed/volumes"
Jul 16 22:36:48.654690 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:48.654604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"911afef5-abb0-4869-b331-aed0f5420d2b","Type":"ContainerStarted","Data":"23e3963638f0cc95a459d26e4b6a74cd9530354f5d49bfb9253b87c782b007fe"}
Jul 16 22:36:48.654690 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:48.654673    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"911afef5-abb0-4869-b331-aed0f5420d2b","Type":"ContainerStarted","Data":"be99f2ca3dacaffa8ba1ef6a24f54b2facff3b0c6eedb480cf8fcf89f23acbe0"}
Jul 16 22:36:48.655066 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:48.654698    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"911afef5-abb0-4869-b331-aed0f5420d2b","Type":"ContainerStarted","Data":"256da9b5cdfa79a1f11839c7d073b4c292eab6644dacd65a733eb120519e1ada"}
Jul 16 22:36:48.655066 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:48.654733    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"911afef5-abb0-4869-b331-aed0f5420d2b","Type":"ContainerStarted","Data":"62d5755930277604f7f7ddf9dbe7a9ea85df5ad936ee8f063fb234a582597436"}
Jul 16 22:36:48.655066 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:48.654755    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"911afef5-abb0-4869-b331-aed0f5420d2b","Type":"ContainerStarted","Data":"1e18c7bc88324995ce137e251c0e6f009a4986e0f3bc630029d5fa3f98dce943"}
Jul 16 22:36:48.655066 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:48.654778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"911afef5-abb0-4869-b331-aed0f5420d2b","Type":"ContainerStarted","Data":"46ca3eae7be7b4feeb83f759a607c1cff676de975972cc6e24a635752a4fd4bf"}
Jul 16 22:36:48.687792 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:48.687734    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.6697201980000003 podStartE2EDuration="2.687717352s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.997154ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:36:48.686041111 +0000 UTC m=+219.434525182" watchObservedRunningTime="2026-07-16 22:36:48.687717352 +0000 UTC m=+219.436201422"
Jul 16 22:36:52.315967 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:36:52.315929    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:37:47.316906 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:37:47.316815    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:37:47.332826 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:37:47.332803    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:37:47.895677 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:37:47.895651    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:38:09.641603 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:38:09.641574    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 22:39:25.116941 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:25.116861    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nwbps"]
Jul 16 22:39:25.119671 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:25.119650    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-nwbps"
Jul 16 22:39:25.124892 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:25.124874    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 16 22:39:25.124990 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:25.124878    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 16 22:39:25.125751 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:25.125565    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-dvq6d\""
Jul 16 22:39:25.132043 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:25.132023    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nwbps"]
Jul 16 22:39:25.209891 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:25.209847    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b854dbc9-f92f-40f0-acff-e435d622961c-bound-sa-token\") pod \"cert-manager-55b4d954d5-nwbps\" (UID: \"b854dbc9-f92f-40f0-acff-e435d622961c\") " pod="cert-manager/cert-manager-55b4d954d5-nwbps"
Jul 16 22:39:25.210072 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:25.209930    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5npd\" (UniqueName: \"kubernetes.io/projected/b854dbc9-f92f-40f0-acff-e435d622961c-kube-api-access-g5npd\") pod \"cert-manager-55b4d954d5-nwbps\" (UID: \"b854dbc9-f92f-40f0-acff-e435d622961c\") " pod="cert-manager/cert-manager-55b4d954d5-nwbps"
Jul 16 22:39:25.310999 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:25.310959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5npd\" (UniqueName: \"kubernetes.io/projected/b854dbc9-f92f-40f0-acff-e435d622961c-kube-api-access-g5npd\") pod \"cert-manager-55b4d954d5-nwbps\" (UID: \"b854dbc9-f92f-40f0-acff-e435d622961c\") " pod="cert-manager/cert-manager-55b4d954d5-nwbps"
Jul 16 22:39:25.311180 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:25.311017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b854dbc9-f92f-40f0-acff-e435d622961c-bound-sa-token\") pod \"cert-manager-55b4d954d5-nwbps\" (UID: \"b854dbc9-f92f-40f0-acff-e435d622961c\") " pod="cert-manager/cert-manager-55b4d954d5-nwbps"
Jul 16 22:39:25.320210 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:25.320173    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b854dbc9-f92f-40f0-acff-e435d622961c-bound-sa-token\") pod \"cert-manager-55b4d954d5-nwbps\" (UID: \"b854dbc9-f92f-40f0-acff-e435d622961c\") " pod="cert-manager/cert-manager-55b4d954d5-nwbps"
Jul 16 22:39:25.320385 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:25.320337    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5npd\" (UniqueName: \"kubernetes.io/projected/b854dbc9-f92f-40f0-acff-e435d622961c-kube-api-access-g5npd\") pod \"cert-manager-55b4d954d5-nwbps\" (UID: \"b854dbc9-f92f-40f0-acff-e435d622961c\") " pod="cert-manager/cert-manager-55b4d954d5-nwbps"
Jul 16 22:39:25.438425 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:25.438340    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-nwbps"
Jul 16 22:39:25.570841 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:25.570809    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nwbps"]
Jul 16 22:39:25.574017 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:39:25.573990    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb854dbc9_f92f_40f0_acff_e435d622961c.slice/crio-3d93cde148d4de255ac8523fe30c08bd378b42bbfc361a599d2a96ec57bd4053 WatchSource:0}: Error finding container 3d93cde148d4de255ac8523fe30c08bd378b42bbfc361a599d2a96ec57bd4053: Status 404 returned error can't find the container with id 3d93cde148d4de255ac8523fe30c08bd378b42bbfc361a599d2a96ec57bd4053
Jul 16 22:39:25.576008 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:25.575989    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:39:26.244210 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:26.244171    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-nwbps" event={"ID":"b854dbc9-f92f-40f0-acff-e435d622961c","Type":"ContainerStarted","Data":"3d93cde148d4de255ac8523fe30c08bd378b42bbfc361a599d2a96ec57bd4053"}
Jul 16 22:39:29.258474 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:29.258433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-nwbps" event={"ID":"b854dbc9-f92f-40f0-acff-e435d622961c","Type":"ContainerStarted","Data":"a6af4409cd3e4f06fc4fc42bb0eccb7bdd750470b754c60c2d0dbcf0c2660f39"}
Jul 16 22:39:29.285898 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:29.285849    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-nwbps" podStartSLOduration=1.18093512 podStartE2EDuration="4.285834906s" totalImagesPullingTime="3.104899786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:39:29.285102123 +0000 UTC m=+380.033586195" watchObservedRunningTime="2026-07-16 22:39:29.285834906 +0000 UTC m=+380.034318976"
Jul 16 22:39:47.988543 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:47.988509    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"]
Jul 16 22:39:47.991714 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:47.991699    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:39:48.001301 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.001282    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 16 22:39:48.001398 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.001383    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 16 22:39:48.002115 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.002097    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 16 22:39:48.002210 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.002100    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 16 22:39:48.002210 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.002151    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xs9sp\""
Jul 16 22:39:48.031555 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.031523    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"]
Jul 16 22:39:48.099713 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.099682    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32bfa590-021b-49b5-a94a-10661dd15166-tmp\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv\" (UID: \"32bfa590-021b-49b5-a94a-10661dd15166\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:39:48.099870 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.099718    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32bfa590-021b-49b5-a94a-10661dd15166-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv\" (UID: \"32bfa590-021b-49b5-a94a-10661dd15166\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:39:48.099870 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.099741    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx5jj\" (UniqueName: \"kubernetes.io/projected/32bfa590-021b-49b5-a94a-10661dd15166-kube-api-access-cx5jj\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv\" (UID: \"32bfa590-021b-49b5-a94a-10661dd15166\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:39:48.099870 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.099812    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/32bfa590-021b-49b5-a94a-10661dd15166-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv\" (UID: \"32bfa590-021b-49b5-a94a-10661dd15166\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:39:48.099968 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.099874    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/32bfa590-021b-49b5-a94a-10661dd15166-manifests\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv\" (UID: \"32bfa590-021b-49b5-a94a-10661dd15166\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:39:48.200998 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.200957    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32bfa590-021b-49b5-a94a-10661dd15166-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv\" (UID: \"32bfa590-021b-49b5-a94a-10661dd15166\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:39:48.200998 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.200997    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx5jj\" (UniqueName: \"kubernetes.io/projected/32bfa590-021b-49b5-a94a-10661dd15166-kube-api-access-cx5jj\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv\" (UID: \"32bfa590-021b-49b5-a94a-10661dd15166\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:39:48.201196 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.201025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/32bfa590-021b-49b5-a94a-10661dd15166-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv\" (UID: \"32bfa590-021b-49b5-a94a-10661dd15166\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:39:48.201196 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.201080    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/32bfa590-021b-49b5-a94a-10661dd15166-manifests\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv\" (UID: \"32bfa590-021b-49b5-a94a-10661dd15166\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:39:48.201196 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.201145    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32bfa590-021b-49b5-a94a-10661dd15166-tmp\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv\" (UID: \"32bfa590-021b-49b5-a94a-10661dd15166\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:39:48.201559 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.201530    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/32bfa590-021b-49b5-a94a-10661dd15166-manifests\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv\" (UID: \"32bfa590-021b-49b5-a94a-10661dd15166\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:39:48.203342 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.203304    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32bfa590-021b-49b5-a94a-10661dd15166-tmp\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv\" (UID: \"32bfa590-021b-49b5-a94a-10661dd15166\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:39:48.203500 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.203481    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32bfa590-021b-49b5-a94a-10661dd15166-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv\" (UID: \"32bfa590-021b-49b5-a94a-10661dd15166\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:39:48.203640 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.203621    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/32bfa590-021b-49b5-a94a-10661dd15166-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv\" (UID: \"32bfa590-021b-49b5-a94a-10661dd15166\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:39:48.208943 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.208921    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx5jj\" (UniqueName: \"kubernetes.io/projected/32bfa590-021b-49b5-a94a-10661dd15166-kube-api-access-cx5jj\") pod \"opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv\" (UID: \"32bfa590-021b-49b5-a94a-10661dd15166\") " pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:39:48.300934 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.300851    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:39:48.442662 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:48.442508    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"]
Jul 16 22:39:48.445501 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:39:48.445473    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32bfa590_021b_49b5_a94a_10661dd15166.slice/crio-e145cc033d3017dbb46ebbf74b5a416c8a094464589ede31dc05bf0d9c80f5d5 WatchSource:0}: Error finding container e145cc033d3017dbb46ebbf74b5a416c8a094464589ede31dc05bf0d9c80f5d5: Status 404 returned error can't find the container with id e145cc033d3017dbb46ebbf74b5a416c8a094464589ede31dc05bf0d9c80f5d5
Jul 16 22:39:49.337760 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:49.337727    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv" event={"ID":"32bfa590-021b-49b5-a94a-10661dd15166","Type":"ContainerStarted","Data":"e145cc033d3017dbb46ebbf74b5a416c8a094464589ede31dc05bf0d9c80f5d5"}
Jul 16 22:39:52.351502 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:52.351463    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv" event={"ID":"32bfa590-021b-49b5-a94a-10661dd15166","Type":"ContainerStarted","Data":"b448db16a4515d627bb8a3f8aaa95571d07b217faaa4ea31e532c2808f18f7ab"}
Jul 16 22:39:53.356913 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:53.356872    2565 generic.go:356] "Generic (PLEG): container finished" podID="32bfa590-021b-49b5-a94a-10661dd15166" containerID="b448db16a4515d627bb8a3f8aaa95571d07b217faaa4ea31e532c2808f18f7ab" exitCode=0
Jul 16 22:39:53.357420 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:53.356962    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv" event={"ID":"32bfa590-021b-49b5-a94a-10661dd15166","Type":"ContainerDied","Data":"b448db16a4515d627bb8a3f8aaa95571d07b217faaa4ea31e532c2808f18f7ab"}
Jul 16 22:39:54.363977 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:54.363940    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv" event={"ID":"32bfa590-021b-49b5-a94a-10661dd15166","Type":"ContainerStarted","Data":"ec6c25ae137f3331106481c861d29b0e0adebeccdac3c591e838b98f0e3db23d"}
Jul 16 22:39:54.364465 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:39:54.364079    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:40:05.369951 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:05.369918    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv"
Jul 16 22:40:05.393389 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:05.393338    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv" podStartSLOduration=13.98605092 podStartE2EDuration="18.393322634s" totalImagesPullingTime="3.312766697s" totalInitContainerRuntime="1.094505017s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:39:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:39:54.394572432 +0000 UTC m=+405.143056503" watchObservedRunningTime="2026-07-16 22:40:05.393322634 +0000 UTC m=+416.141806704"
Jul 16 22:40:27.218151 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.218114    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"]
Jul 16 22:40:27.225882 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.225857    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"
Jul 16 22:40:27.228802 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.228775    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 16 22:40:27.229331 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.229312    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 16 22:40:27.229419 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.229332    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 16 22:40:27.229419 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.229336    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 16 22:40:27.229723 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.229699    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 16 22:40:27.229814 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.229738    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-sj7rv\""
Jul 16 22:40:27.239201 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.239179    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"]
Jul 16 22:40:27.304425 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.304385    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d85de3f9-6def-4ec6-afb3-b5ca6415b331-cert\") pod \"lws-controller-manager-84c74b69b8-p9m6n\" (UID: \"d85de3f9-6def-4ec6-afb3-b5ca6415b331\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"
Jul 16 22:40:27.304615 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.304430    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d85de3f9-6def-4ec6-afb3-b5ca6415b331-metrics-cert\") pod \"lws-controller-manager-84c74b69b8-p9m6n\" (UID: \"d85de3f9-6def-4ec6-afb3-b5ca6415b331\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"
Jul 16 22:40:27.304615 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.304523    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc2tq\" (UniqueName: \"kubernetes.io/projected/d85de3f9-6def-4ec6-afb3-b5ca6415b331-kube-api-access-hc2tq\") pod \"lws-controller-manager-84c74b69b8-p9m6n\" (UID: \"d85de3f9-6def-4ec6-afb3-b5ca6415b331\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"
Jul 16 22:40:27.304615 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.304566    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d85de3f9-6def-4ec6-afb3-b5ca6415b331-manager-config\") pod \"lws-controller-manager-84c74b69b8-p9m6n\" (UID: \"d85de3f9-6def-4ec6-afb3-b5ca6415b331\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"
Jul 16 22:40:27.405012 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.404973    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc2tq\" (UniqueName: \"kubernetes.io/projected/d85de3f9-6def-4ec6-afb3-b5ca6415b331-kube-api-access-hc2tq\") pod \"lws-controller-manager-84c74b69b8-p9m6n\" (UID: \"d85de3f9-6def-4ec6-afb3-b5ca6415b331\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"
Jul 16 22:40:27.405201 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.405037    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d85de3f9-6def-4ec6-afb3-b5ca6415b331-manager-config\") pod \"lws-controller-manager-84c74b69b8-p9m6n\" (UID: \"d85de3f9-6def-4ec6-afb3-b5ca6415b331\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"
Jul 16 22:40:27.405201 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.405086    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d85de3f9-6def-4ec6-afb3-b5ca6415b331-cert\") pod \"lws-controller-manager-84c74b69b8-p9m6n\" (UID: \"d85de3f9-6def-4ec6-afb3-b5ca6415b331\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"
Jul 16 22:40:27.405201 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.405115    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d85de3f9-6def-4ec6-afb3-b5ca6415b331-metrics-cert\") pod \"lws-controller-manager-84c74b69b8-p9m6n\" (UID: \"d85de3f9-6def-4ec6-afb3-b5ca6415b331\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"
Jul 16 22:40:27.405785 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.405755    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d85de3f9-6def-4ec6-afb3-b5ca6415b331-manager-config\") pod \"lws-controller-manager-84c74b69b8-p9m6n\" (UID: \"d85de3f9-6def-4ec6-afb3-b5ca6415b331\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"
Jul 16 22:40:27.407545 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.407522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d85de3f9-6def-4ec6-afb3-b5ca6415b331-cert\") pod \"lws-controller-manager-84c74b69b8-p9m6n\" (UID: \"d85de3f9-6def-4ec6-afb3-b5ca6415b331\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"
Jul 16 22:40:27.407731 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.407708    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d85de3f9-6def-4ec6-afb3-b5ca6415b331-metrics-cert\") pod \"lws-controller-manager-84c74b69b8-p9m6n\" (UID: \"d85de3f9-6def-4ec6-afb3-b5ca6415b331\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"
Jul 16 22:40:27.415149 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.415124    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc2tq\" (UniqueName: \"kubernetes.io/projected/d85de3f9-6def-4ec6-afb3-b5ca6415b331-kube-api-access-hc2tq\") pod \"lws-controller-manager-84c74b69b8-p9m6n\" (UID: \"d85de3f9-6def-4ec6-afb3-b5ca6415b331\") " pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"
Jul 16 22:40:27.535156 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.535092    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"
Jul 16 22:40:27.726728 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:27.726695    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"]
Jul 16 22:40:27.730003 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:40:27.729976    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85de3f9_6def_4ec6_afb3_b5ca6415b331.slice/crio-58076373962e35af4e974a279e54b99151bb7c69adbfbbf7b7987aa0793dce49 WatchSource:0}: Error finding container 58076373962e35af4e974a279e54b99151bb7c69adbfbbf7b7987aa0793dce49: Status 404 returned error can't find the container with id 58076373962e35af4e974a279e54b99151bb7c69adbfbbf7b7987aa0793dce49
Jul 16 22:40:28.506618 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:28.506577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n" event={"ID":"d85de3f9-6def-4ec6-afb3-b5ca6415b331","Type":"ContainerStarted","Data":"58076373962e35af4e974a279e54b99151bb7c69adbfbbf7b7987aa0793dce49"}
Jul 16 22:40:29.549265 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.548895    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"]
Jul 16 22:40:29.560502 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.560474    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.563782 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.563434    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-dpbrw\""
Jul 16 22:40:29.563980 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.563872    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 16 22:40:29.566862 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.566403    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 16 22:40:29.566862 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.566748    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 16 22:40:29.580058 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.580030    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"]
Jul 16 22:40:29.625429 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.625396    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.625613 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.625440    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.625744 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.625713    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.625882 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.625780    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.625882 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.625826    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.625882 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.625866    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.626037 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.625945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.626037 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.626005    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhdn7\" (UniqueName: \"kubernetes.io/projected/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-kube-api-access-hhdn7\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.626127 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.626041    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.727122 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.726506    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.727122 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.726568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.727122 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.726608    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.727122 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.726637    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.727122 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.726676    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhdn7\" (UniqueName: \"kubernetes.io/projected/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-kube-api-access-hhdn7\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.727122 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.726710    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.727122 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.726772    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.727122 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.726794    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.727122 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.726824    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.727122 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.726935    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.727725 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.727231    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.727725 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.727437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.727725 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.727505    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.727725 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.727564    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.729691 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.729626    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.729829 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.729730    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.746528 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.746503    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.749325 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.749298    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhdn7\" (UniqueName: \"kubernetes.io/projected/62707e5b-445c-45ac-8d6a-3ef554d6e4b7-kube-api-access-hhdn7\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4n4xlt\" (UID: \"62707e5b-445c-45ac-8d6a-3ef554d6e4b7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:29.880539 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:29.880500    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:30.519928 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.519893    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n" event={"ID":"d85de3f9-6def-4ec6-afb3-b5ca6415b331","Type":"ContainerStarted","Data":"e50699ae39d1955a20bd9c019f82b7aa2d709bc8003f10bad6ebb427000f0bef"}
Jul 16 22:40:30.520102 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.519980    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"
Jul 16 22:40:30.531936 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.531909    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"]
Jul 16 22:40:30.534670 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:40:30.534647    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62707e5b_445c_45ac_8d6a_3ef554d6e4b7.slice/crio-3e56d01063d6a9f18d6f33ce7f0ab56691438de9c1c7903ee291a5421940ceeb WatchSource:0}: Error finding container 3e56d01063d6a9f18d6f33ce7f0ab56691438de9c1c7903ee291a5421940ceeb: Status 404 returned error can't find the container with id 3e56d01063d6a9f18d6f33ce7f0ab56691438de9c1c7903ee291a5421940ceeb
Jul 16 22:40:30.552820 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.552790    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"]
Jul 16 22:40:30.556218 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.556199    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.570352 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.570329    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 16 22:40:30.570460 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.570423    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 16 22:40:30.572644 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.572630    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 16 22:40:30.581343 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.581303    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jnscn\""
Jul 16 22:40:30.583667 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.583614    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n" podStartSLOduration=0.92786505 podStartE2EDuration="3.583601299s" totalImagesPullingTime="2.655736249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:30.583142158 +0000 UTC m=+441.331626230" watchObservedRunningTime="2026-07-16 22:40:30.583601299 +0000 UTC m=+441.332085370"
Jul 16 22:40:30.587331 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.587306    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"]
Jul 16 22:40:30.635062 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.635036    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4151c7c9-fea2-41d0-93bf-a23d35a1e835-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.635196 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.635089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4151c7c9-fea2-41d0-93bf-a23d35a1e835-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.635196 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.635129    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4151c7c9-fea2-41d0-93bf-a23d35a1e835-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.635196 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.635169    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brnrm\" (UniqueName: \"kubernetes.io/projected/4151c7c9-fea2-41d0-93bf-a23d35a1e835-kube-api-access-brnrm\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.635329 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.635218    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4151c7c9-fea2-41d0-93bf-a23d35a1e835-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.635329 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.635319    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4151c7c9-fea2-41d0-93bf-a23d35a1e835-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.635400 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.635357    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4151c7c9-fea2-41d0-93bf-a23d35a1e835-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.735892 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.735865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4151c7c9-fea2-41d0-93bf-a23d35a1e835-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.736020 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.735903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4151c7c9-fea2-41d0-93bf-a23d35a1e835-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.736020 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.735966    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4151c7c9-fea2-41d0-93bf-a23d35a1e835-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.736114 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.736021    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4151c7c9-fea2-41d0-93bf-a23d35a1e835-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.736114 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.736065    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4151c7c9-fea2-41d0-93bf-a23d35a1e835-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.736114 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.736087    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4151c7c9-fea2-41d0-93bf-a23d35a1e835-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.736278 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.736126    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brnrm\" (UniqueName: \"kubernetes.io/projected/4151c7c9-fea2-41d0-93bf-a23d35a1e835-kube-api-access-brnrm\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.737019 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.736987    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4151c7c9-fea2-41d0-93bf-a23d35a1e835-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.738845 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.738805    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4151c7c9-fea2-41d0-93bf-a23d35a1e835-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.738845 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.738815    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4151c7c9-fea2-41d0-93bf-a23d35a1e835-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.739213 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.739193    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4151c7c9-fea2-41d0-93bf-a23d35a1e835-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.739417 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.739398    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4151c7c9-fea2-41d0-93bf-a23d35a1e835-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.755536 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.755514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brnrm\" (UniqueName: \"kubernetes.io/projected/4151c7c9-fea2-41d0-93bf-a23d35a1e835-kube-api-access-brnrm\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.757480 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.757462    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4151c7c9-fea2-41d0-93bf-a23d35a1e835-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-rx9kb\" (UID: \"4151c7c9-fea2-41d0-93bf-a23d35a1e835\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:30.864978 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:30.864887    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:31.032166 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:40:31.032127    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4151c7c9_fea2_41d0_93bf_a23d35a1e835.slice/crio-7987a0e83a2471f29c0958ca32a8002a1d44c160962c6a68c51ed8ef55f5195e WatchSource:0}: Error finding container 7987a0e83a2471f29c0958ca32a8002a1d44c160962c6a68c51ed8ef55f5195e: Status 404 returned error can't find the container with id 7987a0e83a2471f29c0958ca32a8002a1d44c160962c6a68c51ed8ef55f5195e
Jul 16 22:40:31.034309 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:31.034129    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"]
Jul 16 22:40:31.529157 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:31.529117    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt" event={"ID":"62707e5b-445c-45ac-8d6a-3ef554d6e4b7","Type":"ContainerStarted","Data":"3e56d01063d6a9f18d6f33ce7f0ab56691438de9c1c7903ee291a5421940ceeb"}
Jul 16 22:40:31.531790 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:31.531764    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb" event={"ID":"4151c7c9-fea2-41d0-93bf-a23d35a1e835","Type":"ContainerStarted","Data":"7987a0e83a2471f29c0958ca32a8002a1d44c160962c6a68c51ed8ef55f5195e"}
Jul 16 22:40:34.429060 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:34.429010    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 22:40:34.429326 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:34.429095    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 22:40:34.433433 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:34.433405    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 22:40:34.433510 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:34.433463    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 22:40:34.433510 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:34.433490    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 22:40:34.550223 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:34.550110    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb" event={"ID":"4151c7c9-fea2-41d0-93bf-a23d35a1e835","Type":"ContainerStarted","Data":"3389afed3ef5e428e57e864f2b583a667278aade7285d6481e6d418440e11117"}
Jul 16 22:40:34.550223 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:34.550181    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:34.551923 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:34.551878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt" event={"ID":"62707e5b-445c-45ac-8d6a-3ef554d6e4b7","Type":"ContainerStarted","Data":"b83ea88e27d82efdbb1be80fe0d951de929b7d25a5746aa4301c5d5a4cb10467"}
Jul 16 22:40:34.594552 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:34.594495    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb" podStartSLOduration=1.200325836 podStartE2EDuration="4.594473935s" totalImagesPullingTime="3.394148099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:34.593053664 +0000 UTC m=+445.341537736" watchObservedRunningTime="2026-07-16 22:40:34.594473935 +0000 UTC m=+445.342958008"
Jul 16 22:40:34.881345 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:34.881308    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:35.558549 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:35.558520    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-rx9kb"
Jul 16 22:40:35.598714 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:35.598661    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt" podStartSLOduration=2.702122967 podStartE2EDuration="6.598641906s" totalImagesPullingTime="3.896518939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:34.657966017 +0000 UTC m=+445.406450099" watchObservedRunningTime="2026-07-16 22:40:35.598641906 +0000 UTC m=+446.347125977"
Jul 16 22:40:35.886030 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:35.886002    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:36.561058 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:36.561020    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:36.561964 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:36.561947    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4n4xlt"
Jul 16 22:40:41.533904 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:40:41.533868    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-84c74b69b8-p9m6n"
Jul 16 22:42:57.902449 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:57.902417    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"]
Jul 16 22:42:57.905572 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:57.905556    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:57.907888 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:57.907866    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fzpg2\""
Jul 16 22:42:57.933232 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:57.933175    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"]
Jul 16 22:42:57.990933 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:57.990894    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adee9cd9-4926-40e7-a094-7cd9b70591cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:57.990933 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:57.990934    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adee9cd9-4926-40e7-a094-7cd9b70591cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:57.991187 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:57.990961    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adee9cd9-4926-40e7-a094-7cd9b70591cd-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:57.991187 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:57.990989    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adee9cd9-4926-40e7-a094-7cd9b70591cd-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:57.991187 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:57.991118    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adee9cd9-4926-40e7-a094-7cd9b70591cd-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:57.991187 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:57.991161    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxnbs\" (UniqueName: \"kubernetes.io/projected/adee9cd9-4926-40e7-a094-7cd9b70591cd-kube-api-access-qxnbs\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:57.991464 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:57.991236    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adee9cd9-4926-40e7-a094-7cd9b70591cd-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:57.991464 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:57.991288    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adee9cd9-4926-40e7-a094-7cd9b70591cd-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:57.991464 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:57.991321    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adee9cd9-4926-40e7-a094-7cd9b70591cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.092415 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.092380    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adee9cd9-4926-40e7-a094-7cd9b70591cd-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.092415 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.092420    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adee9cd9-4926-40e7-a094-7cd9b70591cd-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.092639 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.092442    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adee9cd9-4926-40e7-a094-7cd9b70591cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.092639 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.092490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adee9cd9-4926-40e7-a094-7cd9b70591cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.092639 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.092517    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adee9cd9-4926-40e7-a094-7cd9b70591cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.092639 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.092550    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adee9cd9-4926-40e7-a094-7cd9b70591cd-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.092639 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.092581    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adee9cd9-4926-40e7-a094-7cd9b70591cd-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.092893 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.092670    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adee9cd9-4926-40e7-a094-7cd9b70591cd-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.092893 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.092696    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxnbs\" (UniqueName: \"kubernetes.io/projected/adee9cd9-4926-40e7-a094-7cd9b70591cd-kube-api-access-qxnbs\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.092893 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.092858    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adee9cd9-4926-40e7-a094-7cd9b70591cd-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.093052 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.092962    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adee9cd9-4926-40e7-a094-7cd9b70591cd-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.093109 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.093049    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adee9cd9-4926-40e7-a094-7cd9b70591cd-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.093163 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.093147    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adee9cd9-4926-40e7-a094-7cd9b70591cd-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.093373 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.093351    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adee9cd9-4926-40e7-a094-7cd9b70591cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.094739 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.094720    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adee9cd9-4926-40e7-a094-7cd9b70591cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.094966 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.094947    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adee9cd9-4926-40e7-a094-7cd9b70591cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.102842 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.102818    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adee9cd9-4926-40e7-a094-7cd9b70591cd-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.102993 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.102973    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxnbs\" (UniqueName: \"kubernetes.io/projected/adee9cd9-4926-40e7-a094-7cd9b70591cd-kube-api-access-qxnbs\") pod \"maas-default-gateway-openshift-default-56654dcf7-l7jvp\" (UID: \"adee9cd9-4926-40e7-a094-7cd9b70591cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.215978 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.215880    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:58.384755 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.384717    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"]
Jul 16 22:42:58.389658 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:42:58.389626    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadee9cd9_4926_40e7_a094_7cd9b70591cd.slice/crio-170de902de3cdd3c049527f0bc07967289f8cf07754ee6e7d0d83ba8477a89b9 WatchSource:0}: Error finding container 170de902de3cdd3c049527f0bc07967289f8cf07754ee6e7d0d83ba8477a89b9: Status 404 returned error can't find the container with id 170de902de3cdd3c049527f0bc07967289f8cf07754ee6e7d0d83ba8477a89b9
Jul 16 22:42:58.391734 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.391705    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 22:42:58.391800 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.391773    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 22:42:58.391837 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:58.391806    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 22:42:59.176721 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:59.176681    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp" event={"ID":"adee9cd9-4926-40e7-a094-7cd9b70591cd","Type":"ContainerStarted","Data":"9785466da89a6052c7554e66e9b76015ecfc53cb229371583088c5e2d09765f0"}
Jul 16 22:42:59.177094 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:59.176737    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp" event={"ID":"adee9cd9-4926-40e7-a094-7cd9b70591cd","Type":"ContainerStarted","Data":"170de902de3cdd3c049527f0bc07967289f8cf07754ee6e7d0d83ba8477a89b9"}
Jul 16 22:42:59.198485 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:59.198425    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp" podStartSLOduration=2.19840791 podStartE2EDuration="2.19840791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:42:59.196896319 +0000 UTC m=+589.945380389" watchObservedRunningTime="2026-07-16 22:42:59.19840791 +0000 UTC m=+589.946891981"
Jul 16 22:42:59.216642 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:59.216606    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:42:59.221585 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:42:59.221556    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:43:00.181415 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:43:00.181387    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:43:00.182476 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:43:00.182459    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:44:12.980750 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:12.980719    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll"]
Jul 16 22:44:12.983625 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:12.983606    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll"
Jul 16 22:44:12.990381 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:12.990061    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 16 22:44:12.990556 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:12.990534    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:44:12.990632 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:12.990567    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l8rw4\""
Jul 16 22:44:12.990632 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:12.990545    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 16 22:44:12.999272 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:12.999231    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll"]
Jul 16 22:44:13.159454 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.159424    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnm8h\" (UniqueName: \"kubernetes.io/projected/53925486-8958-4f47-8274-1afd2e6cfce4-kube-api-access-gnm8h\") pod \"maas-api-6d6d9dd47b-hznll\" (UID: \"53925486-8958-4f47-8274-1afd2e6cfce4\") " pod="odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll"
Jul 16 22:44:13.159646 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.159468    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53925486-8958-4f47-8274-1afd2e6cfce4-maas-api-tls\") pod \"maas-api-6d6d9dd47b-hznll\" (UID: \"53925486-8958-4f47-8274-1afd2e6cfce4\") " pod="odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll"
Jul 16 22:44:13.159646 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.159499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53925486-8958-4f47-8274-1afd2e6cfce4-openshift-service-ca\") pod \"maas-api-6d6d9dd47b-hznll\" (UID: \"53925486-8958-4f47-8274-1afd2e6cfce4\") " pod="odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll"
Jul 16 22:44:13.260361 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.260270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnm8h\" (UniqueName: \"kubernetes.io/projected/53925486-8958-4f47-8274-1afd2e6cfce4-kube-api-access-gnm8h\") pod \"maas-api-6d6d9dd47b-hznll\" (UID: \"53925486-8958-4f47-8274-1afd2e6cfce4\") " pod="odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll"
Jul 16 22:44:13.260361 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.260321    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53925486-8958-4f47-8274-1afd2e6cfce4-maas-api-tls\") pod \"maas-api-6d6d9dd47b-hznll\" (UID: \"53925486-8958-4f47-8274-1afd2e6cfce4\") " pod="odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll"
Jul 16 22:44:13.260361 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.260339    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53925486-8958-4f47-8274-1afd2e6cfce4-openshift-service-ca\") pod \"maas-api-6d6d9dd47b-hznll\" (UID: \"53925486-8958-4f47-8274-1afd2e6cfce4\") " pod="odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll"
Jul 16 22:44:13.261148 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.261119    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53925486-8958-4f47-8274-1afd2e6cfce4-openshift-service-ca\") pod \"maas-api-6d6d9dd47b-hznll\" (UID: \"53925486-8958-4f47-8274-1afd2e6cfce4\") " pod="odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll"
Jul 16 22:44:13.262932 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.262910    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53925486-8958-4f47-8274-1afd2e6cfce4-maas-api-tls\") pod \"maas-api-6d6d9dd47b-hznll\" (UID: \"53925486-8958-4f47-8274-1afd2e6cfce4\") " pod="odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll"
Jul 16 22:44:13.268726 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.268692    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnm8h\" (UniqueName: \"kubernetes.io/projected/53925486-8958-4f47-8274-1afd2e6cfce4-kube-api-access-gnm8h\") pod \"maas-api-6d6d9dd47b-hznll\" (UID: \"53925486-8958-4f47-8274-1afd2e6cfce4\") " pod="odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll"
Jul 16 22:44:13.294418 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.294390    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll"
Jul 16 22:44:13.452862 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:44:13.452831    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53925486_8958_4f47_8274_1afd2e6cfce4.slice/crio-72f47ec25d75165223d199612c392c9b03066ed29d7da153a92cfbc649af8d4a WatchSource:0}: Error finding container 72f47ec25d75165223d199612c392c9b03066ed29d7da153a92cfbc649af8d4a: Status 404 returned error can't find the container with id 72f47ec25d75165223d199612c392c9b03066ed29d7da153a92cfbc649af8d4a
Jul 16 22:44:13.458647 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.458623    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll"]
Jul 16 22:44:13.501444 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.501415    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll" event={"ID":"53925486-8958-4f47-8274-1afd2e6cfce4","Type":"ContainerStarted","Data":"72f47ec25d75165223d199612c392c9b03066ed29d7da153a92cfbc649af8d4a"}
Jul 16 22:44:13.688040 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.688005    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-gqxs8"]
Jul 16 22:44:13.692463 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.692446    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-gqxs8"
Jul 16 22:44:13.694917 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.694896    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 16 22:44:13.700840 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.700816    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-gqxs8"]
Jul 16 22:44:13.866928 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.866892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9gg9\" (UniqueName: \"kubernetes.io/projected/8c7638b3-99dd-4860-9078-26e471cb24a6-kube-api-access-p9gg9\") pod \"payload-pre-processing-7677b774bb-gqxs8\" (UID: \"8c7638b3-99dd-4860-9078-26e471cb24a6\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-gqxs8"
Jul 16 22:44:13.867081 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.866975    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c7638b3-99dd-4860-9078-26e471cb24a6-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-gqxs8\" (UID: \"8c7638b3-99dd-4860-9078-26e471cb24a6\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-gqxs8"
Jul 16 22:44:13.968404 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.968327    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9gg9\" (UniqueName: \"kubernetes.io/projected/8c7638b3-99dd-4860-9078-26e471cb24a6-kube-api-access-p9gg9\") pod \"payload-pre-processing-7677b774bb-gqxs8\" (UID: \"8c7638b3-99dd-4860-9078-26e471cb24a6\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-gqxs8"
Jul 16 22:44:13.968404 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.968398    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c7638b3-99dd-4860-9078-26e471cb24a6-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-gqxs8\" (UID: \"8c7638b3-99dd-4860-9078-26e471cb24a6\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-gqxs8"
Jul 16 22:44:13.969171 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.969151    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c7638b3-99dd-4860-9078-26e471cb24a6-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-gqxs8\" (UID: \"8c7638b3-99dd-4860-9078-26e471cb24a6\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-gqxs8"
Jul 16 22:44:13.976872 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:13.976854    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9gg9\" (UniqueName: \"kubernetes.io/projected/8c7638b3-99dd-4860-9078-26e471cb24a6-kube-api-access-p9gg9\") pod \"payload-pre-processing-7677b774bb-gqxs8\" (UID: \"8c7638b3-99dd-4860-9078-26e471cb24a6\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-gqxs8"
Jul 16 22:44:14.003989 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:14.003953    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-gqxs8"
Jul 16 22:44:14.172149 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:14.172075    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-gqxs8"]
Jul 16 22:44:14.174906 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:44:14.174874    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7638b3_99dd_4860_9078_26e471cb24a6.slice/crio-8ed4a8149bfab67022c500a5c473b4dd0cc11a751b5b40eb20eb1dfe7a0e195e WatchSource:0}: Error finding container 8ed4a8149bfab67022c500a5c473b4dd0cc11a751b5b40eb20eb1dfe7a0e195e: Status 404 returned error can't find the container with id 8ed4a8149bfab67022c500a5c473b4dd0cc11a751b5b40eb20eb1dfe7a0e195e
Jul 16 22:44:14.508691 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:14.508645    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-gqxs8" event={"ID":"8c7638b3-99dd-4860-9078-26e471cb24a6","Type":"ContainerStarted","Data":"8ed4a8149bfab67022c500a5c473b4dd0cc11a751b5b40eb20eb1dfe7a0e195e"}
Jul 16 22:44:17.526726 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:17.526691    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-gqxs8" event={"ID":"8c7638b3-99dd-4860-9078-26e471cb24a6","Type":"ContainerStarted","Data":"af080ba28f4a913d62a0d899e04a862d72fd357bc8594845f3ef872faa8d9959"}
Jul 16 22:44:17.527217 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:17.526806    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-gqxs8"
Jul 16 22:44:17.528302 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:17.528269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll" event={"ID":"53925486-8958-4f47-8274-1afd2e6cfce4","Type":"ContainerStarted","Data":"1db7991a3c172b3d50756222821f935c4a96abd77192d195e3de625bab22f756"}
Jul 16 22:44:17.528418 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:17.528409    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll"
Jul 16 22:44:17.569508 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:17.569444    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-gqxs8" podStartSLOduration=1.875651359 podStartE2EDuration="4.569402527s" totalImagesPullingTime="2.693751168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:44:17.548620868 +0000 UTC m=+668.297104940" watchObservedRunningTime="2026-07-16 22:44:17.569402527 +0000 UTC m=+668.317886597"
Jul 16 22:44:17.569906 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:17.569862    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll" podStartSLOduration=2.155845044 podStartE2EDuration="5.569852207s" totalImagesPullingTime="3.414007163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:44:17.568971208 +0000 UTC m=+668.317455280" watchObservedRunningTime="2026-07-16 22:44:17.569852207 +0000 UTC m=+668.318336279"
Jul 16 22:44:21.548995 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:21.548916    2565 generic.go:356] "Generic (PLEG): container finished" podID="adee9cd9-4926-40e7-a094-7cd9b70591cd" containerID="9785466da89a6052c7554e66e9b76015ecfc53cb229371583088c5e2d09765f0" exitCode=0
Jul 16 22:44:21.549404 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:21.548988    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp" event={"ID":"adee9cd9-4926-40e7-a094-7cd9b70591cd","Type":"ContainerDied","Data":"9785466da89a6052c7554e66e9b76015ecfc53cb229371583088c5e2d09765f0"}
Jul 16 22:44:21.549404 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:21.549395    2565 scope.go:117] "RemoveContainer" containerID="9785466da89a6052c7554e66e9b76015ecfc53cb229371583088c5e2d09765f0"
Jul 16 22:44:21.550338 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:21.550311    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 22:44:21.550426 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:21.550374    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 22:44:21.550426 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:21.550405    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 22:44:22.557876 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:22.557837    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp" event={"ID":"adee9cd9-4926-40e7-a094-7cd9b70591cd","Type":"ContainerStarted","Data":"d5813b9da85e02f8a7d832326f88eca8c69c6c9b20030fc0efab9e04e9d0adf6"}
Jul 16 22:44:23.216019 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:23.215972    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:44:23.216292 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:23.216230    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-l7jvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body=
Jul 16 22:44:23.216428 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:23.216305    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp" podUID="adee9cd9-4926-40e7-a094-7cd9b70591cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused"
Jul 16 22:44:23.544781 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:23.544678    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d6d9dd47b-hznll"
Jul 16 22:44:24.216966 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:24.216929    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-l7jvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body=
Jul 16 22:44:24.217393 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:24.216987    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp" podUID="adee9cd9-4926-40e7-a094-7cd9b70591cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused"
Jul 16 22:44:25.216481 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:25.216436    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-l7jvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body=
Jul 16 22:44:25.216793 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:25.216751    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp" podUID="adee9cd9-4926-40e7-a094-7cd9b70591cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused"
Jul 16 22:44:26.216637 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:26.216599    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-l7jvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused" start-of-body=
Jul 16 22:44:26.217042 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:26.216676    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp" podUID="adee9cd9-4926-40e7-a094-7cd9b70591cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.25:15021/healthz/ready\": dial tcp 10.132.0.25:15021: connect: connection refused"
Jul 16 22:44:27.220772 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:27.220736    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:44:27.221239 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:27.221128    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:44:27.221790 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:27.221771    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-l7jvp"
Jul 16 22:44:28.535630 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:28.535597    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-gqxs8"
Jul 16 22:44:48.805175 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.805138    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"]
Jul 16 22:44:48.808609 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.808590    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:48.812649 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.812622    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dgvpk\""
Jul 16 22:44:48.812797 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.812675    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 16 22:44:48.813392 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.813371    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 16 22:44:48.813506 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.813405    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 16 22:44:48.825013 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.824989    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"]
Jul 16 22:44:48.889670 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.889638    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017d0692-e8a1-4857-8e2e-0c422706856d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-55dt9\" (UID: \"017d0692-e8a1-4857-8e2e-0c422706856d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:48.889907 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.889724    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017d0692-e8a1-4857-8e2e-0c422706856d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-55dt9\" (UID: \"017d0692-e8a1-4857-8e2e-0c422706856d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:48.889907 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.889782    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017d0692-e8a1-4857-8e2e-0c422706856d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-55dt9\" (UID: \"017d0692-e8a1-4857-8e2e-0c422706856d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:48.889907 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.889833    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017d0692-e8a1-4857-8e2e-0c422706856d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-55dt9\" (UID: \"017d0692-e8a1-4857-8e2e-0c422706856d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:48.890078 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.889913    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017d0692-e8a1-4857-8e2e-0c422706856d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-55dt9\" (UID: \"017d0692-e8a1-4857-8e2e-0c422706856d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:48.890078 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.889989    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdm7c\" (UniqueName: \"kubernetes.io/projected/017d0692-e8a1-4857-8e2e-0c422706856d-kube-api-access-fdm7c\") pod \"e2e-distinct-simulated-kserve-59975d69f9-55dt9\" (UID: \"017d0692-e8a1-4857-8e2e-0c422706856d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:48.990715 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.990679    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017d0692-e8a1-4857-8e2e-0c422706856d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-55dt9\" (UID: \"017d0692-e8a1-4857-8e2e-0c422706856d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:48.990715 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.990721    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017d0692-e8a1-4857-8e2e-0c422706856d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-55dt9\" (UID: \"017d0692-e8a1-4857-8e2e-0c422706856d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:48.990921 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.990743    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdm7c\" (UniqueName: \"kubernetes.io/projected/017d0692-e8a1-4857-8e2e-0c422706856d-kube-api-access-fdm7c\") pod \"e2e-distinct-simulated-kserve-59975d69f9-55dt9\" (UID: \"017d0692-e8a1-4857-8e2e-0c422706856d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:48.990921 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.990860    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017d0692-e8a1-4857-8e2e-0c422706856d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-55dt9\" (UID: \"017d0692-e8a1-4857-8e2e-0c422706856d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:48.991026 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.990919    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017d0692-e8a1-4857-8e2e-0c422706856d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-55dt9\" (UID: \"017d0692-e8a1-4857-8e2e-0c422706856d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:48.991026 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.990955    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017d0692-e8a1-4857-8e2e-0c422706856d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-55dt9\" (UID: \"017d0692-e8a1-4857-8e2e-0c422706856d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:48.991152 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.991127    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/017d0692-e8a1-4857-8e2e-0c422706856d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-55dt9\" (UID: \"017d0692-e8a1-4857-8e2e-0c422706856d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:48.991238 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.991221    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/017d0692-e8a1-4857-8e2e-0c422706856d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-55dt9\" (UID: \"017d0692-e8a1-4857-8e2e-0c422706856d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:48.991338 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.991296    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/017d0692-e8a1-4857-8e2e-0c422706856d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-55dt9\" (UID: \"017d0692-e8a1-4857-8e2e-0c422706856d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:48.993013 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.992989    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/017d0692-e8a1-4857-8e2e-0c422706856d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-55dt9\" (UID: \"017d0692-e8a1-4857-8e2e-0c422706856d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:48.993412 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.993396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/017d0692-e8a1-4857-8e2e-0c422706856d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-55dt9\" (UID: \"017d0692-e8a1-4857-8e2e-0c422706856d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:48.999976 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:48.999942    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdm7c\" (UniqueName: \"kubernetes.io/projected/017d0692-e8a1-4857-8e2e-0c422706856d-kube-api-access-fdm7c\") pod \"e2e-distinct-simulated-kserve-59975d69f9-55dt9\" (UID: \"017d0692-e8a1-4857-8e2e-0c422706856d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:49.119979 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:49.119936    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:44:49.251320 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:49.251294    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"]
Jul 16 22:44:49.256221 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:44:49.254358    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017d0692_e8a1_4857_8e2e_0c422706856d.slice/crio-64d95128f7f88b3a949f040c86ac58bcc788ddd965f6fc3d8fc8858a35b8e652 WatchSource:0}: Error finding container 64d95128f7f88b3a949f040c86ac58bcc788ddd965f6fc3d8fc8858a35b8e652: Status 404 returned error can't find the container with id 64d95128f7f88b3a949f040c86ac58bcc788ddd965f6fc3d8fc8858a35b8e652
Jul 16 22:44:49.257472 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:49.257452    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:44:49.676509 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:44:49.676474    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9" event={"ID":"017d0692-e8a1-4857-8e2e-0c422706856d","Type":"ContainerStarted","Data":"64d95128f7f88b3a949f040c86ac58bcc788ddd965f6fc3d8fc8858a35b8e652"}
Jul 16 22:45:00.139746 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:00.139707    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737365-wbzqw"]
Jul 16 22:45:00.144123 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:00.144103    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-wbzqw"
Jul 16 22:45:00.147709 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:00.147688    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 22:45:00.154219 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:00.154197    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-wbzqw"
Jul 16 22:45:00.155057 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:00.155035    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737365-wbzqw"]
Jul 16 22:45:00.278945 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:00.278921    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737365-wbzqw"]
Jul 16 22:45:00.281686 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:45:00.281658    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1aa0e44_2b47_4ef3_9010_c32b4a31a612.slice/crio-6a7aea1696738996f606f0124ec4716483c47bc9be2d98107ffa8f65499ca83c WatchSource:0}: Error finding container 6a7aea1696738996f606f0124ec4716483c47bc9be2d98107ffa8f65499ca83c: Status 404 returned error can't find the container with id 6a7aea1696738996f606f0124ec4716483c47bc9be2d98107ffa8f65499ca83c
Jul 16 22:45:00.725893 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:00.725863    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-wbzqw" event={"ID":"a1aa0e44-2b47-4ef3-9010-c32b4a31a612","Type":"ContainerStarted","Data":"6a7aea1696738996f606f0124ec4716483c47bc9be2d98107ffa8f65499ca83c"}
Jul 16 22:45:08.760885 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:08.760848    2565 generic.go:356] "Generic (PLEG): container finished" podID="a1aa0e44-2b47-4ef3-9010-c32b4a31a612" containerID="fd71e2585dcf862f6bb9c1e5651508a8b02517718d97f25bb28f9b202229a111" exitCode=0
Jul 16 22:45:08.761241 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:08.760937    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-wbzqw" event={"ID":"a1aa0e44-2b47-4ef3-9010-c32b4a31a612","Type":"ContainerDied","Data":"fd71e2585dcf862f6bb9c1e5651508a8b02517718d97f25bb28f9b202229a111"}
Jul 16 22:45:09.898774 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:09.898749    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-wbzqw"
Jul 16 22:45:10.769911 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:10.769889    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-wbzqw"
Jul 16 22:45:10.770158 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:10.769937    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-wbzqw" event={"ID":"a1aa0e44-2b47-4ef3-9010-c32b4a31a612","Type":"ContainerDied","Data":"6a7aea1696738996f606f0124ec4716483c47bc9be2d98107ffa8f65499ca83c"}
Jul 16 22:45:10.770158 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:10.769982    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a7aea1696738996f606f0124ec4716483c47bc9be2d98107ffa8f65499ca83c"
Jul 16 22:45:15.795839 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:15.795749    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9" event={"ID":"017d0692-e8a1-4857-8e2e-0c422706856d","Type":"ContainerStarted","Data":"a71c306f50397b0343ca99cc78c88cd7408e9e7ab31a259b160764069195e8ed"}
Jul 16 22:45:15.818330 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:15.818284    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9" podStartSLOduration=1.716706031 podStartE2EDuration="27.818267391s" totalImagesPullingTime="26.10156136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:44:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:15.817054151 +0000 UTC m=+726.565538223" watchObservedRunningTime="2026-07-16 22:45:15.818267391 +0000 UTC m=+726.566751480"
Jul 16 22:45:19.120357 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:19.120318    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:45:19.120357 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:19.120371    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:45:19.132824 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:19.132796    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:45:19.824666 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:19.824630    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-55dt9"
Jul 16 22:45:30.858705 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:30.858667    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"]
Jul 16 22:45:30.859117 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:30.859102    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1aa0e44-2b47-4ef3-9010-c32b4a31a612" containerName="cleanup"
Jul 16 22:45:30.859117 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:30.859117    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1aa0e44-2b47-4ef3-9010-c32b4a31a612" containerName="cleanup"
Jul 16 22:45:30.859200 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:30.859179    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1aa0e44-2b47-4ef3-9010-c32b4a31a612" containerName="cleanup"
Jul 16 22:45:30.862216 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:30.862198    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:30.865543 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:30.865520    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 16 22:45:30.883727 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:30.883704    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"]
Jul 16 22:45:30.959190 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:30.959149    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bb59881-cd5f-4089-925d-6d0af975f6f5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t\" (UID: \"8bb59881-cd5f-4089-925d-6d0af975f6f5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:30.959391 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:30.959199    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bb59881-cd5f-4089-925d-6d0af975f6f5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t\" (UID: \"8bb59881-cd5f-4089-925d-6d0af975f6f5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:30.959391 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:30.959298    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bb59881-cd5f-4089-925d-6d0af975f6f5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t\" (UID: \"8bb59881-cd5f-4089-925d-6d0af975f6f5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:30.959391 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:30.959354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bb59881-cd5f-4089-925d-6d0af975f6f5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t\" (UID: \"8bb59881-cd5f-4089-925d-6d0af975f6f5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:30.959391 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:30.959383    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bb59881-cd5f-4089-925d-6d0af975f6f5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t\" (UID: \"8bb59881-cd5f-4089-925d-6d0af975f6f5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:30.959554 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:30.959414    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z84r9\" (UniqueName: \"kubernetes.io/projected/8bb59881-cd5f-4089-925d-6d0af975f6f5-kube-api-access-z84r9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t\" (UID: \"8bb59881-cd5f-4089-925d-6d0af975f6f5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:31.060554 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:31.060511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bb59881-cd5f-4089-925d-6d0af975f6f5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t\" (UID: \"8bb59881-cd5f-4089-925d-6d0af975f6f5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:31.060750 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:31.060573    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bb59881-cd5f-4089-925d-6d0af975f6f5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t\" (UID: \"8bb59881-cd5f-4089-925d-6d0af975f6f5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:31.060750 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:31.060595    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bb59881-cd5f-4089-925d-6d0af975f6f5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t\" (UID: \"8bb59881-cd5f-4089-925d-6d0af975f6f5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:31.060750 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:31.060616    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z84r9\" (UniqueName: \"kubernetes.io/projected/8bb59881-cd5f-4089-925d-6d0af975f6f5-kube-api-access-z84r9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t\" (UID: \"8bb59881-cd5f-4089-925d-6d0af975f6f5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:31.060750 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:31.060652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bb59881-cd5f-4089-925d-6d0af975f6f5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t\" (UID: \"8bb59881-cd5f-4089-925d-6d0af975f6f5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:31.060750 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:31.060673    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bb59881-cd5f-4089-925d-6d0af975f6f5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t\" (UID: \"8bb59881-cd5f-4089-925d-6d0af975f6f5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:31.061013 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:31.060988    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bb59881-cd5f-4089-925d-6d0af975f6f5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t\" (UID: \"8bb59881-cd5f-4089-925d-6d0af975f6f5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:31.061074 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:31.061029    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bb59881-cd5f-4089-925d-6d0af975f6f5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t\" (UID: \"8bb59881-cd5f-4089-925d-6d0af975f6f5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:31.061074 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:31.061067    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bb59881-cd5f-4089-925d-6d0af975f6f5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t\" (UID: \"8bb59881-cd5f-4089-925d-6d0af975f6f5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:31.062890 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:31.062869    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bb59881-cd5f-4089-925d-6d0af975f6f5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t\" (UID: \"8bb59881-cd5f-4089-925d-6d0af975f6f5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:31.063055 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:31.063038    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bb59881-cd5f-4089-925d-6d0af975f6f5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t\" (UID: \"8bb59881-cd5f-4089-925d-6d0af975f6f5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:31.072081 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:31.072059    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z84r9\" (UniqueName: \"kubernetes.io/projected/8bb59881-cd5f-4089-925d-6d0af975f6f5-kube-api-access-z84r9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t\" (UID: \"8bb59881-cd5f-4089-925d-6d0af975f6f5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:31.172614 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:31.172526    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:31.301516 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:31.301484    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"]
Jul 16 22:45:31.304453 ip-10-0-139-23 kubenswrapper[2565]: W0716 22:45:31.304429    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb59881_cd5f_4089_925d_6d0af975f6f5.slice/crio-1d0cb4c02d52eb9cdf4b923b7bfcbd2b77a94a23e15fb12f3372a030fd11d065 WatchSource:0}: Error finding container 1d0cb4c02d52eb9cdf4b923b7bfcbd2b77a94a23e15fb12f3372a030fd11d065: Status 404 returned error can't find the container with id 1d0cb4c02d52eb9cdf4b923b7bfcbd2b77a94a23e15fb12f3372a030fd11d065
Jul 16 22:45:31.873304 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:31.873266    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t" event={"ID":"8bb59881-cd5f-4089-925d-6d0af975f6f5","Type":"ContainerStarted","Data":"1d0cb4c02d52eb9cdf4b923b7bfcbd2b77a94a23e15fb12f3372a030fd11d065"}
Jul 16 22:45:48.949633 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:48.949592    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t" event={"ID":"8bb59881-cd5f-4089-925d-6d0af975f6f5","Type":"ContainerStarted","Data":"d824e852ecd2a64a62a77fc4d17aa6f285068dfbbadc71cf3fc4074abe3c3daa"}
Jul 16 22:45:48.968523 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:48.968479    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t" podStartSLOduration=1.484863289 podStartE2EDuration="18.968464929s" totalImagesPullingTime="17.48360164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:48.967885284 +0000 UTC m=+759.716369356" watchObservedRunningTime="2026-07-16 22:45:48.968464929 +0000 UTC m=+759.716948999"
Jul 16 22:45:51.173537 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:51.173499    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:51.174034 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:51.173568    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:51.185702 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:51.185677    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 22:45:51.973996 ip-10-0-139-23 kubenswrapper[2565]: I0716 22:45:51.973966    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t"
Jul 16 23:00:00.145572 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:00:00.145531    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737380-d66wg"]
Jul 16 23:00:00.148803 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:00:00.148783    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737380-d66wg"
Jul 16 23:00:00.151209 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:00:00.151186    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:00:00.158587 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:00:00.158567    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737380-d66wg"
Jul 16 23:00:00.166299 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:00:00.166267    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737380-d66wg"]
Jul 16 23:00:00.295956 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:00:00.295873    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737380-d66wg"]
Jul 16 23:00:00.298888 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:00:00.298852    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c880144_9643_4922_bd49_8a30c9f98b8c.slice/crio-146c09507541c6584ac92fdecf9b6eabafa31c0dd93029b3609a2f4c51aec385 WatchSource:0}: Error finding container 146c09507541c6584ac92fdecf9b6eabafa31c0dd93029b3609a2f4c51aec385: Status 404 returned error can't find the container with id 146c09507541c6584ac92fdecf9b6eabafa31c0dd93029b3609a2f4c51aec385
Jul 16 23:00:00.923772 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:00:00.923727    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737380-d66wg" event={"ID":"1c880144-9643-4922-bd49-8a30c9f98b8c","Type":"ContainerStarted","Data":"9ad48c8d0fdbfccdfe90114c7df5fa9a33f28136c3a6a6df49af204963abfde0"}
Jul 16 23:00:00.923978 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:00:00.923793    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737380-d66wg" event={"ID":"1c880144-9643-4922-bd49-8a30c9f98b8c","Type":"ContainerStarted","Data":"146c09507541c6584ac92fdecf9b6eabafa31c0dd93029b3609a2f4c51aec385"}
Jul 16 23:00:00.938880 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:00:00.938829    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737380-d66wg" podStartSLOduration=0.93881263 podStartE2EDuration="938.81263ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:00:00.937731977 +0000 UTC m=+1611.686216043" watchObservedRunningTime="2026-07-16 23:00:00.93881263 +0000 UTC m=+1611.687296701"
Jul 16 23:00:01.930078 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:00:01.929983    2565 generic.go:356] "Generic (PLEG): container finished" podID="1c880144-9643-4922-bd49-8a30c9f98b8c" containerID="9ad48c8d0fdbfccdfe90114c7df5fa9a33f28136c3a6a6df49af204963abfde0" exitCode=0
Jul 16 23:00:01.930568 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:00:01.930064    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737380-d66wg" event={"ID":"1c880144-9643-4922-bd49-8a30c9f98b8c","Type":"ContainerDied","Data":"9ad48c8d0fdbfccdfe90114c7df5fa9a33f28136c3a6a6df49af204963abfde0"}
Jul 16 23:00:03.074964 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:00:03.074937    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737380-d66wg"
Jul 16 23:00:03.940911 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:00:03.940870    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737380-d66wg" event={"ID":"1c880144-9643-4922-bd49-8a30c9f98b8c","Type":"ContainerDied","Data":"146c09507541c6584ac92fdecf9b6eabafa31c0dd93029b3609a2f4c51aec385"}
Jul 16 23:00:03.941092 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:00:03.940934    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="146c09507541c6584ac92fdecf9b6eabafa31c0dd93029b3609a2f4c51aec385"
Jul 16 23:00:03.941092 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:00:03.940896    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737380-d66wg"
Jul 16 23:01:15.882118 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:15.882086    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"]
Jul 16 23:01:15.882715 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:15.882688    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c880144-9643-4922-bd49-8a30c9f98b8c" containerName="cleanup"
Jul 16 23:01:15.882715 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:15.882703    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c880144-9643-4922-bd49-8a30c9f98b8c" containerName="cleanup"
Jul 16 23:01:15.882807 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:15.882795    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c880144-9643-4922-bd49-8a30c9f98b8c" containerName="cleanup"
Jul 16 23:01:15.885833 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:15.885809    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:15.888214 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:15.888192    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3933c193-openshift-default-dockercfg-r9mrq\""
Jul 16 23:01:15.902512 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:15.902487    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"]
Jul 16 23:01:15.946793 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:15.946755    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-token\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:15.947001 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:15.946817    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-credential-socket\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:15.947001 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:15.946880    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgmnq\" (UniqueName: \"kubernetes.io/projected/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-kube-api-access-pgmnq\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:15.947001 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:15.946938    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istiod-ca-cert\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:15.947001 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:15.946964    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-data\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:15.947001 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:15.946988    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-podinfo\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:15.947281 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:15.947012    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-workload-certs\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:15.947281 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:15.947061    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-workload-socket\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:15.947281 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:15.947090    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-envoy\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.047947 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.047904    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-token\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.048158 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.047993    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-credential-socket\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.048158 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.048019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgmnq\" (UniqueName: \"kubernetes.io/projected/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-kube-api-access-pgmnq\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.048158 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.048061    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istiod-ca-cert\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.048158 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.048099    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-data\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.048158 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.048128    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-podinfo\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.048158 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.048153    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-workload-certs\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.048548 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.048208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-workload-socket\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.048548 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.048286    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-envoy\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.048548 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.048532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-credential-socket\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.048688 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.048572    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-data\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.048688 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.048642    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-workload-certs\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.048853 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.048835    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-workload-socket\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.048912 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.048894    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istiod-ca-cert\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.050542 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.050525    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-envoy\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.050685 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.050666    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-podinfo\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.056874 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.056842    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-token\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.057002 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.056987    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgmnq\" (UniqueName: \"kubernetes.io/projected/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-kube-api-access-pgmnq\") pod \"e2e-ait-3933c193-openshift-default-57678d7499-n74zt\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") " pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.196357 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.196243    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:16.333860 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.333827    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"]
Jul 16 23:01:16.337540 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:01:16.337512    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6a5e9c_dcf6_403b_a905_aeecfcedc1e6.slice/crio-a040e684b1ec93bea3c4fc560d2e78d5dc518dfb15495042d5ef812ac3e541ce WatchSource:0}: Error finding container a040e684b1ec93bea3c4fc560d2e78d5dc518dfb15495042d5ef812ac3e541ce: Status 404 returned error can't find the container with id a040e684b1ec93bea3c4fc560d2e78d5dc518dfb15495042d5ef812ac3e541ce
Jul 16 23:01:16.339892 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.339856    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:01:16.339993 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.339935    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:01:16.339993 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:16.339969    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:01:17.282518 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:17.282481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt" event={"ID":"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6","Type":"ContainerStarted","Data":"4b749cd434c678c15259dd6f7f34415c1e7c5803d44c641f83b947c17ce08c2f"}
Jul 16 23:01:17.282926 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:17.282565    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt" event={"ID":"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6","Type":"ContainerStarted","Data":"a040e684b1ec93bea3c4fc560d2e78d5dc518dfb15495042d5ef812ac3e541ce"}
Jul 16 23:01:17.355557 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:17.355505    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt" podStartSLOduration=2.355488731 podStartE2EDuration="2.355488731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:01:17.353937186 +0000 UTC m=+1688.102421257" watchObservedRunningTime="2026-07-16 23:01:17.355488731 +0000 UTC m=+1688.103972802"
Jul 16 23:01:17.389509 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:17.389473    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"]
Jul 16 23:01:17.392742 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:17.392725    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:01:17.395487 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:17.395468    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3933c193-serving-cert\""
Jul 16 23:01:17.432076 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:17.432043    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"]
Jul 16 23:01:17.460959 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:17.460924    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e83026bc-2f49-4005-afa7-29b1b1ec22c5-maas-api-tls\") pod \"maas-api-e2e-ait-3933c193-55974dd67c-j2tdc\" (UID: \"e83026bc-2f49-4005-afa7-29b1b1ec22c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:01:17.461118 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:17.461027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83026bc-2f49-4005-afa7-29b1b1ec22c5-openshift-service-ca\") pod \"maas-api-e2e-ait-3933c193-55974dd67c-j2tdc\" (UID: \"e83026bc-2f49-4005-afa7-29b1b1ec22c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:01:17.461118 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:17.461058    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnpf9\" (UniqueName: \"kubernetes.io/projected/e83026bc-2f49-4005-afa7-29b1b1ec22c5-kube-api-access-tnpf9\") pod \"maas-api-e2e-ait-3933c193-55974dd67c-j2tdc\" (UID: \"e83026bc-2f49-4005-afa7-29b1b1ec22c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:01:17.562228 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:17.562144    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83026bc-2f49-4005-afa7-29b1b1ec22c5-openshift-service-ca\") pod \"maas-api-e2e-ait-3933c193-55974dd67c-j2tdc\" (UID: \"e83026bc-2f49-4005-afa7-29b1b1ec22c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:01:17.562228 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:17.562183    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnpf9\" (UniqueName: \"kubernetes.io/projected/e83026bc-2f49-4005-afa7-29b1b1ec22c5-kube-api-access-tnpf9\") pod \"maas-api-e2e-ait-3933c193-55974dd67c-j2tdc\" (UID: \"e83026bc-2f49-4005-afa7-29b1b1ec22c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:01:17.562426 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:17.562270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e83026bc-2f49-4005-afa7-29b1b1ec22c5-maas-api-tls\") pod \"maas-api-e2e-ait-3933c193-55974dd67c-j2tdc\" (UID: \"e83026bc-2f49-4005-afa7-29b1b1ec22c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:01:17.562426 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:01:17.562408    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-serving-cert: secret "maas-api-e2e-ait-3933c193-serving-cert" not found
Jul 16 23:01:17.562497 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:01:17.562461    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e83026bc-2f49-4005-afa7-29b1b1ec22c5-maas-api-tls podName:e83026bc-2f49-4005-afa7-29b1b1ec22c5 nodeName:}" failed. No retries permitted until 2026-07-16 23:01:18.062444571 +0000 UTC m=+1688.810928620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e83026bc-2f49-4005-afa7-29b1b1ec22c5-maas-api-tls") pod "maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" (UID: "e83026bc-2f49-4005-afa7-29b1b1ec22c5") : secret "maas-api-e2e-ait-3933c193-serving-cert" not found
Jul 16 23:01:17.562822 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:17.562801    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83026bc-2f49-4005-afa7-29b1b1ec22c5-openshift-service-ca\") pod \"maas-api-e2e-ait-3933c193-55974dd67c-j2tdc\" (UID: \"e83026bc-2f49-4005-afa7-29b1b1ec22c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:01:17.576080 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:17.576053    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnpf9\" (UniqueName: \"kubernetes.io/projected/e83026bc-2f49-4005-afa7-29b1b1ec22c5-kube-api-access-tnpf9\") pod \"maas-api-e2e-ait-3933c193-55974dd67c-j2tdc\" (UID: \"e83026bc-2f49-4005-afa7-29b1b1ec22c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:01:18.066321 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:18.066280    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e83026bc-2f49-4005-afa7-29b1b1ec22c5-maas-api-tls\") pod \"maas-api-e2e-ait-3933c193-55974dd67c-j2tdc\" (UID: \"e83026bc-2f49-4005-afa7-29b1b1ec22c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:01:18.068584 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:18.068561    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e83026bc-2f49-4005-afa7-29b1b1ec22c5-maas-api-tls\") pod \"maas-api-e2e-ait-3933c193-55974dd67c-j2tdc\" (UID: \"e83026bc-2f49-4005-afa7-29b1b1ec22c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:01:18.196773 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:18.196736    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:18.202288 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:18.202234    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:18.289211 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:18.289180    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:18.290506 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:18.290480    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:18.303468 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:18.303438    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:01:18.480689 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:01:18.480639    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83026bc_2f49_4005_afa7_29b1b1ec22c5.slice/crio-3c17d4631826cd0d91804c8c4c276611046706169d1712367d7fad55a41a2dc5 WatchSource:0}: Error finding container 3c17d4631826cd0d91804c8c4c276611046706169d1712367d7fad55a41a2dc5: Status 404 returned error can't find the container with id 3c17d4631826cd0d91804c8c4c276611046706169d1712367d7fad55a41a2dc5
Jul 16 23:01:18.480839 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:18.480720    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"]
Jul 16 23:01:18.482065 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:18.482045    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 23:01:19.294267 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:19.294219    2565 generic.go:356] "Generic (PLEG): container finished" podID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerID="095c134d6c218cc8d17013a69e7491a09ba4cf1098037dc4d01962e017b8fb21" exitCode=1
Jul 16 23:01:19.294618 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:19.294277    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" event={"ID":"e83026bc-2f49-4005-afa7-29b1b1ec22c5","Type":"ContainerDied","Data":"095c134d6c218cc8d17013a69e7491a09ba4cf1098037dc4d01962e017b8fb21"}
Jul 16 23:01:19.294618 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:19.294333    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" event={"ID":"e83026bc-2f49-4005-afa7-29b1b1ec22c5","Type":"ContainerStarted","Data":"3c17d4631826cd0d91804c8c4c276611046706169d1712367d7fad55a41a2dc5"}
Jul 16 23:01:19.294618 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:19.294547    2565 scope.go:117] "RemoveContainer" containerID="095c134d6c218cc8d17013a69e7491a09ba4cf1098037dc4d01962e017b8fb21"
Jul 16 23:01:20.300703 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:20.300670    2565 generic.go:356] "Generic (PLEG): container finished" podID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerID="3a9ed29ba53c082fd94a7d7b3998a0aefc26f2928e4a306d4cb334ce3c7d5c68" exitCode=1
Jul 16 23:01:20.301165 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:20.300751    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" event={"ID":"e83026bc-2f49-4005-afa7-29b1b1ec22c5","Type":"ContainerDied","Data":"3a9ed29ba53c082fd94a7d7b3998a0aefc26f2928e4a306d4cb334ce3c7d5c68"}
Jul 16 23:01:20.301165 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:20.300819    2565 scope.go:117] "RemoveContainer" containerID="095c134d6c218cc8d17013a69e7491a09ba4cf1098037dc4d01962e017b8fb21"
Jul 16 23:01:20.301165 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:20.300989    2565 scope.go:117] "RemoveContainer" containerID="3a9ed29ba53c082fd94a7d7b3998a0aefc26f2928e4a306d4cb334ce3c7d5c68"
Jul 16 23:01:20.301369 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:01:20.301187    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:01:21.307825 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:21.307795    2565 scope.go:117] "RemoveContainer" containerID="3a9ed29ba53c082fd94a7d7b3998a0aefc26f2928e4a306d4cb334ce3c7d5c68"
Jul 16 23:01:21.308326 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:01:21.308016    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:01:23.304485 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:23.304450    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:01:23.304885 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:23.304841    2565 scope.go:117] "RemoveContainer" containerID="3a9ed29ba53c082fd94a7d7b3998a0aefc26f2928e4a306d4cb334ce3c7d5c68"
Jul 16 23:01:23.305024 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:01:23.305005    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:01:28.303641 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:28.303601    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:01:28.304046 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:28.304014    2565 scope.go:117] "RemoveContainer" containerID="3a9ed29ba53c082fd94a7d7b3998a0aefc26f2928e4a306d4cb334ce3c7d5c68"
Jul 16 23:01:28.304222 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:01:28.304200    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:01:33.290729 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:33.290691    2565 patch_prober.go:28] interesting pod/e2e-ait-3933c193-openshift-default-57678d7499-n74zt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 16 23:01:33.291141 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:33.290749    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt" podUID="dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 16 23:01:33.378398 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:33.378363    2565 generic.go:356] "Generic (PLEG): container finished" podID="dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" containerID="4b749cd434c678c15259dd6f7f34415c1e7c5803d44c641f83b947c17ce08c2f" exitCode=0
Jul 16 23:01:33.378573 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:33.378438    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt" event={"ID":"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6","Type":"ContainerDied","Data":"4b749cd434c678c15259dd6f7f34415c1e7c5803d44c641f83b947c17ce08c2f"}
Jul 16 23:01:33.378900 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:33.378885    2565 scope.go:117] "RemoveContainer" containerID="4b749cd434c678c15259dd6f7f34415c1e7c5803d44c641f83b947c17ce08c2f"
Jul 16 23:01:33.379970 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:33.379942    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:01:33.380023 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:33.380010    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:01:33.380056 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:33.380038    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:01:34.386212 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:34.386177    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt" event={"ID":"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6","Type":"ContainerStarted","Data":"65ec4118d40daddd735899231d9685b60d1c25eeb7c988ecd169a5c887630fa8"}
Jul 16 23:01:35.197398 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:35.197355    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:35.197687 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:35.197666    2565 patch_prober.go:28] interesting pod/e2e-ait-3933c193-openshift-default-57678d7499-n74zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 16 23:01:35.197735 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:35.197711    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt" podUID="dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 16 23:01:36.196835 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:36.196801    2565 patch_prober.go:28] interesting pod/e2e-ait-3933c193-openshift-default-57678d7499-n74zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 16 23:01:36.197238 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:36.196860    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt" podUID="dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 16 23:01:37.196891 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:37.196848    2565 patch_prober.go:28] interesting pod/e2e-ait-3933c193-openshift-default-57678d7499-n74zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 16 23:01:37.197305 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:37.196909    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt" podUID="dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 16 23:01:38.196865 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:38.196826    2565 patch_prober.go:28] interesting pod/e2e-ait-3933c193-openshift-default-57678d7499-n74zt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 16 23:01:38.197047 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:38.196891    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt" podUID="dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 16 23:01:39.201001 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:39.200966    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:39.201388 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:39.201231    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:39.202027 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:39.202009    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:01:40.720689 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:40.720652    2565 scope.go:117] "RemoveContainer" containerID="3a9ed29ba53c082fd94a7d7b3998a0aefc26f2928e4a306d4cb334ce3c7d5c68"
Jul 16 23:01:41.424825 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:41.424793    2565 generic.go:356] "Generic (PLEG): container finished" podID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerID="d2202d836dd4f1e3cbd45fa54f23c85ddd98ee7617395530324e9f7093719b14" exitCode=1
Jul 16 23:01:41.424989 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:41.424851    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" event={"ID":"e83026bc-2f49-4005-afa7-29b1b1ec22c5","Type":"ContainerDied","Data":"d2202d836dd4f1e3cbd45fa54f23c85ddd98ee7617395530324e9f7093719b14"}
Jul 16 23:01:41.424989 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:41.424899    2565 scope.go:117] "RemoveContainer" containerID="3a9ed29ba53c082fd94a7d7b3998a0aefc26f2928e4a306d4cb334ce3c7d5c68"
Jul 16 23:01:41.425225 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:41.425205    2565 scope.go:117] "RemoveContainer" containerID="d2202d836dd4f1e3cbd45fa54f23c85ddd98ee7617395530324e9f7093719b14"
Jul 16 23:01:41.425479 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:01:41.425457    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:01:43.304411 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:43.304369    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:01:43.304877 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:43.304775    2565 scope.go:117] "RemoveContainer" containerID="d2202d836dd4f1e3cbd45fa54f23c85ddd98ee7617395530324e9f7093719b14"
Jul 16 23:01:43.304967 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:01:43.304947    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:01:48.304301 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:48.304240    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:01:48.304700 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:01:48.304673    2565 scope.go:117] "RemoveContainer" containerID="d2202d836dd4f1e3cbd45fa54f23c85ddd98ee7617395530324e9f7093719b14"
Jul 16 23:01:48.304860 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:01:48.304840    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:02:00.720676 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:02:00.720643    2565 scope.go:117] "RemoveContainer" containerID="d2202d836dd4f1e3cbd45fa54f23c85ddd98ee7617395530324e9f7093719b14"
Jul 16 23:02:00.721143 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:02:00.720821    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:02:13.720824 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:02:13.720792    2565 scope.go:117] "RemoveContainer" containerID="d2202d836dd4f1e3cbd45fa54f23c85ddd98ee7617395530324e9f7093719b14"
Jul 16 23:02:15.594038 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:02:15.594003    2565 generic.go:356] "Generic (PLEG): container finished" podID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerID="75613eba85684fc2f580fcd70f440f5bceba09b9b53e3a4be2dc33f9b7f3cf77" exitCode=1
Jul 16 23:02:15.594531 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:02:15.594081    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" event={"ID":"e83026bc-2f49-4005-afa7-29b1b1ec22c5","Type":"ContainerDied","Data":"75613eba85684fc2f580fcd70f440f5bceba09b9b53e3a4be2dc33f9b7f3cf77"}
Jul 16 23:02:15.594531 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:02:15.594149    2565 scope.go:117] "RemoveContainer" containerID="d2202d836dd4f1e3cbd45fa54f23c85ddd98ee7617395530324e9f7093719b14"
Jul 16 23:02:15.594531 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:02:15.594495    2565 scope.go:117] "RemoveContainer" containerID="75613eba85684fc2f580fcd70f440f5bceba09b9b53e3a4be2dc33f9b7f3cf77"
Jul 16 23:02:15.594726 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:02:15.594705    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:02:18.303958 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:02:18.303919    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:02:18.304391 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:02:18.304004    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:02:18.304391 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:02:18.304369    2565 scope.go:117] "RemoveContainer" containerID="75613eba85684fc2f580fcd70f440f5bceba09b9b53e3a4be2dc33f9b7f3cf77"
Jul 16 23:02:18.304556 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:02:18.304536    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:02:28.720341 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:02:28.720307    2565 scope.go:117] "RemoveContainer" containerID="75613eba85684fc2f580fcd70f440f5bceba09b9b53e3a4be2dc33f9b7f3cf77"
Jul 16 23:02:28.720765 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:02:28.720517    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:02:39.726030 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:02:39.725995    2565 scope.go:117] "RemoveContainer" containerID="75613eba85684fc2f580fcd70f440f5bceba09b9b53e3a4be2dc33f9b7f3cf77"
Jul 16 23:02:39.726629 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:02:39.726212    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:02:50.720312 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:02:50.720271    2565 scope.go:117] "RemoveContainer" containerID="75613eba85684fc2f580fcd70f440f5bceba09b9b53e3a4be2dc33f9b7f3cf77"
Jul 16 23:02:50.720771 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:02:50.720473    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:03:05.720986 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:05.720905    2565 scope.go:117] "RemoveContainer" containerID="75613eba85684fc2f580fcd70f440f5bceba09b9b53e3a4be2dc33f9b7f3cf77"
Jul 16 23:03:06.829781 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:06.829748    2565 generic.go:356] "Generic (PLEG): container finished" podID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerID="a173dfc8d957dab55097910fdf668ca01e8b9b6e9b3f0ad4f8f22f752861973b" exitCode=1
Jul 16 23:03:06.830169 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:06.829827    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" event={"ID":"e83026bc-2f49-4005-afa7-29b1b1ec22c5","Type":"ContainerDied","Data":"a173dfc8d957dab55097910fdf668ca01e8b9b6e9b3f0ad4f8f22f752861973b"}
Jul 16 23:03:06.830169 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:06.829897    2565 scope.go:117] "RemoveContainer" containerID="75613eba85684fc2f580fcd70f440f5bceba09b9b53e3a4be2dc33f9b7f3cf77"
Jul 16 23:03:06.830308 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:06.830177    2565 scope.go:117] "RemoveContainer" containerID="a173dfc8d957dab55097910fdf668ca01e8b9b6e9b3f0ad4f8f22f752861973b"
Jul 16 23:03:06.830429 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:03:06.830407    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:03:08.304346 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:08.304237    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:03:08.304346 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:08.304328    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:03:08.304753 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:08.304665    2565 scope.go:117] "RemoveContainer" containerID="a173dfc8d957dab55097910fdf668ca01e8b9b6e9b3f0ad4f8f22f752861973b"
Jul 16 23:03:08.304859 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:03:08.304838    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:03:12.237724 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:12.237687    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm"]
Jul 16 23:03:12.241235 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:12.241215    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm"
Jul 16 23:03:12.243710 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:12.243688    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:03:12.250346 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:12.250319    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm"]
Jul 16 23:03:12.332030 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:12.331990    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7edea1f0-5c45-4f6d-8954-3f721dfb8db6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-hffhm\" (UID: \"7edea1f0-5c45-4f6d-8954-3f721dfb8db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm"
Jul 16 23:03:12.432646 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:12.432602    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7edea1f0-5c45-4f6d-8954-3f721dfb8db6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-hffhm\" (UID: \"7edea1f0-5c45-4f6d-8954-3f721dfb8db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm"
Jul 16 23:03:12.433332 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:12.433305    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7edea1f0-5c45-4f6d-8954-3f721dfb8db6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-hffhm\" (UID: \"7edea1f0-5c45-4f6d-8954-3f721dfb8db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm"
Jul 16 23:03:12.554419 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:12.554315    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm"
Jul 16 23:03:12.686967 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:12.686928    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm"]
Jul 16 23:03:12.691194 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:03:12.691172    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7edea1f0_5c45_4f6d_8954_3f721dfb8db6.slice/crio-cd85ef774aa03049302c96ec3d6d135350df1adbe1809f0e21aceba590ab7d19 WatchSource:0}: Error finding container cd85ef774aa03049302c96ec3d6d135350df1adbe1809f0e21aceba590ab7d19: Status 404 returned error can't find the container with id cd85ef774aa03049302c96ec3d6d135350df1adbe1809f0e21aceba590ab7d19
Jul 16 23:03:12.863926 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:12.863893    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm" event={"ID":"7edea1f0-5c45-4f6d-8954-3f721dfb8db6","Type":"ContainerStarted","Data":"cd85ef774aa03049302c96ec3d6d135350df1adbe1809f0e21aceba590ab7d19"}
Jul 16 23:03:13.870113 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:13.870080    2565 generic.go:356] "Generic (PLEG): container finished" podID="7edea1f0-5c45-4f6d-8954-3f721dfb8db6" containerID="18ce9c39fef3edc40f36eff05ff798b4bd8036e9a40880b8ae5755feeae87d76" exitCode=7
Jul 16 23:03:13.870520 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:13.870152    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm" event={"ID":"7edea1f0-5c45-4f6d-8954-3f721dfb8db6","Type":"ContainerDied","Data":"18ce9c39fef3edc40f36eff05ff798b4bd8036e9a40880b8ae5755feeae87d76"}
Jul 16 23:03:13.870520 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:13.870414    2565 scope.go:117] "RemoveContainer" containerID="18ce9c39fef3edc40f36eff05ff798b4bd8036e9a40880b8ae5755feeae87d76"
Jul 16 23:03:14.877438 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:14.877407    2565 generic.go:356] "Generic (PLEG): container finished" podID="7edea1f0-5c45-4f6d-8954-3f721dfb8db6" containerID="f2c1bdb706acc875bdfd2bb05a476b7b04b1bd77387c01d8886be5c4044c052f" exitCode=7
Jul 16 23:03:14.877438 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:14.877441    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm" event={"ID":"7edea1f0-5c45-4f6d-8954-3f721dfb8db6","Type":"ContainerDied","Data":"f2c1bdb706acc875bdfd2bb05a476b7b04b1bd77387c01d8886be5c4044c052f"}
Jul 16 23:03:14.877898 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:14.877485    2565 scope.go:117] "RemoveContainer" containerID="18ce9c39fef3edc40f36eff05ff798b4bd8036e9a40880b8ae5755feeae87d76"
Jul 16 23:03:14.877898 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:14.877677    2565 scope.go:117] "RemoveContainer" containerID="f2c1bdb706acc875bdfd2bb05a476b7b04b1bd77387c01d8886be5c4044c052f"
Jul 16 23:03:14.877981 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:03:14.877922    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-hffhm_odh-ai-gateway-infra(7edea1f0-5c45-4f6d-8954-3f721dfb8db6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm" podUID="7edea1f0-5c45-4f6d-8954-3f721dfb8db6"
Jul 16 23:03:15.884317 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:15.884278    2565 scope.go:117] "RemoveContainer" containerID="f2c1bdb706acc875bdfd2bb05a476b7b04b1bd77387c01d8886be5c4044c052f"
Jul 16 23:03:15.884823 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:03:15.884552    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-hffhm_odh-ai-gateway-infra(7edea1f0-5c45-4f6d-8954-3f721dfb8db6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm" podUID="7edea1f0-5c45-4f6d-8954-3f721dfb8db6"
Jul 16 23:03:21.720491 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:21.720453    2565 scope.go:117] "RemoveContainer" containerID="a173dfc8d957dab55097910fdf668ca01e8b9b6e9b3f0ad4f8f22f752861973b"
Jul 16 23:03:21.720866 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:03:21.720699    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:03:28.720768 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:28.720721    2565 scope.go:117] "RemoveContainer" containerID="f2c1bdb706acc875bdfd2bb05a476b7b04b1bd77387c01d8886be5c4044c052f"
Jul 16 23:03:29.950053 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:29.950017    2565 generic.go:356] "Generic (PLEG): container finished" podID="7edea1f0-5c45-4f6d-8954-3f721dfb8db6" containerID="d18ec546ca1d41e76a5bfa1673b844d0a9c94f15d1214515bdfa74e535883d5a" exitCode=7
Jul 16 23:03:29.950482 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:29.950100    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm" event={"ID":"7edea1f0-5c45-4f6d-8954-3f721dfb8db6","Type":"ContainerDied","Data":"d18ec546ca1d41e76a5bfa1673b844d0a9c94f15d1214515bdfa74e535883d5a"}
Jul 16 23:03:29.950482 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:29.950174    2565 scope.go:117] "RemoveContainer" containerID="f2c1bdb706acc875bdfd2bb05a476b7b04b1bd77387c01d8886be5c4044c052f"
Jul 16 23:03:29.950608 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:29.950534    2565 scope.go:117] "RemoveContainer" containerID="d18ec546ca1d41e76a5bfa1673b844d0a9c94f15d1214515bdfa74e535883d5a"
Jul 16 23:03:29.950789 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:03:29.950764    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-hffhm_odh-ai-gateway-infra(7edea1f0-5c45-4f6d-8954-3f721dfb8db6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm" podUID="7edea1f0-5c45-4f6d-8954-3f721dfb8db6"
Jul 16 23:03:30.996076 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:30.996045    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm"]
Jul 16 23:03:31.131589 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:31.131564    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm"
Jul 16 23:03:31.206915 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:31.206885    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7edea1f0-5c45-4f6d-8954-3f721dfb8db6-maas-api-service-ca\") pod \"7edea1f0-5c45-4f6d-8954-3f721dfb8db6\" (UID: \"7edea1f0-5c45-4f6d-8954-3f721dfb8db6\") "
Jul 16 23:03:31.207277 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:31.207234    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7edea1f0-5c45-4f6d-8954-3f721dfb8db6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7edea1f0-5c45-4f6d-8954-3f721dfb8db6" (UID: "7edea1f0-5c45-4f6d-8954-3f721dfb8db6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:03:31.308595 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:31.308519    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7edea1f0-5c45-4f6d-8954-3f721dfb8db6-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:03:31.962409 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:31.962370    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm" event={"ID":"7edea1f0-5c45-4f6d-8954-3f721dfb8db6","Type":"ContainerDied","Data":"cd85ef774aa03049302c96ec3d6d135350df1adbe1809f0e21aceba590ab7d19"}
Jul 16 23:03:31.962601 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:31.962440    2565 scope.go:117] "RemoveContainer" containerID="d18ec546ca1d41e76a5bfa1673b844d0a9c94f15d1214515bdfa74e535883d5a"
Jul 16 23:03:31.962601 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:31.962385    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm"
Jul 16 23:03:31.991935 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:31.991862    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm"]
Jul 16 23:03:31.994857 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:31.994833    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hffhm"]
Jul 16 23:03:33.726299 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:33.726266    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7edea1f0-5c45-4f6d-8954-3f721dfb8db6" path="/var/lib/kubelet/pods/7edea1f0-5c45-4f6d-8954-3f721dfb8db6/volumes"
Jul 16 23:03:33.726916 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:33.726901    2565 scope.go:117] "RemoveContainer" containerID="a173dfc8d957dab55097910fdf668ca01e8b9b6e9b3f0ad4f8f22f752861973b"
Jul 16 23:03:33.727074 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:03:33.727058    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:03:45.726445 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:03:45.726414    2565 scope.go:117] "RemoveContainer" containerID="a173dfc8d957dab55097910fdf668ca01e8b9b6e9b3f0ad4f8f22f752861973b"
Jul 16 23:03:45.726923 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:03:45.726597    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:04:01.720942 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:01.720904    2565 scope.go:117] "RemoveContainer" containerID="a173dfc8d957dab55097910fdf668ca01e8b9b6e9b3f0ad4f8f22f752861973b"
Jul 16 23:04:01.721361 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:04:01.721149    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:04:15.720645 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:15.720613    2565 scope.go:117] "RemoveContainer" containerID="a173dfc8d957dab55097910fdf668ca01e8b9b6e9b3f0ad4f8f22f752861973b"
Jul 16 23:04:15.721052 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:04:15.720802    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3933c193-55974dd67c-j2tdc_odh-ai-gateway-infra(e83026bc-2f49-4005-afa7-29b1b1ec22c5)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5"
Jul 16 23:04:22.271748 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:22.271704    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"]
Jul 16 23:04:22.272391 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:22.271920    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt" podUID="dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" containerName="istio-proxy" containerID="cri-o://65ec4118d40daddd735899231d9685b60d1c25eeb7c988ecd169a5c887630fa8" gracePeriod=30
Jul 16 23:04:24.203363 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:24.203324    2565 patch_prober.go:28] interesting pod/e2e-ait-3933c193-openshift-default-57678d7499-n74zt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 23:04:24.203766 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:24.203384    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt" podUID="dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 23:04:27.527468 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.527439    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:04:27.637725 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.637695    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgmnq\" (UniqueName: \"kubernetes.io/projected/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-kube-api-access-pgmnq\") pod \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") "
Jul 16 23:04:27.637725 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.637728    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-token\") pod \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") "
Jul 16 23:04:27.637956 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.637756    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-workload-certs\") pod \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") "
Jul 16 23:04:27.637956 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.637781    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-workload-socket\") pod \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") "
Jul 16 23:04:27.637956 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.637825    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istiod-ca-cert\") pod \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") "
Jul 16 23:04:27.637956 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.637903    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-credential-socket\") pod \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") "
Jul 16 23:04:27.637956 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.637944    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-data\") pod \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") "
Jul 16 23:04:27.638285 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.637975    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-podinfo\") pod \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") "
Jul 16 23:04:27.638285 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.638001    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-envoy\") pod \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\" (UID: \"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6\") "
Jul 16 23:04:27.638285 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.638015    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" (UID: "dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:04:27.638285 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.638182    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" (UID: "dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:04:27.638285 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.638204    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" (UID: "dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:04:27.638546 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.638323    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-credential-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:04:27.638546 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.638341    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-workload-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:04:27.638546 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.638355    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-workload-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:04:27.638546 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.638423    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" (UID: "dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:04:27.638974 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.638940    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-data" (OuterVolumeSpecName: "istio-data") pod "dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" (UID: "dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:04:27.640358 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.640312    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-token" (OuterVolumeSpecName: "istio-token") pod "dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" (UID: "dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:04:27.640358 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.640336    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" (UID: "dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 23:04:27.640485 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.640396    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" (UID: "dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:04:27.640485 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.640411    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-kube-api-access-pgmnq" (OuterVolumeSpecName: "kube-api-access-pgmnq") pod "dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" (UID: "dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6"). InnerVolumeSpecName "kube-api-access-pgmnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:04:27.738983 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.738953    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-data\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:04:27.738983 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.738983    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-podinfo\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:04:27.739126 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.738991    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-envoy\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:04:27.739126 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.739000    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgmnq\" (UniqueName: \"kubernetes.io/projected/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-kube-api-access-pgmnq\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:04:27.739126 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.739009    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istio-token\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:04:27.739126 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:27.739018    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6-istiod-ca-cert\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:04:28.217190 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.217143    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"]
Jul 16 23:04:28.230797 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.230765    2565 generic.go:356] "Generic (PLEG): container finished" podID="dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" containerID="65ec4118d40daddd735899231d9685b60d1c25eeb7c988ecd169a5c887630fa8" exitCode=0
Jul 16 23:04:28.230976 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.230814    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"
Jul 16 23:04:28.230976 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.230819    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt" event={"ID":"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6","Type":"ContainerDied","Data":"65ec4118d40daddd735899231d9685b60d1c25eeb7c988ecd169a5c887630fa8"}
Jul 16 23:04:28.230976 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.230863    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt" event={"ID":"dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6","Type":"ContainerDied","Data":"a040e684b1ec93bea3c4fc560d2e78d5dc518dfb15495042d5ef812ac3e541ce"}
Jul 16 23:04:28.230976 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.230883    2565 scope.go:117] "RemoveContainer" containerID="65ec4118d40daddd735899231d9685b60d1c25eeb7c988ecd169a5c887630fa8"
Jul 16 23:04:28.240282 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.240240    2565 scope.go:117] "RemoveContainer" containerID="4b749cd434c678c15259dd6f7f34415c1e7c5803d44c641f83b947c17ce08c2f"
Jul 16 23:04:28.250294 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.250166    2565 scope.go:117] "RemoveContainer" containerID="65ec4118d40daddd735899231d9685b60d1c25eeb7c988ecd169a5c887630fa8"
Jul 16 23:04:28.250666 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:04:28.250521    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ec4118d40daddd735899231d9685b60d1c25eeb7c988ecd169a5c887630fa8\": container with ID starting with 65ec4118d40daddd735899231d9685b60d1c25eeb7c988ecd169a5c887630fa8 not found: ID does not exist" containerID="65ec4118d40daddd735899231d9685b60d1c25eeb7c988ecd169a5c887630fa8"
Jul 16 23:04:28.250666 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.250564    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ec4118d40daddd735899231d9685b60d1c25eeb7c988ecd169a5c887630fa8"} err="failed to get container status \"65ec4118d40daddd735899231d9685b60d1c25eeb7c988ecd169a5c887630fa8\": rpc error: code = NotFound desc = could not find container \"65ec4118d40daddd735899231d9685b60d1c25eeb7c988ecd169a5c887630fa8\": container with ID starting with 65ec4118d40daddd735899231d9685b60d1c25eeb7c988ecd169a5c887630fa8 not found: ID does not exist"
Jul 16 23:04:28.250666 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.250590    2565 scope.go:117] "RemoveContainer" containerID="4b749cd434c678c15259dd6f7f34415c1e7c5803d44c641f83b947c17ce08c2f"
Jul 16 23:04:28.251357 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:04:28.251326    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b749cd434c678c15259dd6f7f34415c1e7c5803d44c641f83b947c17ce08c2f\": container with ID starting with 4b749cd434c678c15259dd6f7f34415c1e7c5803d44c641f83b947c17ce08c2f not found: ID does not exist" containerID="4b749cd434c678c15259dd6f7f34415c1e7c5803d44c641f83b947c17ce08c2f"
Jul 16 23:04:28.251485 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.251363    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b749cd434c678c15259dd6f7f34415c1e7c5803d44c641f83b947c17ce08c2f"} err="failed to get container status \"4b749cd434c678c15259dd6f7f34415c1e7c5803d44c641f83b947c17ce08c2f\": rpc error: code = NotFound desc = could not find container \"4b749cd434c678c15259dd6f7f34415c1e7c5803d44c641f83b947c17ce08c2f\": container with ID starting with 4b749cd434c678c15259dd6f7f34415c1e7c5803d44c641f83b947c17ce08c2f not found: ID does not exist"
Jul 16 23:04:28.257177 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.257152    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"]
Jul 16 23:04:28.261353 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.261330    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3933c193-openshift-default-57678d7499-n74zt"]
Jul 16 23:04:28.360236 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.360203    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:04:28.445637 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.445607    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83026bc-2f49-4005-afa7-29b1b1ec22c5-openshift-service-ca\") pod \"e83026bc-2f49-4005-afa7-29b1b1ec22c5\" (UID: \"e83026bc-2f49-4005-afa7-29b1b1ec22c5\") "
Jul 16 23:04:28.445828 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.445648    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnpf9\" (UniqueName: \"kubernetes.io/projected/e83026bc-2f49-4005-afa7-29b1b1ec22c5-kube-api-access-tnpf9\") pod \"e83026bc-2f49-4005-afa7-29b1b1ec22c5\" (UID: \"e83026bc-2f49-4005-afa7-29b1b1ec22c5\") "
Jul 16 23:04:28.445828 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.445697    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e83026bc-2f49-4005-afa7-29b1b1ec22c5-maas-api-tls\") pod \"e83026bc-2f49-4005-afa7-29b1b1ec22c5\" (UID: \"e83026bc-2f49-4005-afa7-29b1b1ec22c5\") "
Jul 16 23:04:28.446060 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.446035    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e83026bc-2f49-4005-afa7-29b1b1ec22c5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e83026bc-2f49-4005-afa7-29b1b1ec22c5" (UID: "e83026bc-2f49-4005-afa7-29b1b1ec22c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:04:28.447873 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.447851    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e83026bc-2f49-4005-afa7-29b1b1ec22c5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e83026bc-2f49-4005-afa7-29b1b1ec22c5" (UID: "e83026bc-2f49-4005-afa7-29b1b1ec22c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:04:28.447939 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.447910    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e83026bc-2f49-4005-afa7-29b1b1ec22c5-kube-api-access-tnpf9" (OuterVolumeSpecName: "kube-api-access-tnpf9") pod "e83026bc-2f49-4005-afa7-29b1b1ec22c5" (UID: "e83026bc-2f49-4005-afa7-29b1b1ec22c5"). InnerVolumeSpecName "kube-api-access-tnpf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:04:28.546922 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.546850    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83026bc-2f49-4005-afa7-29b1b1ec22c5-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:04:28.546922 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.546877    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnpf9\" (UniqueName: \"kubernetes.io/projected/e83026bc-2f49-4005-afa7-29b1b1ec22c5-kube-api-access-tnpf9\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:04:28.546922 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:28.546887    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e83026bc-2f49-4005-afa7-29b1b1ec22c5-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:04:29.237965 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:29.237931    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc" event={"ID":"e83026bc-2f49-4005-afa7-29b1b1ec22c5","Type":"ContainerDied","Data":"3c17d4631826cd0d91804c8c4c276611046706169d1712367d7fad55a41a2dc5"}
Jul 16 23:04:29.238156 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:29.237996    2565 scope.go:117] "RemoveContainer" containerID="a173dfc8d957dab55097910fdf668ca01e8b9b6e9b3f0ad4f8f22f752861973b"
Jul 16 23:04:29.238156 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:29.237942    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"
Jul 16 23:04:29.263127 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:29.263104    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"]
Jul 16 23:04:29.267104 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:29.267082    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3933c193-55974dd67c-j2tdc"]
Jul 16 23:04:29.726213 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:29.726176    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" path="/var/lib/kubelet/pods/dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6/volumes"
Jul 16 23:04:29.726629 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:29.726614    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" path="/var/lib/kubelet/pods/e83026bc-2f49-4005-afa7-29b1b1ec22c5/volumes"
Jul 16 23:04:35.315442 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315351    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"]
Jul 16 23:04:35.315793 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315712    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerName="maas-api"
Jul 16 23:04:35.315793 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315722    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerName="maas-api"
Jul 16 23:04:35.315793 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315731    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7edea1f0-5c45-4f6d-8954-3f721dfb8db6" containerName="revoke-keys"
Jul 16 23:04:35.315793 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315736    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7edea1f0-5c45-4f6d-8954-3f721dfb8db6" containerName="revoke-keys"
Jul 16 23:04:35.315793 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315745    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" containerName="istio-proxy"
Jul 16 23:04:35.315793 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315750    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" containerName="istio-proxy"
Jul 16 23:04:35.315793 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315758    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerName="maas-api"
Jul 16 23:04:35.315793 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315762    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerName="maas-api"
Jul 16 23:04:35.315793 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315770    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7edea1f0-5c45-4f6d-8954-3f721dfb8db6" containerName="revoke-keys"
Jul 16 23:04:35.315793 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315775    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7edea1f0-5c45-4f6d-8954-3f721dfb8db6" containerName="revoke-keys"
Jul 16 23:04:35.315793 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315784    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" containerName="istio-proxy"
Jul 16 23:04:35.315793 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315789    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" containerName="istio-proxy"
Jul 16 23:04:35.315793 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315796    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerName="maas-api"
Jul 16 23:04:35.315793 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315801    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerName="maas-api"
Jul 16 23:04:35.316196 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315807    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerName="maas-api"
Jul 16 23:04:35.316196 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315812    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerName="maas-api"
Jul 16 23:04:35.316196 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315817    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7edea1f0-5c45-4f6d-8954-3f721dfb8db6" containerName="revoke-keys"
Jul 16 23:04:35.316196 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315822    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7edea1f0-5c45-4f6d-8954-3f721dfb8db6" containerName="revoke-keys"
Jul 16 23:04:35.316196 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315832    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerName="maas-api"
Jul 16 23:04:35.316196 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315837    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerName="maas-api"
Jul 16 23:04:35.316196 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315887    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7edea1f0-5c45-4f6d-8954-3f721dfb8db6" containerName="revoke-keys"
Jul 16 23:04:35.316196 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315894    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7edea1f0-5c45-4f6d-8954-3f721dfb8db6" containerName="revoke-keys"
Jul 16 23:04:35.316196 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315901    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerName="maas-api"
Jul 16 23:04:35.316196 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315906    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerName="maas-api"
Jul 16 23:04:35.316196 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315911    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerName="maas-api"
Jul 16 23:04:35.316196 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.315917    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" containerName="istio-proxy"
Jul 16 23:04:35.320430 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.320409    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.324476 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.324442    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-06b30c8f-gw-openshift-default-dockercfg-bv9gd\""
Jul 16 23:04:35.343699 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.343670    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"]
Jul 16 23:04:35.410686 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.410655    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a970c0-3918-4f54-83ee-c560de2c1929-istio-podinfo\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.410869 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.410693    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-workload-certs\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.410869 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.410714    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-istio-envoy\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.410869 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.410734    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl4c4\" (UniqueName: \"kubernetes.io/projected/53a970c0-3918-4f54-83ee-c560de2c1929-kube-api-access-hl4c4\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.410869 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.410797    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-credential-socket\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.410869 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.410817    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a970c0-3918-4f54-83ee-c560de2c1929-istiod-ca-cert\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.410869 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.410847    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-workload-socket\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.410869 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.410866    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-istio-data\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.411183 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.411003    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53a970c0-3918-4f54-83ee-c560de2c1929-istio-token\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.511806 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.511773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53a970c0-3918-4f54-83ee-c560de2c1929-istio-token\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.511967 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.511824    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a970c0-3918-4f54-83ee-c560de2c1929-istio-podinfo\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.511967 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.511857    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-workload-certs\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.511967 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.511882    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-istio-envoy\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.511967 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.511914    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl4c4\" (UniqueName: \"kubernetes.io/projected/53a970c0-3918-4f54-83ee-c560de2c1929-kube-api-access-hl4c4\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.511967 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.511942    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-credential-socket\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.512286 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.511971    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a970c0-3918-4f54-83ee-c560de2c1929-istiod-ca-cert\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.512286 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.512017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-workload-socket\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.512286 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.512038    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-istio-data\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.512455 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.512295    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-workload-certs\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.512455 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.512428    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-credential-socket\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.512544 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.512452    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-workload-socket\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.512544 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.512469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-istio-data\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.512747 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.512723    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a970c0-3918-4f54-83ee-c560de2c1929-istiod-ca-cert\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.514330 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.514310    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-istio-envoy\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.514500 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.514486    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a970c0-3918-4f54-83ee-c560de2c1929-istio-podinfo\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.525614 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.525583    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53a970c0-3918-4f54-83ee-c560de2c1929-istio-token\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.525823 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.525804    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl4c4\" (UniqueName: \"kubernetes.io/projected/53a970c0-3918-4f54-83ee-c560de2c1929-kube-api-access-hl4c4\") pod \"e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") " pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.630773 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.630733    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:35.783929 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.783891    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"]
Jul 16 23:04:35.787766 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:04:35.787742    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a970c0_3918_4f54_83ee_c560de2c1929.slice/crio-ad0fe3725786116be2ce562a3c0533c7843095084ca2a5f005a25e3a7df37d4c WatchSource:0}: Error finding container ad0fe3725786116be2ce562a3c0533c7843095084ca2a5f005a25e3a7df37d4c: Status 404 returned error can't find the container with id ad0fe3725786116be2ce562a3c0533c7843095084ca2a5f005a25e3a7df37d4c
Jul 16 23:04:35.789996 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.789962    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:04:35.790048 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.790036    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:04:35.790081 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:35.790067    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:04:36.271950 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:36.271912    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm" event={"ID":"53a970c0-3918-4f54-83ee-c560de2c1929","Type":"ContainerStarted","Data":"b99690bdd28ee5e684757bc88c12a421a73b11d36bece0453374b771d7c7e858"}
Jul 16 23:04:36.272138 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:36.271967    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm" event={"ID":"53a970c0-3918-4f54-83ee-c560de2c1929","Type":"ContainerStarted","Data":"ad0fe3725786116be2ce562a3c0533c7843095084ca2a5f005a25e3a7df37d4c"}
Jul 16 23:04:36.631051 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:36.631015    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:36.635821 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:36.635793    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:36.668412 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:36.668363    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm" podStartSLOduration=1.668346592 podStartE2EDuration="1.668346592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:04:36.313336927 +0000 UTC m=+1887.061820999" watchObservedRunningTime="2026-07-16 23:04:36.668346592 +0000 UTC m=+1887.416830663"
Jul 16 23:04:36.775888 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:36.775851    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"]
Jul 16 23:04:36.776313 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:36.776299    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc6a5e9c-dcf6-403b-a905-aeecfcedc1e6" containerName="istio-proxy"
Jul 16 23:04:36.776375 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:36.776318    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7edea1f0-5c45-4f6d-8954-3f721dfb8db6" containerName="revoke-keys"
Jul 16 23:04:36.776375 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:36.776325    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerName="maas-api"
Jul 16 23:04:36.776375 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:36.776331    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e83026bc-2f49-4005-afa7-29b1b1ec22c5" containerName="maas-api"
Jul 16 23:04:36.779376 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:36.779357    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:04:36.782628 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:36.782611    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-06b30c8f-serving-cert\""
Jul 16 23:04:36.801407 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:36.801384    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"]
Jul 16 23:04:36.923637 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:36.923546    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f142e0-b44b-44e5-b87b-269244a1ac18-openshift-service-ca\") pod \"maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd\" (UID: \"20f142e0-b44b-44e5-b87b-269244a1ac18\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:04:36.923637 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:36.923610    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f142e0-b44b-44e5-b87b-269244a1ac18-maas-api-tls\") pod \"maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd\" (UID: \"20f142e0-b44b-44e5-b87b-269244a1ac18\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:04:36.923816 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:36.923672    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp4zj\" (UniqueName: \"kubernetes.io/projected/20f142e0-b44b-44e5-b87b-269244a1ac18-kube-api-access-vp4zj\") pod \"maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd\" (UID: \"20f142e0-b44b-44e5-b87b-269244a1ac18\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:04:37.024472 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:37.024433    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f142e0-b44b-44e5-b87b-269244a1ac18-openshift-service-ca\") pod \"maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd\" (UID: \"20f142e0-b44b-44e5-b87b-269244a1ac18\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:04:37.024627 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:37.024563    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f142e0-b44b-44e5-b87b-269244a1ac18-maas-api-tls\") pod \"maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd\" (UID: \"20f142e0-b44b-44e5-b87b-269244a1ac18\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:04:37.024693 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:37.024674    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp4zj\" (UniqueName: \"kubernetes.io/projected/20f142e0-b44b-44e5-b87b-269244a1ac18-kube-api-access-vp4zj\") pod \"maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd\" (UID: \"20f142e0-b44b-44e5-b87b-269244a1ac18\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:04:37.024921 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:04:37.024720    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-serving-cert: secret "maas-api-e2e-migrate-06b30c8f-serving-cert" not found
Jul 16 23:04:37.024921 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:04:37.024788    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20f142e0-b44b-44e5-b87b-269244a1ac18-maas-api-tls podName:20f142e0-b44b-44e5-b87b-269244a1ac18 nodeName:}" failed. No retries permitted until 2026-07-16 23:04:37.524772229 +0000 UTC m=+1888.273256277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/20f142e0-b44b-44e5-b87b-269244a1ac18-maas-api-tls") pod "maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd" (UID: "20f142e0-b44b-44e5-b87b-269244a1ac18") : secret "maas-api-e2e-migrate-06b30c8f-serving-cert" not found
Jul 16 23:04:37.025129 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:37.025112    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f142e0-b44b-44e5-b87b-269244a1ac18-openshift-service-ca\") pod \"maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd\" (UID: \"20f142e0-b44b-44e5-b87b-269244a1ac18\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:04:37.038694 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:37.038664    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp4zj\" (UniqueName: \"kubernetes.io/projected/20f142e0-b44b-44e5-b87b-269244a1ac18-kube-api-access-vp4zj\") pod \"maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd\" (UID: \"20f142e0-b44b-44e5-b87b-269244a1ac18\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:04:37.276405 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:37.276321    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:37.277224 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:37.277200    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:37.529342 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:37.529233    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f142e0-b44b-44e5-b87b-269244a1ac18-maas-api-tls\") pod \"maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd\" (UID: \"20f142e0-b44b-44e5-b87b-269244a1ac18\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:04:37.531709 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:37.531687    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f142e0-b44b-44e5-b87b-269244a1ac18-maas-api-tls\") pod \"maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd\" (UID: \"20f142e0-b44b-44e5-b87b-269244a1ac18\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:04:37.690014 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:37.689978    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:04:37.838096 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:37.838070    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"]
Jul 16 23:04:37.840665 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:04:37.840639    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f142e0_b44b_44e5_b87b_269244a1ac18.slice/crio-3414e39e04461b76af3c06d21264499454da05655dbaf91c40e94a237ba837ff WatchSource:0}: Error finding container 3414e39e04461b76af3c06d21264499454da05655dbaf91c40e94a237ba837ff: Status 404 returned error can't find the container with id 3414e39e04461b76af3c06d21264499454da05655dbaf91c40e94a237ba837ff
Jul 16 23:04:38.281534 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:38.281502    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd" event={"ID":"20f142e0-b44b-44e5-b87b-269244a1ac18","Type":"ContainerStarted","Data":"3414e39e04461b76af3c06d21264499454da05655dbaf91c40e94a237ba837ff"}
Jul 16 23:04:39.288654 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:39.288617    2565 generic.go:356] "Generic (PLEG): container finished" podID="20f142e0-b44b-44e5-b87b-269244a1ac18" containerID="a8d69d1b3ad902e6565cf855d809a526f371905928907368bceab0b6710719a3" exitCode=1
Jul 16 23:04:39.289145 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:39.288686    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd" event={"ID":"20f142e0-b44b-44e5-b87b-269244a1ac18","Type":"ContainerDied","Data":"a8d69d1b3ad902e6565cf855d809a526f371905928907368bceab0b6710719a3"}
Jul 16 23:04:39.289145 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:39.289001    2565 scope.go:117] "RemoveContainer" containerID="a8d69d1b3ad902e6565cf855d809a526f371905928907368bceab0b6710719a3"
Jul 16 23:04:40.295110 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:40.295024    2565 generic.go:356] "Generic (PLEG): container finished" podID="20f142e0-b44b-44e5-b87b-269244a1ac18" containerID="23c63772df0f5b89f073736da17bde4865ceb290a0de902f3939a95770992777" exitCode=1
Jul 16 23:04:40.295110 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:40.295066    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd" event={"ID":"20f142e0-b44b-44e5-b87b-269244a1ac18","Type":"ContainerDied","Data":"23c63772df0f5b89f073736da17bde4865ceb290a0de902f3939a95770992777"}
Jul 16 23:04:40.295580 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:40.295136    2565 scope.go:117] "RemoveContainer" containerID="a8d69d1b3ad902e6565cf855d809a526f371905928907368bceab0b6710719a3"
Jul 16 23:04:40.295580 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:40.295383    2565 scope.go:117] "RemoveContainer" containerID="23c63772df0f5b89f073736da17bde4865ceb290a0de902f3939a95770992777"
Jul 16 23:04:40.295671 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:04:40.295625    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd_odh-ai-gateway-infra(20f142e0-b44b-44e5-b87b-269244a1ac18)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18"
Jul 16 23:04:41.141213 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:41.141164    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl"]
Jul 16 23:04:41.146691 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:41.146655    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl"
Jul 16 23:04:41.151137 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:41.151109    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:04:41.170464 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:41.170431    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl"]
Jul 16 23:04:41.264957 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:41.264915    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cdc0d23-d250-45db-a0da-b8b007522cf7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl\" (UID: \"3cdc0d23-d250-45db-a0da-b8b007522cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl"
Jul 16 23:04:41.302532 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:41.302502    2565 scope.go:117] "RemoveContainer" containerID="23c63772df0f5b89f073736da17bde4865ceb290a0de902f3939a95770992777"
Jul 16 23:04:41.302902 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:04:41.302700    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd_odh-ai-gateway-infra(20f142e0-b44b-44e5-b87b-269244a1ac18)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18"
Jul 16 23:04:41.366363 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:41.366321    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cdc0d23-d250-45db-a0da-b8b007522cf7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl\" (UID: \"3cdc0d23-d250-45db-a0da-b8b007522cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl"
Jul 16 23:04:41.366976 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:41.366955    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cdc0d23-d250-45db-a0da-b8b007522cf7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl\" (UID: \"3cdc0d23-d250-45db-a0da-b8b007522cf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl"
Jul 16 23:04:41.463057 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:41.462969    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl"
Jul 16 23:04:41.625953 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:41.625921    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl"]
Jul 16 23:04:41.627677 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:04:41.627647    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cdc0d23_d250_45db_a0da_b8b007522cf7.slice/crio-c95ec4b69ff8f9ec0888ec5030a42173f6a5bfca43ab13943a93fa5b10ea43b2 WatchSource:0}: Error finding container c95ec4b69ff8f9ec0888ec5030a42173f6a5bfca43ab13943a93fa5b10ea43b2: Status 404 returned error can't find the container with id c95ec4b69ff8f9ec0888ec5030a42173f6a5bfca43ab13943a93fa5b10ea43b2
Jul 16 23:04:42.309023 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:42.308922    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl" event={"ID":"3cdc0d23-d250-45db-a0da-b8b007522cf7","Type":"ContainerStarted","Data":"183dfc6bb59128444cb02fd2267ca2fc515bd4f71d641263cc965802372e14b8"}
Jul 16 23:04:42.309023 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:42.308978    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl" event={"ID":"3cdc0d23-d250-45db-a0da-b8b007522cf7","Type":"ContainerStarted","Data":"c95ec4b69ff8f9ec0888ec5030a42173f6a5bfca43ab13943a93fa5b10ea43b2"}
Jul 16 23:04:42.327539 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:42.327489    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl" podStartSLOduration=1.3274731530000001 podStartE2EDuration="1.327473153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:04:42.325155757 +0000 UTC m=+1893.073639828" watchObservedRunningTime="2026-07-16 23:04:42.327473153 +0000 UTC m=+1893.075957223"
Jul 16 23:04:42.690570 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:42.690529    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:04:42.690934 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:42.690918    2565 scope.go:117] "RemoveContainer" containerID="23c63772df0f5b89f073736da17bde4865ceb290a0de902f3939a95770992777"
Jul 16 23:04:42.691126 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:04:42.691104    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd_odh-ai-gateway-infra(20f142e0-b44b-44e5-b87b-269244a1ac18)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18"
Jul 16 23:04:43.318737 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:43.318698    2565 generic.go:356] "Generic (PLEG): container finished" podID="3cdc0d23-d250-45db-a0da-b8b007522cf7" containerID="183dfc6bb59128444cb02fd2267ca2fc515bd4f71d641263cc965802372e14b8" exitCode=7
Jul 16 23:04:43.319141 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:43.318777    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl" event={"ID":"3cdc0d23-d250-45db-a0da-b8b007522cf7","Type":"ContainerDied","Data":"183dfc6bb59128444cb02fd2267ca2fc515bd4f71d641263cc965802372e14b8"}
Jul 16 23:04:43.319141 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:43.318992    2565 scope.go:117] "RemoveContainer" containerID="183dfc6bb59128444cb02fd2267ca2fc515bd4f71d641263cc965802372e14b8"
Jul 16 23:04:44.326554 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:44.326516    2565 generic.go:356] "Generic (PLEG): container finished" podID="3cdc0d23-d250-45db-a0da-b8b007522cf7" containerID="6aab8d01b594d92002578b335443b7e8de13c7080ebe265a612b78b351d598e6" exitCode=7
Jul 16 23:04:44.326973 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:44.326552    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl" event={"ID":"3cdc0d23-d250-45db-a0da-b8b007522cf7","Type":"ContainerDied","Data":"6aab8d01b594d92002578b335443b7e8de13c7080ebe265a612b78b351d598e6"}
Jul 16 23:04:44.326973 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:44.326617    2565 scope.go:117] "RemoveContainer" containerID="183dfc6bb59128444cb02fd2267ca2fc515bd4f71d641263cc965802372e14b8"
Jul 16 23:04:44.326973 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:44.326789    2565 scope.go:117] "RemoveContainer" containerID="6aab8d01b594d92002578b335443b7e8de13c7080ebe265a612b78b351d598e6"
Jul 16 23:04:44.327117 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:04:44.327016    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl_odh-ai-gateway-infra(3cdc0d23-d250-45db-a0da-b8b007522cf7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl" podUID="3cdc0d23-d250-45db-a0da-b8b007522cf7"
Jul 16 23:04:45.333893 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:45.333860    2565 scope.go:117] "RemoveContainer" containerID="6aab8d01b594d92002578b335443b7e8de13c7080ebe265a612b78b351d598e6"
Jul 16 23:04:45.334308 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:04:45.334048    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl_odh-ai-gateway-infra(3cdc0d23-d250-45db-a0da-b8b007522cf7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl" podUID="3cdc0d23-d250-45db-a0da-b8b007522cf7"
Jul 16 23:04:47.345259 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:47.345222    2565 generic.go:356] "Generic (PLEG): container finished" podID="53a970c0-3918-4f54-83ee-c560de2c1929" containerID="b99690bdd28ee5e684757bc88c12a421a73b11d36bece0453374b771d7c7e858" exitCode=0
Jul 16 23:04:47.345798 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:47.345303    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm" event={"ID":"53a970c0-3918-4f54-83ee-c560de2c1929","Type":"ContainerDied","Data":"b99690bdd28ee5e684757bc88c12a421a73b11d36bece0453374b771d7c7e858"}
Jul 16 23:04:47.345871 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:47.345815    2565 scope.go:117] "RemoveContainer" containerID="b99690bdd28ee5e684757bc88c12a421a73b11d36bece0453374b771d7c7e858"
Jul 16 23:04:47.346861 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:47.346833    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:04:47.346920 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:47.346901    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:04:47.346980 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:47.346958    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:04:47.690981 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:47.690866    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:04:47.691395 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:47.691375    2565 scope.go:117] "RemoveContainer" containerID="23c63772df0f5b89f073736da17bde4865ceb290a0de902f3939a95770992777"
Jul 16 23:04:47.691623 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:04:47.691602    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd_odh-ai-gateway-infra(20f142e0-b44b-44e5-b87b-269244a1ac18)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18"
Jul 16 23:04:48.356747 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:48.356679    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm" event={"ID":"53a970c0-3918-4f54-83ee-c560de2c1929","Type":"ContainerStarted","Data":"73cefee6978f677d79a920cb3c4fff4449e93fe643db5feaeefd5c23047ddb68"}
Jul 16 23:04:48.631915 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:48.631810    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:48.633052 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:48.633023    2565 patch_prober.go:28] interesting pod/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Jul 16 23:04:48.633191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:48.633077    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm" podUID="53a970c0-3918-4f54-83ee-c560de2c1929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Jul 16 23:04:49.632062 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:49.632016    2565 patch_prober.go:28] interesting pod/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Jul 16 23:04:49.632512 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:49.632095    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm" podUID="53a970c0-3918-4f54-83ee-c560de2c1929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Jul 16 23:04:50.631682 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:50.631635    2565 patch_prober.go:28] interesting pod/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Jul 16 23:04:50.631904 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:50.631707    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm" podUID="53a970c0-3918-4f54-83ee-c560de2c1929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Jul 16 23:04:51.632171 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:51.632133    2565 patch_prober.go:28] interesting pod/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Jul 16 23:04:51.632606 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:51.632190    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm" podUID="53a970c0-3918-4f54-83ee-c560de2c1929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Jul 16 23:04:52.276602 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:52.276561    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:52.344068 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:52.344032    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj"]
Jul 16 23:04:52.347641 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:52.347622    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj"
Jul 16 23:04:52.356268 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:52.356217    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj"]
Jul 16 23:04:52.371562 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:52.371531    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bb94817-efe5-424c-9841-4debc1b028d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-c57rj\" (UID: \"5bb94817-efe5-424c-9841-4debc1b028d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj"
Jul 16 23:04:52.472751 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:52.472700    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bb94817-efe5-424c-9841-4debc1b028d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-c57rj\" (UID: \"5bb94817-efe5-424c-9841-4debc1b028d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj"
Jul 16 23:04:52.473385 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:52.473362    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bb94817-efe5-424c-9841-4debc1b028d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-c57rj\" (UID: \"5bb94817-efe5-424c-9841-4debc1b028d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj"
Jul 16 23:04:52.636006 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:52.635976    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:52.636991 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:52.636959    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:04:52.659824 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:52.659788    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj"
Jul 16 23:04:52.825173 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:52.825102    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj"]
Jul 16 23:04:52.827650 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:04:52.827618    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb94817_efe5_424c_9841_4debc1b028d2.slice/crio-951d781371a1612db3626b0087600bd61f9b6a9882cafdf71aa1a2ee3a93202a WatchSource:0}: Error finding container 951d781371a1612db3626b0087600bd61f9b6a9882cafdf71aa1a2ee3a93202a: Status 404 returned error can't find the container with id 951d781371a1612db3626b0087600bd61f9b6a9882cafdf71aa1a2ee3a93202a
Jul 16 23:04:53.388311 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:53.388265    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj" event={"ID":"5bb94817-efe5-424c-9841-4debc1b028d2","Type":"ContainerStarted","Data":"8df4960ec1a817c86c5b8f7800301bb88c1377fdbdac440feb9e6e2947836bc4"}
Jul 16 23:04:53.388581 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:53.388326    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj" event={"ID":"5bb94817-efe5-424c-9841-4debc1b028d2","Type":"ContainerStarted","Data":"951d781371a1612db3626b0087600bd61f9b6a9882cafdf71aa1a2ee3a93202a"}
Jul 16 23:04:53.407796 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:53.407742    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj" podStartSLOduration=1.40772376 podStartE2EDuration="1.40772376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:04:53.404911808 +0000 UTC m=+1904.153395880" watchObservedRunningTime="2026-07-16 23:04:53.40772376 +0000 UTC m=+1904.156207830"
Jul 16 23:04:54.395158 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:54.395119    2565 generic.go:356] "Generic (PLEG): container finished" podID="5bb94817-efe5-424c-9841-4debc1b028d2" containerID="8df4960ec1a817c86c5b8f7800301bb88c1377fdbdac440feb9e6e2947836bc4" exitCode=6
Jul 16 23:04:54.395158 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:54.395153    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj" event={"ID":"5bb94817-efe5-424c-9841-4debc1b028d2","Type":"ContainerDied","Data":"8df4960ec1a817c86c5b8f7800301bb88c1377fdbdac440feb9e6e2947836bc4"}
Jul 16 23:04:54.395741 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:54.395411    2565 scope.go:117] "RemoveContainer" containerID="8df4960ec1a817c86c5b8f7800301bb88c1377fdbdac440feb9e6e2947836bc4"
Jul 16 23:04:55.402103 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:55.402070    2565 generic.go:356] "Generic (PLEG): container finished" podID="5bb94817-efe5-424c-9841-4debc1b028d2" containerID="7bf3c747b344b3a25999162cfe342916119f56daed0c2ed8aa9433a962f767ff" exitCode=6
Jul 16 23:04:55.402541 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:55.402127    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj" event={"ID":"5bb94817-efe5-424c-9841-4debc1b028d2","Type":"ContainerDied","Data":"7bf3c747b344b3a25999162cfe342916119f56daed0c2ed8aa9433a962f767ff"}
Jul 16 23:04:55.402541 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:55.402201    2565 scope.go:117] "RemoveContainer" containerID="8df4960ec1a817c86c5b8f7800301bb88c1377fdbdac440feb9e6e2947836bc4"
Jul 16 23:04:55.402541 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:55.402455    2565 scope.go:117] "RemoveContainer" containerID="7bf3c747b344b3a25999162cfe342916119f56daed0c2ed8aa9433a962f767ff"
Jul 16 23:04:55.402705 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:04:55.402679    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-c57rj_odh-ai-gateway-infra(5bb94817-efe5-424c-9841-4debc1b028d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj" podUID="5bb94817-efe5-424c-9841-4debc1b028d2"
Jul 16 23:04:56.408480 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:56.408450    2565 scope.go:117] "RemoveContainer" containerID="7bf3c747b344b3a25999162cfe342916119f56daed0c2ed8aa9433a962f767ff"
Jul 16 23:04:56.408878 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:04:56.408651    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-c57rj_odh-ai-gateway-infra(5bb94817-efe5-424c-9841-4debc1b028d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj" podUID="5bb94817-efe5-424c-9841-4debc1b028d2"
Jul 16 23:04:57.720831 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:57.720784    2565 scope.go:117] "RemoveContainer" containerID="6aab8d01b594d92002578b335443b7e8de13c7080ebe265a612b78b351d598e6"
Jul 16 23:04:58.420731 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:58.420692    2565 generic.go:356] "Generic (PLEG): container finished" podID="3cdc0d23-d250-45db-a0da-b8b007522cf7" containerID="01ec861571970eee311aca3d55b5780eec5bda6ebdd60853ff44d63283df985e" exitCode=7
Jul 16 23:04:58.420908 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:58.420763    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl" event={"ID":"3cdc0d23-d250-45db-a0da-b8b007522cf7","Type":"ContainerDied","Data":"01ec861571970eee311aca3d55b5780eec5bda6ebdd60853ff44d63283df985e"}
Jul 16 23:04:58.420908 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:58.420835    2565 scope.go:117] "RemoveContainer" containerID="6aab8d01b594d92002578b335443b7e8de13c7080ebe265a612b78b351d598e6"
Jul 16 23:04:58.421136 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:58.421119    2565 scope.go:117] "RemoveContainer" containerID="01ec861571970eee311aca3d55b5780eec5bda6ebdd60853ff44d63283df985e"
Jul 16 23:04:58.421430 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:04:58.421408    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl_odh-ai-gateway-infra(3cdc0d23-d250-45db-a0da-b8b007522cf7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl" podUID="3cdc0d23-d250-45db-a0da-b8b007522cf7"
Jul 16 23:04:59.457552 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:59.457509    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl"]
Jul 16 23:04:59.597549 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:59.597519    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl"
Jul 16 23:04:59.637849 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:59.637810    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cdc0d23-d250-45db-a0da-b8b007522cf7-maas-api-service-ca\") pod \"3cdc0d23-d250-45db-a0da-b8b007522cf7\" (UID: \"3cdc0d23-d250-45db-a0da-b8b007522cf7\") "
Jul 16 23:04:59.638179 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:59.638158    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cdc0d23-d250-45db-a0da-b8b007522cf7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3cdc0d23-d250-45db-a0da-b8b007522cf7" (UID: "3cdc0d23-d250-45db-a0da-b8b007522cf7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:04:59.739897 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:04:59.739774    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cdc0d23-d250-45db-a0da-b8b007522cf7-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:05:00.432940 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:00.432913    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl"
Jul 16 23:05:00.432940 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:00.432928    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl" event={"ID":"3cdc0d23-d250-45db-a0da-b8b007522cf7","Type":"ContainerDied","Data":"c95ec4b69ff8f9ec0888ec5030a42173f6a5bfca43ab13943a93fa5b10ea43b2"}
Jul 16 23:05:00.433214 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:00.432986    2565 scope.go:117] "RemoveContainer" containerID="01ec861571970eee311aca3d55b5780eec5bda6ebdd60853ff44d63283df985e"
Jul 16 23:05:00.463257 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:00.463219    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl"]
Jul 16 23:05:00.469830 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:00.469797    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-mx6vl"]
Jul 16 23:05:00.721317 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:00.721218    2565 scope.go:117] "RemoveContainer" containerID="23c63772df0f5b89f073736da17bde4865ceb290a0de902f3939a95770992777"
Jul 16 23:05:01.439956 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:01.439923    2565 generic.go:356] "Generic (PLEG): container finished" podID="20f142e0-b44b-44e5-b87b-269244a1ac18" containerID="0ac829e4424acb6559f0f1bb0bb1fbdc963a058edab5c005e0b7210c38187771" exitCode=1
Jul 16 23:05:01.440174 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:01.439991    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd" event={"ID":"20f142e0-b44b-44e5-b87b-269244a1ac18","Type":"ContainerDied","Data":"0ac829e4424acb6559f0f1bb0bb1fbdc963a058edab5c005e0b7210c38187771"}
Jul 16 23:05:01.440174 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:01.440052    2565 scope.go:117] "RemoveContainer" containerID="23c63772df0f5b89f073736da17bde4865ceb290a0de902f3939a95770992777"
Jul 16 23:05:01.440519 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:01.440479    2565 scope.go:117] "RemoveContainer" containerID="0ac829e4424acb6559f0f1bb0bb1fbdc963a058edab5c005e0b7210c38187771"
Jul 16 23:05:01.440756 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:05:01.440734    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd_odh-ai-gateway-infra(20f142e0-b44b-44e5-b87b-269244a1ac18)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18"
Jul 16 23:05:01.727567 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:01.727476    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cdc0d23-d250-45db-a0da-b8b007522cf7" path="/var/lib/kubelet/pods/3cdc0d23-d250-45db-a0da-b8b007522cf7/volumes"
Jul 16 23:05:02.690373 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:02.690328    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:05:02.690784 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:02.690765    2565 scope.go:117] "RemoveContainer" containerID="0ac829e4424acb6559f0f1bb0bb1fbdc963a058edab5c005e0b7210c38187771"
Jul 16 23:05:02.691004 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:05:02.690979    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd_odh-ai-gateway-infra(20f142e0-b44b-44e5-b87b-269244a1ac18)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18"
Jul 16 23:05:07.690656 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:07.690619    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:05:07.691143 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:07.691123    2565 scope.go:117] "RemoveContainer" containerID="0ac829e4424acb6559f0f1bb0bb1fbdc963a058edab5c005e0b7210c38187771"
Jul 16 23:05:07.691388 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:05:07.691366    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd_odh-ai-gateway-infra(20f142e0-b44b-44e5-b87b-269244a1ac18)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18"
Jul 16 23:05:09.722773 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:09.722741    2565 scope.go:117] "RemoveContainer" containerID="7bf3c747b344b3a25999162cfe342916119f56daed0c2ed8aa9433a962f767ff"
Jul 16 23:05:10.489981 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:10.489949    2565 generic.go:356] "Generic (PLEG): container finished" podID="5bb94817-efe5-424c-9841-4debc1b028d2" containerID="ab731d1e8a2bd41403d5170f0efbb31251bf4ed422a160eeb9c952520052042b" exitCode=6
Jul 16 23:05:10.490178 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:10.489995    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj" event={"ID":"5bb94817-efe5-424c-9841-4debc1b028d2","Type":"ContainerDied","Data":"ab731d1e8a2bd41403d5170f0efbb31251bf4ed422a160eeb9c952520052042b"}
Jul 16 23:05:10.490178 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:10.490042    2565 scope.go:117] "RemoveContainer" containerID="7bf3c747b344b3a25999162cfe342916119f56daed0c2ed8aa9433a962f767ff"
Jul 16 23:05:10.490400 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:10.490381    2565 scope.go:117] "RemoveContainer" containerID="ab731d1e8a2bd41403d5170f0efbb31251bf4ed422a160eeb9c952520052042b"
Jul 16 23:05:10.490650 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:05:10.490630    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-c57rj_odh-ai-gateway-infra(5bb94817-efe5-424c-9841-4debc1b028d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj" podUID="5bb94817-efe5-424c-9841-4debc1b028d2"
Jul 16 23:05:11.527600 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:11.527565    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj"]
Jul 16 23:05:11.661691 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:11.661666    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj"
Jul 16 23:05:11.750048 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:11.750010    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bb94817-efe5-424c-9841-4debc1b028d2-maas-api-service-ca\") pod \"5bb94817-efe5-424c-9841-4debc1b028d2\" (UID: \"5bb94817-efe5-424c-9841-4debc1b028d2\") "
Jul 16 23:05:11.750448 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:11.750418    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb94817-efe5-424c-9841-4debc1b028d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5bb94817-efe5-424c-9841-4debc1b028d2" (UID: "5bb94817-efe5-424c-9841-4debc1b028d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:05:11.851626 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:11.851579    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bb94817-efe5-424c-9841-4debc1b028d2-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:05:12.502566 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:12.502537    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj"
Jul 16 23:05:12.502566 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:12.502552    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj" event={"ID":"5bb94817-efe5-424c-9841-4debc1b028d2","Type":"ContainerDied","Data":"951d781371a1612db3626b0087600bd61f9b6a9882cafdf71aa1a2ee3a93202a"}
Jul 16 23:05:12.502810 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:12.502652    2565 scope.go:117] "RemoveContainer" containerID="ab731d1e8a2bd41403d5170f0efbb31251bf4ed422a160eeb9c952520052042b"
Jul 16 23:05:12.530652 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:12.530615    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj"]
Jul 16 23:05:12.538378 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:12.538344    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-c57rj"]
Jul 16 23:05:13.726633 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:13.726602    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bb94817-efe5-424c-9841-4debc1b028d2" path="/var/lib/kubelet/pods/5bb94817-efe5-424c-9841-4debc1b028d2/volumes"
Jul 16 23:05:22.720436 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:22.720404    2565 scope.go:117] "RemoveContainer" containerID="0ac829e4424acb6559f0f1bb0bb1fbdc963a058edab5c005e0b7210c38187771"
Jul 16 23:05:23.561420 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:23.561362    2565 generic.go:356] "Generic (PLEG): container finished" podID="20f142e0-b44b-44e5-b87b-269244a1ac18" containerID="69f36023bd61536e6222079cf985647a602cf2a1ca3d97ba1e0138014ad1d2fc" exitCode=1
Jul 16 23:05:23.561420 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:23.561416    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd" event={"ID":"20f142e0-b44b-44e5-b87b-269244a1ac18","Type":"ContainerDied","Data":"69f36023bd61536e6222079cf985647a602cf2a1ca3d97ba1e0138014ad1d2fc"}
Jul 16 23:05:23.561904 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:23.561479    2565 scope.go:117] "RemoveContainer" containerID="0ac829e4424acb6559f0f1bb0bb1fbdc963a058edab5c005e0b7210c38187771"
Jul 16 23:05:23.561963 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:23.561922    2565 scope.go:117] "RemoveContainer" containerID="69f36023bd61536e6222079cf985647a602cf2a1ca3d97ba1e0138014ad1d2fc"
Jul 16 23:05:23.562132 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:05:23.562112    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd_odh-ai-gateway-infra(20f142e0-b44b-44e5-b87b-269244a1ac18)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18"
Jul 16 23:05:27.691032 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:27.690983    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:05:27.691472 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:27.691054    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:05:27.691472 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:27.691438    2565 scope.go:117] "RemoveContainer" containerID="69f36023bd61536e6222079cf985647a602cf2a1ca3d97ba1e0138014ad1d2fc"
Jul 16 23:05:27.691649 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:05:27.691628    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd_odh-ai-gateway-infra(20f142e0-b44b-44e5-b87b-269244a1ac18)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18"
Jul 16 23:05:38.720665 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:38.720634    2565 scope.go:117] "RemoveContainer" containerID="69f36023bd61536e6222079cf985647a602cf2a1ca3d97ba1e0138014ad1d2fc"
Jul 16 23:05:38.721057 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:05:38.720852    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd_odh-ai-gateway-infra(20f142e0-b44b-44e5-b87b-269244a1ac18)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18"
Jul 16 23:05:41.238114 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:41.238076    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"]
Jul 16 23:05:41.238572 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:41.238290    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm" podUID="53a970c0-3918-4f54-83ee-c560de2c1929" containerName="istio-proxy" containerID="cri-o://73cefee6978f677d79a920cb3c4fff4449e93fe643db5feaeefd5c23047ddb68" gracePeriod=30
Jul 16 23:05:46.493063 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.493035    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:05:46.589893 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.589860    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-istio-envoy\") pod \"53a970c0-3918-4f54-83ee-c560de2c1929\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") "
Jul 16 23:05:46.590033 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.589905    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-credential-socket\") pod \"53a970c0-3918-4f54-83ee-c560de2c1929\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") "
Jul 16 23:05:46.590033 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.589931    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-workload-socket\") pod \"53a970c0-3918-4f54-83ee-c560de2c1929\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") "
Jul 16 23:05:46.590033 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.589963    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a970c0-3918-4f54-83ee-c560de2c1929-istio-podinfo\") pod \"53a970c0-3918-4f54-83ee-c560de2c1929\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") "
Jul 16 23:05:46.590033 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.589991    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a970c0-3918-4f54-83ee-c560de2c1929-istiod-ca-cert\") pod \"53a970c0-3918-4f54-83ee-c560de2c1929\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") "
Jul 16 23:05:46.590243 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.590126    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-workload-certs\") pod \"53a970c0-3918-4f54-83ee-c560de2c1929\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") "
Jul 16 23:05:46.590243 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.590170    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-istio-data\") pod \"53a970c0-3918-4f54-83ee-c560de2c1929\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") "
Jul 16 23:05:46.590243 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.590212    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53a970c0-3918-4f54-83ee-c560de2c1929-istio-token\") pod \"53a970c0-3918-4f54-83ee-c560de2c1929\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") "
Jul 16 23:05:46.590421 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.590240    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "53a970c0-3918-4f54-83ee-c560de2c1929" (UID: "53a970c0-3918-4f54-83ee-c560de2c1929"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:05:46.590421 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.590274    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl4c4\" (UniqueName: \"kubernetes.io/projected/53a970c0-3918-4f54-83ee-c560de2c1929-kube-api-access-hl4c4\") pod \"53a970c0-3918-4f54-83ee-c560de2c1929\" (UID: \"53a970c0-3918-4f54-83ee-c560de2c1929\") "
Jul 16 23:05:46.590421 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.590301    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "53a970c0-3918-4f54-83ee-c560de2c1929" (UID: "53a970c0-3918-4f54-83ee-c560de2c1929"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:05:46.590663 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.590599    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-credential-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:05:46.590663 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.590624    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-workload-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:05:46.590830 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.590722    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "53a970c0-3918-4f54-83ee-c560de2c1929" (UID: "53a970c0-3918-4f54-83ee-c560de2c1929"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:05:46.590940 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.590919    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53a970c0-3918-4f54-83ee-c560de2c1929-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "53a970c0-3918-4f54-83ee-c560de2c1929" (UID: "53a970c0-3918-4f54-83ee-c560de2c1929"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:05:46.590980 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.590925    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-istio-data" (OuterVolumeSpecName: "istio-data") pod "53a970c0-3918-4f54-83ee-c560de2c1929" (UID: "53a970c0-3918-4f54-83ee-c560de2c1929"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:05:46.592457 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.592435    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "53a970c0-3918-4f54-83ee-c560de2c1929" (UID: "53a970c0-3918-4f54-83ee-c560de2c1929"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:05:46.592556 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.592452    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53a970c0-3918-4f54-83ee-c560de2c1929-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "53a970c0-3918-4f54-83ee-c560de2c1929" (UID: "53a970c0-3918-4f54-83ee-c560de2c1929"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 23:05:46.592707 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.592678    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a970c0-3918-4f54-83ee-c560de2c1929-kube-api-access-hl4c4" (OuterVolumeSpecName: "kube-api-access-hl4c4") pod "53a970c0-3918-4f54-83ee-c560de2c1929" (UID: "53a970c0-3918-4f54-83ee-c560de2c1929"). InnerVolumeSpecName "kube-api-access-hl4c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:05:46.592767 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.592693    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a970c0-3918-4f54-83ee-c560de2c1929-istio-token" (OuterVolumeSpecName: "istio-token") pod "53a970c0-3918-4f54-83ee-c560de2c1929" (UID: "53a970c0-3918-4f54-83ee-c560de2c1929"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:05:46.675295 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.675235    2565 generic.go:356] "Generic (PLEG): container finished" podID="53a970c0-3918-4f54-83ee-c560de2c1929" containerID="73cefee6978f677d79a920cb3c4fff4449e93fe643db5feaeefd5c23047ddb68" exitCode=0
Jul 16 23:05:46.675295 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.675298    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm" event={"ID":"53a970c0-3918-4f54-83ee-c560de2c1929","Type":"ContainerDied","Data":"73cefee6978f677d79a920cb3c4fff4449e93fe643db5feaeefd5c23047ddb68"}
Jul 16 23:05:46.675558 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.675306    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"
Jul 16 23:05:46.675558 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.675352    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm" event={"ID":"53a970c0-3918-4f54-83ee-c560de2c1929","Type":"ContainerDied","Data":"ad0fe3725786116be2ce562a3c0533c7843095084ca2a5f005a25e3a7df37d4c"}
Jul 16 23:05:46.675558 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.675382    2565 scope.go:117] "RemoveContainer" containerID="73cefee6978f677d79a920cb3c4fff4449e93fe643db5feaeefd5c23047ddb68"
Jul 16 23:05:46.684812 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.684785    2565 scope.go:117] "RemoveContainer" containerID="b99690bdd28ee5e684757bc88c12a421a73b11d36bece0453374b771d7c7e858"
Jul 16 23:05:46.692117 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.692091    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl4c4\" (UniqueName: \"kubernetes.io/projected/53a970c0-3918-4f54-83ee-c560de2c1929-kube-api-access-hl4c4\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:05:46.692213 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.692123    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-istio-envoy\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:05:46.692213 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.692140    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a970c0-3918-4f54-83ee-c560de2c1929-istio-podinfo\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:05:46.692213 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.692153    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a970c0-3918-4f54-83ee-c560de2c1929-istiod-ca-cert\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:05:46.692213 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.692167    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-workload-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:05:46.692213 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.692179    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a970c0-3918-4f54-83ee-c560de2c1929-istio-data\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:05:46.692213 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.692191    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53a970c0-3918-4f54-83ee-c560de2c1929-istio-token\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:05:46.692442 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.692406    2565 scope.go:117] "RemoveContainer" containerID="73cefee6978f677d79a920cb3c4fff4449e93fe643db5feaeefd5c23047ddb68"
Jul 16 23:05:46.692680 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:05:46.692661    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73cefee6978f677d79a920cb3c4fff4449e93fe643db5feaeefd5c23047ddb68\": container with ID starting with 73cefee6978f677d79a920cb3c4fff4449e93fe643db5feaeefd5c23047ddb68 not found: ID does not exist" containerID="73cefee6978f677d79a920cb3c4fff4449e93fe643db5feaeefd5c23047ddb68"
Jul 16 23:05:46.692719 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.692688    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73cefee6978f677d79a920cb3c4fff4449e93fe643db5feaeefd5c23047ddb68"} err="failed to get container status \"73cefee6978f677d79a920cb3c4fff4449e93fe643db5feaeefd5c23047ddb68\": rpc error: code = NotFound desc = could not find container \"73cefee6978f677d79a920cb3c4fff4449e93fe643db5feaeefd5c23047ddb68\": container with ID starting with 73cefee6978f677d79a920cb3c4fff4449e93fe643db5feaeefd5c23047ddb68 not found: ID does not exist"
Jul 16 23:05:46.692719 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.692705    2565 scope.go:117] "RemoveContainer" containerID="b99690bdd28ee5e684757bc88c12a421a73b11d36bece0453374b771d7c7e858"
Jul 16 23:05:46.692954 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:05:46.692937    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b99690bdd28ee5e684757bc88c12a421a73b11d36bece0453374b771d7c7e858\": container with ID starting with b99690bdd28ee5e684757bc88c12a421a73b11d36bece0453374b771d7c7e858 not found: ID does not exist" containerID="b99690bdd28ee5e684757bc88c12a421a73b11d36bece0453374b771d7c7e858"
Jul 16 23:05:46.692987 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.692962    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b99690bdd28ee5e684757bc88c12a421a73b11d36bece0453374b771d7c7e858"} err="failed to get container status \"b99690bdd28ee5e684757bc88c12a421a73b11d36bece0453374b771d7c7e858\": rpc error: code = NotFound desc = could not find container \"b99690bdd28ee5e684757bc88c12a421a73b11d36bece0453374b771d7c7e858\": container with ID starting with b99690bdd28ee5e684757bc88c12a421a73b11d36bece0453374b771d7c7e858 not found: ID does not exist"
Jul 16 23:05:46.704814 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.704791    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"]
Jul 16 23:05:46.709962 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.709942    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-06b30c8f-gw-openshift-default-6d4d89c747-dmzdm"]
Jul 16 23:05:46.855597 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.855565    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"]
Jul 16 23:05:46.981597 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:46.981571    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:05:47.095708 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:47.095672    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp4zj\" (UniqueName: \"kubernetes.io/projected/20f142e0-b44b-44e5-b87b-269244a1ac18-kube-api-access-vp4zj\") pod \"20f142e0-b44b-44e5-b87b-269244a1ac18\" (UID: \"20f142e0-b44b-44e5-b87b-269244a1ac18\") "
Jul 16 23:05:47.095917 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:47.095745    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f142e0-b44b-44e5-b87b-269244a1ac18-maas-api-tls\") pod \"20f142e0-b44b-44e5-b87b-269244a1ac18\" (UID: \"20f142e0-b44b-44e5-b87b-269244a1ac18\") "
Jul 16 23:05:47.095917 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:47.095888    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f142e0-b44b-44e5-b87b-269244a1ac18-openshift-service-ca\") pod \"20f142e0-b44b-44e5-b87b-269244a1ac18\" (UID: \"20f142e0-b44b-44e5-b87b-269244a1ac18\") "
Jul 16 23:05:47.096272 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:47.096221    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f142e0-b44b-44e5-b87b-269244a1ac18-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "20f142e0-b44b-44e5-b87b-269244a1ac18" (UID: "20f142e0-b44b-44e5-b87b-269244a1ac18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:05:47.097859 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:47.097836    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f142e0-b44b-44e5-b87b-269244a1ac18-kube-api-access-vp4zj" (OuterVolumeSpecName: "kube-api-access-vp4zj") pod "20f142e0-b44b-44e5-b87b-269244a1ac18" (UID: "20f142e0-b44b-44e5-b87b-269244a1ac18"). InnerVolumeSpecName "kube-api-access-vp4zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:05:47.098042 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:47.098022    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f142e0-b44b-44e5-b87b-269244a1ac18-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "20f142e0-b44b-44e5-b87b-269244a1ac18" (UID: "20f142e0-b44b-44e5-b87b-269244a1ac18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:05:47.196927 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:47.196830    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20f142e0-b44b-44e5-b87b-269244a1ac18-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:05:47.196927 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:47.196861    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp4zj\" (UniqueName: \"kubernetes.io/projected/20f142e0-b44b-44e5-b87b-269244a1ac18-kube-api-access-vp4zj\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:05:47.196927 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:47.196873    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20f142e0-b44b-44e5-b87b-269244a1ac18-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:05:47.682709 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:47.682674    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"
Jul 16 23:05:47.682709 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:47.682689    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd" event={"ID":"20f142e0-b44b-44e5-b87b-269244a1ac18","Type":"ContainerDied","Data":"3414e39e04461b76af3c06d21264499454da05655dbaf91c40e94a237ba837ff"}
Jul 16 23:05:47.683285 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:47.682756    2565 scope.go:117] "RemoveContainer" containerID="69f36023bd61536e6222079cf985647a602cf2a1ca3d97ba1e0138014ad1d2fc"
Jul 16 23:05:47.711014 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:47.710968    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"]
Jul 16 23:05:47.717192 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:47.717166    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-06b30c8f-5c74df448d-jdqtd"]
Jul 16 23:05:47.725444 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:47.725419    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18" path="/var/lib/kubelet/pods/20f142e0-b44b-44e5-b87b-269244a1ac18/volumes"
Jul 16 23:05:47.725872 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:47.725859    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53a970c0-3918-4f54-83ee-c560de2c1929" path="/var/lib/kubelet/pods/53a970c0-3918-4f54-83ee-c560de2c1929/volumes"
Jul 16 23:05:52.390626 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.390591    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7"]
Jul 16 23:05:52.391007 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.390941    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18" containerName="maas-api"
Jul 16 23:05:52.391007 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.390953    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18" containerName="maas-api"
Jul 16 23:05:52.391007 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.390960    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18" containerName="maas-api"
Jul 16 23:05:52.391007 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.390966    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18" containerName="maas-api"
Jul 16 23:05:52.391007 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.390973    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cdc0d23-d250-45db-a0da-b8b007522cf7" containerName="revoke-keys"
Jul 16 23:05:52.391007 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.390979    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cdc0d23-d250-45db-a0da-b8b007522cf7" containerName="revoke-keys"
Jul 16 23:05:52.391007 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.390986    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb94817-efe5-424c-9841-4debc1b028d2" containerName="revoke-keys"
Jul 16 23:05:52.391007 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.390992    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb94817-efe5-424c-9841-4debc1b028d2" containerName="revoke-keys"
Jul 16 23:05:52.391007 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391001    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18" containerName="maas-api"
Jul 16 23:05:52.391007 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391006    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18" containerName="maas-api"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391015    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb94817-efe5-424c-9841-4debc1b028d2" containerName="revoke-keys"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391020    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb94817-efe5-424c-9841-4debc1b028d2" containerName="revoke-keys"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391030    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53a970c0-3918-4f54-83ee-c560de2c1929" containerName="istio-proxy"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391035    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="53a970c0-3918-4f54-83ee-c560de2c1929" containerName="istio-proxy"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391044    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb94817-efe5-424c-9841-4debc1b028d2" containerName="revoke-keys"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391050    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb94817-efe5-424c-9841-4debc1b028d2" containerName="revoke-keys"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391056    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cdc0d23-d250-45db-a0da-b8b007522cf7" containerName="revoke-keys"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391061    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cdc0d23-d250-45db-a0da-b8b007522cf7" containerName="revoke-keys"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391071    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53a970c0-3918-4f54-83ee-c560de2c1929" containerName="istio-proxy"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391076    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="53a970c0-3918-4f54-83ee-c560de2c1929" containerName="istio-proxy"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391123    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cdc0d23-d250-45db-a0da-b8b007522cf7" containerName="revoke-keys"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391130    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cdc0d23-d250-45db-a0da-b8b007522cf7" containerName="revoke-keys"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391136    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb94817-efe5-424c-9841-4debc1b028d2" containerName="revoke-keys"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391141    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cdc0d23-d250-45db-a0da-b8b007522cf7" containerName="revoke-keys"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391147    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="53a970c0-3918-4f54-83ee-c560de2c1929" containerName="istio-proxy"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391152    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18" containerName="maas-api"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391158    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18" containerName="maas-api"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391163    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="53a970c0-3918-4f54-83ee-c560de2c1929" containerName="istio-proxy"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391169    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb94817-efe5-424c-9841-4debc1b028d2" containerName="revoke-keys"
Jul 16 23:05:52.391345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.391174    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18" containerName="maas-api"
Jul 16 23:05:52.395537 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.395515    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7"
Jul 16 23:05:52.397954 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.397929    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:05:52.403787 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.403759    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7"]
Jul 16 23:05:52.443860 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.443824    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1399565b-99c9-4b8f-9ab2-a6b330fe50e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-hr2z7\" (UID: \"1399565b-99c9-4b8f-9ab2-a6b330fe50e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7"
Jul 16 23:05:52.545915 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.544975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1399565b-99c9-4b8f-9ab2-a6b330fe50e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-hr2z7\" (UID: \"1399565b-99c9-4b8f-9ab2-a6b330fe50e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7"
Jul 16 23:05:52.545915 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.545862    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1399565b-99c9-4b8f-9ab2-a6b330fe50e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-hr2z7\" (UID: \"1399565b-99c9-4b8f-9ab2-a6b330fe50e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7"
Jul 16 23:05:52.706439 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.706353    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7"
Jul 16 23:05:52.838677 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:52.838538    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7"]
Jul 16 23:05:52.841380 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:05:52.841349    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1399565b_99c9_4b8f_9ab2_a6b330fe50e7.slice/crio-676096f796745350fc13c22520127872054de6b486a25e49cc807b8167954505 WatchSource:0}: Error finding container 676096f796745350fc13c22520127872054de6b486a25e49cc807b8167954505: Status 404 returned error can't find the container with id 676096f796745350fc13c22520127872054de6b486a25e49cc807b8167954505
Jul 16 23:05:53.714367 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:53.714287    2565 generic.go:356] "Generic (PLEG): container finished" podID="1399565b-99c9-4b8f-9ab2-a6b330fe50e7" containerID="d73f4ce9d4aad7ebb0f3772c70ea9c4e1dfab61c5b4723477c7587b65a47168d" exitCode=6
Jul 16 23:05:53.714825 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:53.714374    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7" event={"ID":"1399565b-99c9-4b8f-9ab2-a6b330fe50e7","Type":"ContainerDied","Data":"d73f4ce9d4aad7ebb0f3772c70ea9c4e1dfab61c5b4723477c7587b65a47168d"}
Jul 16 23:05:53.714825 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:53.714433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7" event={"ID":"1399565b-99c9-4b8f-9ab2-a6b330fe50e7","Type":"ContainerStarted","Data":"676096f796745350fc13c22520127872054de6b486a25e49cc807b8167954505"}
Jul 16 23:05:53.714825 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:53.714536    2565 scope.go:117] "RemoveContainer" containerID="d73f4ce9d4aad7ebb0f3772c70ea9c4e1dfab61c5b4723477c7587b65a47168d"
Jul 16 23:05:53.918353 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:53.918314    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"]
Jul 16 23:05:53.927935 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:53.927901    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18" containerName="maas-api"
Jul 16 23:05:53.927935 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:53.927938    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18" containerName="maas-api"
Jul 16 23:05:53.928143 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:53.927952    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cdc0d23-d250-45db-a0da-b8b007522cf7" containerName="revoke-keys"
Jul 16 23:05:53.928143 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:53.927959    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cdc0d23-d250-45db-a0da-b8b007522cf7" containerName="revoke-keys"
Jul 16 23:05:53.928143 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:53.928071    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="20f142e0-b44b-44e5-b87b-269244a1ac18" containerName="maas-api"
Jul 16 23:05:53.928143 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:53.928089    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb94817-efe5-424c-9841-4debc1b028d2" containerName="revoke-keys"
Jul 16 23:05:53.932587 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:53.932563    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:53.935138 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:53.935116    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a1d48f5b-openshift-default-dockercfg-l8k46\""
Jul 16 23:05:53.943855 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:53.943820    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"]
Jul 16 23:05:54.059703 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.059675    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b6kf\" (UniqueName: \"kubernetes.io/projected/d9047909-33ef-44b2-9934-0804433a485d-kube-api-access-8b6kf\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.059854 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.059709    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-credential-socket\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.059854 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.059728    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9047909-33ef-44b2-9934-0804433a485d-istiod-ca-cert\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.059854 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.059749    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-workload-certs\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.059854 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.059828    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9047909-33ef-44b2-9934-0804433a485d-istio-podinfo\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.060021 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.059868    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-istio-data\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.060021 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.059888    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9047909-33ef-44b2-9934-0804433a485d-istio-token\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.060021 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.059917    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-istio-envoy\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.060021 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.059966    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-workload-socket\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.160456 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.160420    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b6kf\" (UniqueName: \"kubernetes.io/projected/d9047909-33ef-44b2-9934-0804433a485d-kube-api-access-8b6kf\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.160456 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.160461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-credential-socket\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.160687 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.160484    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9047909-33ef-44b2-9934-0804433a485d-istiod-ca-cert\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.160687 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.160512    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-workload-certs\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.160687 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.160583    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9047909-33ef-44b2-9934-0804433a485d-istio-podinfo\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.160687 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.160609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-istio-data\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.160687 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.160636    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9047909-33ef-44b2-9934-0804433a485d-istio-token\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.160687 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.160671    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-istio-envoy\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.160998 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.160755    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-workload-socket\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.160998 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.160911    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-credential-socket\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.160998 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.160958    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-workload-certs\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.161150 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.161078    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-istio-data\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.161150 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.161121    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-workload-socket\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.161277 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.161238    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9047909-33ef-44b2-9934-0804433a485d-istiod-ca-cert\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.162969 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.162945    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-istio-envoy\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.163105 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.162993    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9047909-33ef-44b2-9934-0804433a485d-istio-podinfo\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.168507 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.168474    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9047909-33ef-44b2-9934-0804433a485d-istio-token\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.168621 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.168605    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b6kf\" (UniqueName: \"kubernetes.io/projected/d9047909-33ef-44b2-9934-0804433a485d-kube-api-access-8b6kf\") pod \"e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") " pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.244865 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.244831    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:54.381283 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.381238    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"]
Jul 16 23:05:54.383620 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:05:54.383594    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9047909_33ef_44b2_9934_0804433a485d.slice/crio-c47e0af644d363b04991facdcd5e0e92d2062f48ceda378462d67718ada8e324 WatchSource:0}: Error finding container c47e0af644d363b04991facdcd5e0e92d2062f48ceda378462d67718ada8e324: Status 404 returned error can't find the container with id c47e0af644d363b04991facdcd5e0e92d2062f48ceda378462d67718ada8e324
Jul 16 23:05:54.385812 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.385778    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:05:54.385892 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.385846    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:05:54.385892 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.385876    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:05:54.721822 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.721731    2565 generic.go:356] "Generic (PLEG): container finished" podID="1399565b-99c9-4b8f-9ab2-a6b330fe50e7" containerID="34045710273ac2e9441070ba299c32f934168c89f6fad184abee1fa70c004466" exitCode=6
Jul 16 23:05:54.722326 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.721819    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7" event={"ID":"1399565b-99c9-4b8f-9ab2-a6b330fe50e7","Type":"ContainerDied","Data":"34045710273ac2e9441070ba299c32f934168c89f6fad184abee1fa70c004466"}
Jul 16 23:05:54.722326 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.721876    2565 scope.go:117] "RemoveContainer" containerID="d73f4ce9d4aad7ebb0f3772c70ea9c4e1dfab61c5b4723477c7587b65a47168d"
Jul 16 23:05:54.722326 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.722053    2565 scope.go:117] "RemoveContainer" containerID="34045710273ac2e9441070ba299c32f934168c89f6fad184abee1fa70c004466"
Jul 16 23:05:54.722326 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:05:54.722310    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-hr2z7_odh-ai-gateway-infra(1399565b-99c9-4b8f-9ab2-a6b330fe50e7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7" podUID="1399565b-99c9-4b8f-9ab2-a6b330fe50e7"
Jul 16 23:05:54.723696 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.723671    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz" event={"ID":"d9047909-33ef-44b2-9934-0804433a485d","Type":"ContainerStarted","Data":"95fab35c8a42331cf577a1976b6500b46ef79b08351c29c05081874c96b9ff17"}
Jul 16 23:05:54.723767 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.723727    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz" event={"ID":"d9047909-33ef-44b2-9934-0804433a485d","Type":"ContainerStarted","Data":"c47e0af644d363b04991facdcd5e0e92d2062f48ceda378462d67718ada8e324"}
Jul 16 23:05:54.761099 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:54.761033    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz" podStartSLOduration=1.7610133879999998 podStartE2EDuration="1.761013388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:05:54.759632432 +0000 UTC m=+1965.508116531" watchObservedRunningTime="2026-07-16 23:05:54.761013388 +0000 UTC m=+1965.509497459"
Jul 16 23:05:55.245588 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.245547    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:55.250666 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.250640    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:55.366576 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.366542    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"]
Jul 16 23:05:55.371113 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.371095    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:05:55.381931 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.381913    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a1d48f5b-serving-cert\""
Jul 16 23:05:55.395799 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.395769    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"]
Jul 16 23:05:55.471946 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.471910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a54521-363e-4fa7-8b7e-7ae778ed472e-maas-api-tls\") pod \"maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb\" (UID: \"71a54521-363e-4fa7-8b7e-7ae778ed472e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:05:55.472115 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.471971    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wzcs\" (UniqueName: \"kubernetes.io/projected/71a54521-363e-4fa7-8b7e-7ae778ed472e-kube-api-access-8wzcs\") pod \"maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb\" (UID: \"71a54521-363e-4fa7-8b7e-7ae778ed472e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:05:55.472115 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.471994    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a54521-363e-4fa7-8b7e-7ae778ed472e-openshift-service-ca\") pod \"maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb\" (UID: \"71a54521-363e-4fa7-8b7e-7ae778ed472e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:05:55.573414 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.573327    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wzcs\" (UniqueName: \"kubernetes.io/projected/71a54521-363e-4fa7-8b7e-7ae778ed472e-kube-api-access-8wzcs\") pod \"maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb\" (UID: \"71a54521-363e-4fa7-8b7e-7ae778ed472e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:05:55.573414 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.573365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a54521-363e-4fa7-8b7e-7ae778ed472e-openshift-service-ca\") pod \"maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb\" (UID: \"71a54521-363e-4fa7-8b7e-7ae778ed472e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:05:55.573602 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.573433    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a54521-363e-4fa7-8b7e-7ae778ed472e-maas-api-tls\") pod \"maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb\" (UID: \"71a54521-363e-4fa7-8b7e-7ae778ed472e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:05:55.573984 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.573964    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a54521-363e-4fa7-8b7e-7ae778ed472e-openshift-service-ca\") pod \"maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb\" (UID: \"71a54521-363e-4fa7-8b7e-7ae778ed472e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:05:55.575908 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.575888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a54521-363e-4fa7-8b7e-7ae778ed472e-maas-api-tls\") pod \"maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb\" (UID: \"71a54521-363e-4fa7-8b7e-7ae778ed472e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:05:55.592372 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.592349    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wzcs\" (UniqueName: \"kubernetes.io/projected/71a54521-363e-4fa7-8b7e-7ae778ed472e-kube-api-access-8wzcs\") pod \"maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb\" (UID: \"71a54521-363e-4fa7-8b7e-7ae778ed472e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:05:55.681174 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.681136    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:05:55.730686 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.730659    2565 scope.go:117] "RemoveContainer" containerID="34045710273ac2e9441070ba299c32f934168c89f6fad184abee1fa70c004466"
Jul 16 23:05:55.731436 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.730815    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:55.731436 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:05:55.730918    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-hr2z7_odh-ai-gateway-infra(1399565b-99c9-4b8f-9ab2-a6b330fe50e7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7" podUID="1399565b-99c9-4b8f-9ab2-a6b330fe50e7"
Jul 16 23:05:55.732206 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.732182    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:05:55.831038 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:55.831007    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"]
Jul 16 23:05:55.832626 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:05:55.832601    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a54521_363e_4fa7_8b7e_7ae778ed472e.slice/crio-9042dc643a697f2d4ed88565bead41c503d7dbd0082947091fcb3f99bcdac41d WatchSource:0}: Error finding container 9042dc643a697f2d4ed88565bead41c503d7dbd0082947091fcb3f99bcdac41d: Status 404 returned error can't find the container with id 9042dc643a697f2d4ed88565bead41c503d7dbd0082947091fcb3f99bcdac41d
Jul 16 23:05:56.740025 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:56.739993    2565 generic.go:356] "Generic (PLEG): container finished" podID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerID="2a243e1794350538d573b9425f5cdb498f66acfc1e783ed1f362f259dfc96e3d" exitCode=1
Jul 16 23:05:56.740584 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:56.740061    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" event={"ID":"71a54521-363e-4fa7-8b7e-7ae778ed472e","Type":"ContainerDied","Data":"2a243e1794350538d573b9425f5cdb498f66acfc1e783ed1f362f259dfc96e3d"}
Jul 16 23:05:56.740584 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:56.740110    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" event={"ID":"71a54521-363e-4fa7-8b7e-7ae778ed472e","Type":"ContainerStarted","Data":"9042dc643a697f2d4ed88565bead41c503d7dbd0082947091fcb3f99bcdac41d"}
Jul 16 23:05:56.740584 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:56.740326    2565 scope.go:117] "RemoveContainer" containerID="2a243e1794350538d573b9425f5cdb498f66acfc1e783ed1f362f259dfc96e3d"
Jul 16 23:05:57.746934 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:57.746836    2565 generic.go:356] "Generic (PLEG): container finished" podID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerID="d21ce0262fdd692fd1c40f1511fff93c4f0bd39273c2d402cfdbd19caa60425c" exitCode=1
Jul 16 23:05:57.746934 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:57.746914    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" event={"ID":"71a54521-363e-4fa7-8b7e-7ae778ed472e","Type":"ContainerDied","Data":"d21ce0262fdd692fd1c40f1511fff93c4f0bd39273c2d402cfdbd19caa60425c"}
Jul 16 23:05:57.747482 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:57.746975    2565 scope.go:117] "RemoveContainer" containerID="2a243e1794350538d573b9425f5cdb498f66acfc1e783ed1f362f259dfc96e3d"
Jul 16 23:05:57.747482 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:57.747241    2565 scope.go:117] "RemoveContainer" containerID="d21ce0262fdd692fd1c40f1511fff93c4f0bd39273c2d402cfdbd19caa60425c"
Jul 16 23:05:57.747585 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:05:57.747506    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:05:58.754062 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:05:58.754030    2565 scope.go:117] "RemoveContainer" containerID="d21ce0262fdd692fd1c40f1511fff93c4f0bd39273c2d402cfdbd19caa60425c"
Jul 16 23:05:58.754491 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:05:58.754233    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:06:00.682085 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:00.682030    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:06:00.682546 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:00.682500    2565 scope.go:117] "RemoveContainer" containerID="d21ce0262fdd692fd1c40f1511fff93c4f0bd39273c2d402cfdbd19caa60425c"
Jul 16 23:06:00.682714 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:06:00.682692    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:06:05.681313 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:05.681234    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:06:05.681838 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:05.681663    2565 scope.go:117] "RemoveContainer" containerID="d21ce0262fdd692fd1c40f1511fff93c4f0bd39273c2d402cfdbd19caa60425c"
Jul 16 23:06:05.681838 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:06:05.681827    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:06:06.720344 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:06.720311    2565 scope.go:117] "RemoveContainer" containerID="34045710273ac2e9441070ba299c32f934168c89f6fad184abee1fa70c004466"
Jul 16 23:06:06.797868 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:06.797832    2565 generic.go:356] "Generic (PLEG): container finished" podID="d9047909-33ef-44b2-9934-0804433a485d" containerID="95fab35c8a42331cf577a1976b6500b46ef79b08351c29c05081874c96b9ff17" exitCode=0
Jul 16 23:06:06.798051 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:06.797908    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz" event={"ID":"d9047909-33ef-44b2-9934-0804433a485d","Type":"ContainerDied","Data":"95fab35c8a42331cf577a1976b6500b46ef79b08351c29c05081874c96b9ff17"}
Jul 16 23:06:06.798394 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:06.798376    2565 scope.go:117] "RemoveContainer" containerID="95fab35c8a42331cf577a1976b6500b46ef79b08351c29c05081874c96b9ff17"
Jul 16 23:06:06.799387 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:06.799360    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:06:06.799447 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:06.799433    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:06:06.799490 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:06.799464    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:06:07.804998 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:07.804903    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz" event={"ID":"d9047909-33ef-44b2-9934-0804433a485d","Type":"ContainerStarted","Data":"a5a45f3a6178bbbcb6cfffe221551fbbd40c6fa59ece6f9c65ccd655380b9a63"}
Jul 16 23:06:07.806708 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:07.806684    2565 generic.go:356] "Generic (PLEG): container finished" podID="1399565b-99c9-4b8f-9ab2-a6b330fe50e7" containerID="8ce4f38561bf26ca15423dbf41a751e28014187ab936094b9ec94003b5396f44" exitCode=6
Jul 16 23:06:07.806829 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:07.806732    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7" event={"ID":"1399565b-99c9-4b8f-9ab2-a6b330fe50e7","Type":"ContainerDied","Data":"8ce4f38561bf26ca15423dbf41a751e28014187ab936094b9ec94003b5396f44"}
Jul 16 23:06:07.806829 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:07.806782    2565 scope.go:117] "RemoveContainer" containerID="34045710273ac2e9441070ba299c32f934168c89f6fad184abee1fa70c004466"
Jul 16 23:06:07.807064 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:07.807046    2565 scope.go:117] "RemoveContainer" containerID="8ce4f38561bf26ca15423dbf41a751e28014187ab936094b9ec94003b5396f44"
Jul 16 23:06:07.807333 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:06:07.807311    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-hr2z7_odh-ai-gateway-infra(1399565b-99c9-4b8f-9ab2-a6b330fe50e7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7" podUID="1399565b-99c9-4b8f-9ab2-a6b330fe50e7"
Jul 16 23:06:08.246152 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:08.246106    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:06:08.247360 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:08.247325    2565 patch_prober.go:28] interesting pod/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 16 23:06:08.247494 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:08.247402    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz" podUID="d9047909-33ef-44b2-9934-0804433a485d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 16 23:06:08.888002 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:08.887963    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7"]
Jul 16 23:06:09.053272 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:09.052571    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7"
Jul 16 23:06:09.211325 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:09.211201    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1399565b-99c9-4b8f-9ab2-a6b330fe50e7-maas-api-service-ca\") pod \"1399565b-99c9-4b8f-9ab2-a6b330fe50e7\" (UID: \"1399565b-99c9-4b8f-9ab2-a6b330fe50e7\") "
Jul 16 23:06:09.211564 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:09.211542    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1399565b-99c9-4b8f-9ab2-a6b330fe50e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1399565b-99c9-4b8f-9ab2-a6b330fe50e7" (UID: "1399565b-99c9-4b8f-9ab2-a6b330fe50e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:06:09.245321 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:09.245270    2565 patch_prober.go:28] interesting pod/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 16 23:06:09.245504 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:09.245362    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz" podUID="d9047909-33ef-44b2-9934-0804433a485d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 16 23:06:09.312464 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:09.312422    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1399565b-99c9-4b8f-9ab2-a6b330fe50e7-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:06:09.820706 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:09.820674    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7"
Jul 16 23:06:09.820921 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:09.820674    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7" event={"ID":"1399565b-99c9-4b8f-9ab2-a6b330fe50e7","Type":"ContainerDied","Data":"676096f796745350fc13c22520127872054de6b486a25e49cc807b8167954505"}
Jul 16 23:06:09.820921 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:09.820820    2565 scope.go:117] "RemoveContainer" containerID="8ce4f38561bf26ca15423dbf41a751e28014187ab936094b9ec94003b5396f44"
Jul 16 23:06:09.839472 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:09.839435    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7"]
Jul 16 23:06:09.841621 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:09.841576    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-hr2z7"]
Jul 16 23:06:10.246275 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:10.246213    2565 patch_prober.go:28] interesting pod/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 16 23:06:10.246820 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:10.246314    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz" podUID="d9047909-33ef-44b2-9934-0804433a485d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 16 23:06:10.731206 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:10.731153    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:06:11.245710 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:11.245662    2565 patch_prober.go:28] interesting pod/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 16 23:06:11.245946 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:11.245748    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz" podUID="d9047909-33ef-44b2-9934-0804433a485d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 16 23:06:11.726680 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:11.726645    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1399565b-99c9-4b8f-9ab2-a6b330fe50e7" path="/var/lib/kubelet/pods/1399565b-99c9-4b8f-9ab2-a6b330fe50e7/volumes"
Jul 16 23:06:12.250279 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:12.250221    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:06:12.251323 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:12.251296    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:06:17.721374 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:17.721333    2565 scope.go:117] "RemoveContainer" containerID="d21ce0262fdd692fd1c40f1511fff93c4f0bd39273c2d402cfdbd19caa60425c"
Jul 16 23:06:18.866296 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:18.866262    2565 generic.go:356] "Generic (PLEG): container finished" podID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerID="0b220e2a489892017f4ccff8bdb699ebea68b73f7af576099284f8dc112fb3ff" exitCode=1
Jul 16 23:06:18.866714 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:18.866321    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" event={"ID":"71a54521-363e-4fa7-8b7e-7ae778ed472e","Type":"ContainerDied","Data":"0b220e2a489892017f4ccff8bdb699ebea68b73f7af576099284f8dc112fb3ff"}
Jul 16 23:06:18.866714 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:18.866383    2565 scope.go:117] "RemoveContainer" containerID="d21ce0262fdd692fd1c40f1511fff93c4f0bd39273c2d402cfdbd19caa60425c"
Jul 16 23:06:18.866795 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:18.866723    2565 scope.go:117] "RemoveContainer" containerID="0b220e2a489892017f4ccff8bdb699ebea68b73f7af576099284f8dc112fb3ff"
Jul 16 23:06:18.866985 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:06:18.866964    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:06:20.681537 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:20.681501    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:06:20.681925 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:20.681895    2565 scope.go:117] "RemoveContainer" containerID="0b220e2a489892017f4ccff8bdb699ebea68b73f7af576099284f8dc112fb3ff"
Jul 16 23:06:20.682156 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:06:20.682132    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:06:25.682271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:25.682214    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:06:25.682769 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:25.682591    2565 scope.go:117] "RemoveContainer" containerID="0b220e2a489892017f4ccff8bdb699ebea68b73f7af576099284f8dc112fb3ff"
Jul 16 23:06:25.682812 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:06:25.682766    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:06:36.720728 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:36.720695    2565 scope.go:117] "RemoveContainer" containerID="0b220e2a489892017f4ccff8bdb699ebea68b73f7af576099284f8dc112fb3ff"
Jul 16 23:06:36.721155 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:06:36.720879    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:06:50.720851 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:50.720816    2565 scope.go:117] "RemoveContainer" containerID="0b220e2a489892017f4ccff8bdb699ebea68b73f7af576099284f8dc112fb3ff"
Jul 16 23:06:50.721269 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:50.721132    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 23:06:52.025707 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:52.025676    2565 generic.go:356] "Generic (PLEG): container finished" podID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerID="c0c65986cfb44011d0701f86a2ec96de5fe7bcb66ca2661049fa1b005c488420" exitCode=1
Jul 16 23:06:52.026130 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:52.025726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" event={"ID":"71a54521-363e-4fa7-8b7e-7ae778ed472e","Type":"ContainerDied","Data":"c0c65986cfb44011d0701f86a2ec96de5fe7bcb66ca2661049fa1b005c488420"}
Jul 16 23:06:52.026130 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:52.025791    2565 scope.go:117] "RemoveContainer" containerID="0b220e2a489892017f4ccff8bdb699ebea68b73f7af576099284f8dc112fb3ff"
Jul 16 23:06:52.026130 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:52.026108    2565 scope.go:117] "RemoveContainer" containerID="c0c65986cfb44011d0701f86a2ec96de5fe7bcb66ca2661049fa1b005c488420"
Jul 16 23:06:52.026342 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:06:52.026314    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:06:55.682156 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:55.682111    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:06:55.682597 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:55.682203    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:06:55.682597 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:06:55.682556    2565 scope.go:117] "RemoveContainer" containerID="c0c65986cfb44011d0701f86a2ec96de5fe7bcb66ca2661049fa1b005c488420"
Jul 16 23:06:55.682765 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:06:55.682742    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:07:07.721380 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:07:07.721344    2565 scope.go:117] "RemoveContainer" containerID="c0c65986cfb44011d0701f86a2ec96de5fe7bcb66ca2661049fa1b005c488420"
Jul 16 23:07:07.721866 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:07:07.721542    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:07:20.720467 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:07:20.720429    2565 scope.go:117] "RemoveContainer" containerID="c0c65986cfb44011d0701f86a2ec96de5fe7bcb66ca2661049fa1b005c488420"
Jul 16 23:07:20.720924 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:07:20.720616    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:07:32.721121 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:07:32.721026    2565 scope.go:117] "RemoveContainer" containerID="c0c65986cfb44011d0701f86a2ec96de5fe7bcb66ca2661049fa1b005c488420"
Jul 16 23:07:34.232874 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:07:34.232844    2565 generic.go:356] "Generic (PLEG): container finished" podID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerID="419ec012b0ece2dc8ac71fd89c40986e77abdc1e7ea8e40b3db9e0a6f772af7d" exitCode=1
Jul 16 23:07:34.233278 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:07:34.232919    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" event={"ID":"71a54521-363e-4fa7-8b7e-7ae778ed472e","Type":"ContainerDied","Data":"419ec012b0ece2dc8ac71fd89c40986e77abdc1e7ea8e40b3db9e0a6f772af7d"}
Jul 16 23:07:34.233278 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:07:34.232978    2565 scope.go:117] "RemoveContainer" containerID="c0c65986cfb44011d0701f86a2ec96de5fe7bcb66ca2661049fa1b005c488420"
Jul 16 23:07:34.233401 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:07:34.233358    2565 scope.go:117] "RemoveContainer" containerID="419ec012b0ece2dc8ac71fd89c40986e77abdc1e7ea8e40b3db9e0a6f772af7d"
Jul 16 23:07:34.233554 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:07:34.233529    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:07:35.681811 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:07:35.681777    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:07:35.682239 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:07:35.681842    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:07:35.682239 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:07:35.682164    2565 scope.go:117] "RemoveContainer" containerID="419ec012b0ece2dc8ac71fd89c40986e77abdc1e7ea8e40b3db9e0a6f772af7d"
Jul 16 23:07:35.682393 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:07:35.682373    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:07:48.721141 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:07:48.721107    2565 scope.go:117] "RemoveContainer" containerID="419ec012b0ece2dc8ac71fd89c40986e77abdc1e7ea8e40b3db9e0a6f772af7d"
Jul 16 23:07:48.721666 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:07:48.721309    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:08:02.720899 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:08:02.720865    2565 scope.go:117] "RemoveContainer" containerID="419ec012b0ece2dc8ac71fd89c40986e77abdc1e7ea8e40b3db9e0a6f772af7d"
Jul 16 23:08:02.721390 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:08:02.721073    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:08:14.720670 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:08:14.720630    2565 scope.go:117] "RemoveContainer" containerID="419ec012b0ece2dc8ac71fd89c40986e77abdc1e7ea8e40b3db9e0a6f772af7d"
Jul 16 23:08:14.721116 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:08:14.720869    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:08:27.721376 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:08:27.721339    2565 scope.go:117] "RemoveContainer" containerID="419ec012b0ece2dc8ac71fd89c40986e77abdc1e7ea8e40b3db9e0a6f772af7d"
Jul 16 23:08:27.721881 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:08:27.721548    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:08:41.720811 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:08:41.720775    2565 scope.go:117] "RemoveContainer" containerID="419ec012b0ece2dc8ac71fd89c40986e77abdc1e7ea8e40b3db9e0a6f772af7d"
Jul 16 23:08:41.721225 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:08:41.721010    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:08:54.721121 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:08:54.721086    2565 scope.go:117] "RemoveContainer" containerID="419ec012b0ece2dc8ac71fd89c40986e77abdc1e7ea8e40b3db9e0a6f772af7d"
Jul 16 23:08:55.627045 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:08:55.627007    2565 generic.go:356] "Generic (PLEG): container finished" podID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerID="1fda79b79d99855eb92515ebecd25dc0ed3f52150df533bb538474c3f53b5f01" exitCode=1
Jul 16 23:08:55.627221 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:08:55.627077    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" event={"ID":"71a54521-363e-4fa7-8b7e-7ae778ed472e","Type":"ContainerDied","Data":"1fda79b79d99855eb92515ebecd25dc0ed3f52150df533bb538474c3f53b5f01"}
Jul 16 23:08:55.627221 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:08:55.627137    2565 scope.go:117] "RemoveContainer" containerID="419ec012b0ece2dc8ac71fd89c40986e77abdc1e7ea8e40b3db9e0a6f772af7d"
Jul 16 23:08:55.627519 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:08:55.627497    2565 scope.go:117] "RemoveContainer" containerID="1fda79b79d99855eb92515ebecd25dc0ed3f52150df533bb538474c3f53b5f01"
Jul 16 23:08:55.627759 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:08:55.627737    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:08:55.682119 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:08:55.682087    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:08:55.682382 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:08:55.682178    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:08:56.633243 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:08:56.633209    2565 scope.go:117] "RemoveContainer" containerID="1fda79b79d99855eb92515ebecd25dc0ed3f52150df533bb538474c3f53b5f01"
Jul 16 23:08:56.633661 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:08:56.633440    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:09:11.727932 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:09:11.727848    2565 scope.go:117] "RemoveContainer" containerID="1fda79b79d99855eb92515ebecd25dc0ed3f52150df533bb538474c3f53b5f01"
Jul 16 23:09:11.728993 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:09:11.728960    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:09:26.721349 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:09:26.721314    2565 scope.go:117] "RemoveContainer" containerID="1fda79b79d99855eb92515ebecd25dc0ed3f52150df533bb538474c3f53b5f01"
Jul 16 23:09:26.721736 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:09:26.721532    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:09:40.721026 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:09:40.720994    2565 scope.go:117] "RemoveContainer" containerID="1fda79b79d99855eb92515ebecd25dc0ed3f52150df533bb538474c3f53b5f01"
Jul 16 23:09:40.721463 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:09:40.721191    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:09:51.720908 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:09:51.720869    2565 scope.go:117] "RemoveContainer" containerID="1fda79b79d99855eb92515ebecd25dc0ed3f52150df533bb538474c3f53b5f01"
Jul 16 23:09:51.721437 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:09:51.721089    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:10:05.720708 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:10:05.720671    2565 scope.go:117] "RemoveContainer" containerID="1fda79b79d99855eb92515ebecd25dc0ed3f52150df533bb538474c3f53b5f01"
Jul 16 23:10:05.721183 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:10:05.720898    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:10:20.720534 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:10:20.720494    2565 scope.go:117] "RemoveContainer" containerID="1fda79b79d99855eb92515ebecd25dc0ed3f52150df533bb538474c3f53b5f01"
Jul 16 23:10:20.721097 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:10:20.720778    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:10:32.720972 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:10:32.720882    2565 scope.go:117] "RemoveContainer" containerID="1fda79b79d99855eb92515ebecd25dc0ed3f52150df533bb538474c3f53b5f01"
Jul 16 23:10:32.721425 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:10:32.721090    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:10:43.720391 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:10:43.720357    2565 scope.go:117] "RemoveContainer" containerID="1fda79b79d99855eb92515ebecd25dc0ed3f52150df533bb538474c3f53b5f01"
Jul 16 23:10:43.720791 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:10:43.720562    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:10:58.720696 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:10:58.720658    2565 scope.go:117] "RemoveContainer" containerID="1fda79b79d99855eb92515ebecd25dc0ed3f52150df533bb538474c3f53b5f01"
Jul 16 23:10:58.721205 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:10:58.720873    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:11:10.720614 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:11:10.720579    2565 scope.go:117] "RemoveContainer" containerID="1fda79b79d99855eb92515ebecd25dc0ed3f52150df533bb538474c3f53b5f01"
Jul 16 23:11:10.723185 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:11:10.720777    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:11:25.720340 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:11:25.720300    2565 scope.go:117] "RemoveContainer" containerID="1fda79b79d99855eb92515ebecd25dc0ed3f52150df533bb538474c3f53b5f01"
Jul 16 23:11:25.720806 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:11:25.720523    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:11:40.721120 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:11:40.721084    2565 scope.go:117] "RemoveContainer" containerID="1fda79b79d99855eb92515ebecd25dc0ed3f52150df533bb538474c3f53b5f01"
Jul 16 23:11:41.446430 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:11:41.446394    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" event={"ID":"71a54521-363e-4fa7-8b7e-7ae778ed472e","Type":"ContainerStarted","Data":"881c9efaec3b2589764cc88a12f33063dcbd41a72947bcce1d7763ff2c223f02"}
Jul 16 23:11:41.446682 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:11:41.446659    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:11:41.465928 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:11:41.465865    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podStartSLOduration=343.533547525 podStartE2EDuration="5m46.465847264s" totalImagesPullingTime="2.932299739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:05:55 +0000 UTC" imagePullSessionsCount=7 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:11:41.463137207 +0000 UTC m=+2312.211621281" watchObservedRunningTime="2026-07-16 23:11:41.465847264 +0000 UTC m=+2312.214331335"
Jul 16 23:11:42.453052 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:11:42.453019    2565 generic.go:356] "Generic (PLEG): container finished" podID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerID="881c9efaec3b2589764cc88a12f33063dcbd41a72947bcce1d7763ff2c223f02" exitCode=1
Jul 16 23:11:42.453573 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:11:42.453116    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" event={"ID":"71a54521-363e-4fa7-8b7e-7ae778ed472e","Type":"ContainerDied","Data":"881c9efaec3b2589764cc88a12f33063dcbd41a72947bcce1d7763ff2c223f02"}
Jul 16 23:11:42.453573 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:11:42.453186    2565 scope.go:117] "RemoveContainer" containerID="1fda79b79d99855eb92515ebecd25dc0ed3f52150df533bb538474c3f53b5f01"
Jul 16 23:11:42.453573 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:11:42.453439    2565 scope.go:117] "RemoveContainer" containerID="881c9efaec3b2589764cc88a12f33063dcbd41a72947bcce1d7763ff2c223f02"
Jul 16 23:11:42.453764 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:11:42.453665    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:11:43.459457 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:11:43.459425    2565 scope.go:117] "RemoveContainer" containerID="881c9efaec3b2589764cc88a12f33063dcbd41a72947bcce1d7763ff2c223f02"
Jul 16 23:11:43.459851 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:11:43.459618    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:11:45.682323 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:11:45.682287    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:11:45.682746 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:11:45.682731    2565 scope.go:117] "RemoveContainer" containerID="881c9efaec3b2589764cc88a12f33063dcbd41a72947bcce1d7763ff2c223f02"
Jul 16 23:11:45.682924 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:11:45.682902    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:11:57.720663 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:11:57.720625    2565 scope.go:117] "RemoveContainer" containerID="881c9efaec3b2589764cc88a12f33063dcbd41a72947bcce1d7763ff2c223f02"
Jul 16 23:11:57.721225 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:11:57.720877    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb_odh-ai-gateway-infra(71a54521-363e-4fa7-8b7e-7ae778ed472e)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e"
Jul 16 23:12:00.434440 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:00.434399    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"]
Jul 16 23:12:00.434919 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:00.434610    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz" podUID="d9047909-33ef-44b2-9934-0804433a485d" containerName="istio-proxy" containerID="cri-o://a5a45f3a6178bbbcb6cfffe221551fbbd40c6fa59ece6f9c65ccd655380b9a63" gracePeriod=30
Jul 16 23:12:05.572070 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.572035    2565 generic.go:356] "Generic (PLEG): container finished" podID="d9047909-33ef-44b2-9934-0804433a485d" containerID="a5a45f3a6178bbbcb6cfffe221551fbbd40c6fa59ece6f9c65ccd655380b9a63" exitCode=0
Jul 16 23:12:05.572507 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.572090    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz" event={"ID":"d9047909-33ef-44b2-9934-0804433a485d","Type":"ContainerDied","Data":"a5a45f3a6178bbbcb6cfffe221551fbbd40c6fa59ece6f9c65ccd655380b9a63"}
Jul 16 23:12:05.572507 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.572184    2565 scope.go:117] "RemoveContainer" containerID="95fab35c8a42331cf577a1976b6500b46ef79b08351c29c05081874c96b9ff17"
Jul 16 23:12:05.697339 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.697308    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:12:05.715264 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.715158    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"]
Jul 16 23:12:05.745951 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.745922    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-workload-certs\") pod \"d9047909-33ef-44b2-9934-0804433a485d\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") "
Jul 16 23:12:05.745951 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.745955    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9047909-33ef-44b2-9934-0804433a485d-istio-token\") pod \"d9047909-33ef-44b2-9934-0804433a485d\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") "
Jul 16 23:12:05.746210 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.745977    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-istio-envoy\") pod \"d9047909-33ef-44b2-9934-0804433a485d\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") "
Jul 16 23:12:05.746210 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.746037    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-istio-data\") pod \"d9047909-33ef-44b2-9934-0804433a485d\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") "
Jul 16 23:12:05.746210 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.746100    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9047909-33ef-44b2-9934-0804433a485d-istio-podinfo\") pod \"d9047909-33ef-44b2-9934-0804433a485d\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") "
Jul 16 23:12:05.746210 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.746124    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-workload-socket\") pod \"d9047909-33ef-44b2-9934-0804433a485d\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") "
Jul 16 23:12:05.746210 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.746177    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b6kf\" (UniqueName: \"kubernetes.io/projected/d9047909-33ef-44b2-9934-0804433a485d-kube-api-access-8b6kf\") pod \"d9047909-33ef-44b2-9934-0804433a485d\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") "
Jul 16 23:12:05.746505 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.746237    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-credential-socket\") pod \"d9047909-33ef-44b2-9934-0804433a485d\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") "
Jul 16 23:12:05.746505 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.746260    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d9047909-33ef-44b2-9934-0804433a485d" (UID: "d9047909-33ef-44b2-9934-0804433a485d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:12:05.746505 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.746284    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9047909-33ef-44b2-9934-0804433a485d-istiod-ca-cert\") pod \"d9047909-33ef-44b2-9934-0804433a485d\" (UID: \"d9047909-33ef-44b2-9934-0804433a485d\") "
Jul 16 23:12:05.746670 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.746545    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-workload-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:12:05.746867 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.746843    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9047909-33ef-44b2-9934-0804433a485d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d9047909-33ef-44b2-9934-0804433a485d" (UID: "d9047909-33ef-44b2-9934-0804433a485d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:12:05.747127 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.747102    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d9047909-33ef-44b2-9934-0804433a485d" (UID: "d9047909-33ef-44b2-9934-0804433a485d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:12:05.747449 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.747200    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-istio-data" (OuterVolumeSpecName: "istio-data") pod "d9047909-33ef-44b2-9934-0804433a485d" (UID: "d9047909-33ef-44b2-9934-0804433a485d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:12:05.747818 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.747773    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d9047909-33ef-44b2-9934-0804433a485d" (UID: "d9047909-33ef-44b2-9934-0804433a485d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:12:05.749285 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.749204    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9047909-33ef-44b2-9934-0804433a485d-kube-api-access-8b6kf" (OuterVolumeSpecName: "kube-api-access-8b6kf") pod "d9047909-33ef-44b2-9934-0804433a485d" (UID: "d9047909-33ef-44b2-9934-0804433a485d"). InnerVolumeSpecName "kube-api-access-8b6kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:12:05.749873 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.749840    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9047909-33ef-44b2-9934-0804433a485d-istio-token" (OuterVolumeSpecName: "istio-token") pod "d9047909-33ef-44b2-9934-0804433a485d" (UID: "d9047909-33ef-44b2-9934-0804433a485d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:12:05.750132 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.750107    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9047909-33ef-44b2-9934-0804433a485d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d9047909-33ef-44b2-9934-0804433a485d" (UID: "d9047909-33ef-44b2-9934-0804433a485d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 23:12:05.750213 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.750118    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d9047909-33ef-44b2-9934-0804433a485d" (UID: "d9047909-33ef-44b2-9934-0804433a485d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:12:05.848051 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.848010    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9047909-33ef-44b2-9934-0804433a485d-istio-token\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:12:05.848051 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.848042    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-istio-envoy\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:12:05.848051 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.848052    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-istio-data\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:12:05.848051 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.848061    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9047909-33ef-44b2-9934-0804433a485d-istio-podinfo\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:12:05.848441 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.848069    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-workload-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:12:05.848441 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.848077    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b6kf\" (UniqueName: \"kubernetes.io/projected/d9047909-33ef-44b2-9934-0804433a485d-kube-api-access-8b6kf\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:12:05.848441 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.848087    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9047909-33ef-44b2-9934-0804433a485d-credential-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:12:05.848441 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.848095    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9047909-33ef-44b2-9934-0804433a485d-istiod-ca-cert\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:12:05.871786 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.871760    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:12:05.949103 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.949065    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wzcs\" (UniqueName: \"kubernetes.io/projected/71a54521-363e-4fa7-8b7e-7ae778ed472e-kube-api-access-8wzcs\") pod \"71a54521-363e-4fa7-8b7e-7ae778ed472e\" (UID: \"71a54521-363e-4fa7-8b7e-7ae778ed472e\") "
Jul 16 23:12:05.949103 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.949111    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a54521-363e-4fa7-8b7e-7ae778ed472e-openshift-service-ca\") pod \"71a54521-363e-4fa7-8b7e-7ae778ed472e\" (UID: \"71a54521-363e-4fa7-8b7e-7ae778ed472e\") "
Jul 16 23:12:05.949400 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.949158    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a54521-363e-4fa7-8b7e-7ae778ed472e-maas-api-tls\") pod \"71a54521-363e-4fa7-8b7e-7ae778ed472e\" (UID: \"71a54521-363e-4fa7-8b7e-7ae778ed472e\") "
Jul 16 23:12:05.949606 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.949575    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a54521-363e-4fa7-8b7e-7ae778ed472e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "71a54521-363e-4fa7-8b7e-7ae778ed472e" (UID: "71a54521-363e-4fa7-8b7e-7ae778ed472e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:12:05.951356 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.951333    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a54521-363e-4fa7-8b7e-7ae778ed472e-kube-api-access-8wzcs" (OuterVolumeSpecName: "kube-api-access-8wzcs") pod "71a54521-363e-4fa7-8b7e-7ae778ed472e" (UID: "71a54521-363e-4fa7-8b7e-7ae778ed472e"). InnerVolumeSpecName "kube-api-access-8wzcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:12:05.951356 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:05.951347    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a54521-363e-4fa7-8b7e-7ae778ed472e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "71a54521-363e-4fa7-8b7e-7ae778ed472e" (UID: "71a54521-363e-4fa7-8b7e-7ae778ed472e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:12:06.050415 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:06.050312    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71a54521-363e-4fa7-8b7e-7ae778ed472e-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:12:06.050415 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:06.050349    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wzcs\" (UniqueName: \"kubernetes.io/projected/71a54521-363e-4fa7-8b7e-7ae778ed472e-kube-api-access-8wzcs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:12:06.050415 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:06.050363    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71a54521-363e-4fa7-8b7e-7ae778ed472e-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:12:06.578358 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:06.578325    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"
Jul 16 23:12:06.578358 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:06.578349    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz" event={"ID":"d9047909-33ef-44b2-9934-0804433a485d","Type":"ContainerDied","Data":"c47e0af644d363b04991facdcd5e0e92d2062f48ceda378462d67718ada8e324"}
Jul 16 23:12:06.578881 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:06.578413    2565 scope.go:117] "RemoveContainer" containerID="a5a45f3a6178bbbcb6cfffe221551fbbd40c6fa59ece6f9c65ccd655380b9a63"
Jul 16 23:12:06.579754 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:06.579715    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb" event={"ID":"71a54521-363e-4fa7-8b7e-7ae778ed472e","Type":"ContainerDied","Data":"9042dc643a697f2d4ed88565bead41c503d7dbd0082947091fcb3f99bcdac41d"}
Jul 16 23:12:06.579855 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:06.579737    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"
Jul 16 23:12:06.587305 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:06.587280    2565 scope.go:117] "RemoveContainer" containerID="881c9efaec3b2589764cc88a12f33063dcbd41a72947bcce1d7763ff2c223f02"
Jul 16 23:12:06.603586 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:06.603553    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"]
Jul 16 23:12:06.607138 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:06.607109    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d48f5b-6c94db4d44-7x4mb"]
Jul 16 23:12:06.618970 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:06.618934    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"]
Jul 16 23:12:06.622517 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:06.622486    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a1d48f5b-openshift-default-85545bc7cc-6l4pz"]
Jul 16 23:12:07.726024 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:07.725988    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" path="/var/lib/kubelet/pods/71a54521-363e-4fa7-8b7e-7ae778ed472e/volumes"
Jul 16 23:12:07.726502 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:07.726487    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9047909-33ef-44b2-9934-0804433a485d" path="/var/lib/kubelet/pods/d9047909-33ef-44b2-9934-0804433a485d/volumes"
Jul 16 23:12:12.073284 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.073234    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"]
Jul 16 23:12:12.073778 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.073762    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:12.073848 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.073782    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:12.073848 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.073795    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:12.073848 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.073803    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:12.073848 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.073821    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:12.073848 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.073830    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:12.074113 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.073848    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1399565b-99c9-4b8f-9ab2-a6b330fe50e7" containerName="revoke-keys"
Jul 16 23:12:12.074113 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.073857    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1399565b-99c9-4b8f-9ab2-a6b330fe50e7" containerName="revoke-keys"
Jul 16 23:12:12.074113 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.073876    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9047909-33ef-44b2-9934-0804433a485d" containerName="istio-proxy"
Jul 16 23:12:12.074113 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.073884    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9047909-33ef-44b2-9934-0804433a485d" containerName="istio-proxy"
Jul 16 23:12:12.074113 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.073897    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1399565b-99c9-4b8f-9ab2-a6b330fe50e7" containerName="revoke-keys"
Jul 16 23:12:12.074113 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.073905    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1399565b-99c9-4b8f-9ab2-a6b330fe50e7" containerName="revoke-keys"
Jul 16 23:12:12.074113 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.073917    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:12.074113 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.073925    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:12.074113 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.073934    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9047909-33ef-44b2-9934-0804433a485d" containerName="istio-proxy"
Jul 16 23:12:12.074113 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.073943    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9047909-33ef-44b2-9934-0804433a485d" containerName="istio-proxy"
Jul 16 23:12:12.074113 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.074035    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:12.074113 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.074049    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:12.074113 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.074060    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9047909-33ef-44b2-9934-0804433a485d" containerName="istio-proxy"
Jul 16 23:12:12.074768 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.074132    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:12.074768 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.074143    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:12.074768 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.074172    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1399565b-99c9-4b8f-9ab2-a6b330fe50e7" containerName="revoke-keys"
Jul 16 23:12:12.074768 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.074182    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9047909-33ef-44b2-9934-0804433a485d" containerName="istio-proxy"
Jul 16 23:12:12.074768 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.074193    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1399565b-99c9-4b8f-9ab2-a6b330fe50e7" containerName="revoke-keys"
Jul 16 23:12:12.074768 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.074204    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1399565b-99c9-4b8f-9ab2-a6b330fe50e7" containerName="revoke-keys"
Jul 16 23:12:12.092716 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.092678    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"]
Jul 16 23:12:12.092906 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.092884    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.097133 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.097105    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f263f2f2-openshift-default-dockercfg-f4ggq\""
Jul 16 23:12:12.203534 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.203496    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-data\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.203534 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.203539    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-token\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.203771 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.203561    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istiod-ca-cert\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.203771 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.203695    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-workload-certs\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.203771 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.203740    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-podinfo\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.203946 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.203815    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-credential-socket\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.203946 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.203855    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-workload-socket\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.203946 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.203890    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrz62\" (UniqueName: \"kubernetes.io/projected/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-kube-api-access-zrz62\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.203946 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.203920    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-envoy\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.305358 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.305322    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-data\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.305544 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.305365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-token\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.305544 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.305386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istiod-ca-cert\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.305544 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.305426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-workload-certs\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.305734 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.305575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-podinfo\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.305734 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.305643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-credential-socket\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.305734 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.305675    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-workload-socket\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.305734 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.305714    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrz62\" (UniqueName: \"kubernetes.io/projected/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-kube-api-access-zrz62\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.305948 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.305744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-envoy\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.305948 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.305777    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-data\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.306056 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.306016    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-workload-socket\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.306159 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.306130    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-credential-socket\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.306328 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.306265    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-workload-certs\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.306663 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.306639    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istiod-ca-cert\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.308361 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.308335    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-podinfo\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.308472 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.308391    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-envoy\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.313518 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.313489    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrz62\" (UniqueName: \"kubernetes.io/projected/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-kube-api-access-zrz62\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.313661 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.313646    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-token\") pod \"e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") " pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.405740 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.405699    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:12.547193 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.547163    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"]
Jul 16 23:12:12.549415 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:12:12.549386    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8f6578_4632_459d_a15b_22e8d7c9ee5a.slice/crio-b3b3e5e5db43d5bb40e166f66369634be9088a41646729506f5f0ec41bd3c4d5 WatchSource:0}: Error finding container b3b3e5e5db43d5bb40e166f66369634be9088a41646729506f5f0ec41bd3c4d5: Status 404 returned error can't find the container with id b3b3e5e5db43d5bb40e166f66369634be9088a41646729506f5f0ec41bd3c4d5
Jul 16 23:12:12.551632 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.551595    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:12:12.551719 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.551665    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:12:12.551719 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.551695    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:12:12.609923 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:12.609890    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg" event={"ID":"2f8f6578-4632-459d-a15b-22e8d7c9ee5a","Type":"ContainerStarted","Data":"b3b3e5e5db43d5bb40e166f66369634be9088a41646729506f5f0ec41bd3c4d5"}
Jul 16 23:12:13.344378 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.344342    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"]
Jul 16 23:12:13.344877 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.344744    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:13.344877 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.344761    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:13.344877 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.344769    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:13.344877 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.344776    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:13.344877 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.344785    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:13.344877 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.344791    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:13.344877 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.344807    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1399565b-99c9-4b8f-9ab2-a6b330fe50e7" containerName="revoke-keys"
Jul 16 23:12:13.344877 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.344813    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1399565b-99c9-4b8f-9ab2-a6b330fe50e7" containerName="revoke-keys"
Jul 16 23:12:13.344877 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.344873    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:13.345215 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.344884    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:13.345215 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.344889    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a54521-363e-4fa7-8b7e-7ae778ed472e" containerName="maas-api"
Jul 16 23:12:13.347988 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.347970    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:12:13.350132 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.350104    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f263f2f2-serving-cert\""
Jul 16 23:12:13.358821 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.358794    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"]
Jul 16 23:12:13.418004 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.417959    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646ea6a5-9067-42ac-aa64-7863ccb3f553-openshift-service-ca\") pod \"maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r\" (UID: \"646ea6a5-9067-42ac-aa64-7863ccb3f553\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:12:13.418218 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.417999    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58k6\" (UniqueName: \"kubernetes.io/projected/646ea6a5-9067-42ac-aa64-7863ccb3f553-kube-api-access-d58k6\") pod \"maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r\" (UID: \"646ea6a5-9067-42ac-aa64-7863ccb3f553\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:12:13.418218 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.418127    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646ea6a5-9067-42ac-aa64-7863ccb3f553-maas-api-tls\") pod \"maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r\" (UID: \"646ea6a5-9067-42ac-aa64-7863ccb3f553\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:12:13.519544 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.519499    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646ea6a5-9067-42ac-aa64-7863ccb3f553-openshift-service-ca\") pod \"maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r\" (UID: \"646ea6a5-9067-42ac-aa64-7863ccb3f553\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:12:13.519544 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.519538    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58k6\" (UniqueName: \"kubernetes.io/projected/646ea6a5-9067-42ac-aa64-7863ccb3f553-kube-api-access-d58k6\") pod \"maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r\" (UID: \"646ea6a5-9067-42ac-aa64-7863ccb3f553\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:12:13.519808 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.519587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646ea6a5-9067-42ac-aa64-7863ccb3f553-maas-api-tls\") pod \"maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r\" (UID: \"646ea6a5-9067-42ac-aa64-7863ccb3f553\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:12:13.520370 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.520344    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646ea6a5-9067-42ac-aa64-7863ccb3f553-openshift-service-ca\") pod \"maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r\" (UID: \"646ea6a5-9067-42ac-aa64-7863ccb3f553\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:12:13.522221 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.522196    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646ea6a5-9067-42ac-aa64-7863ccb3f553-maas-api-tls\") pod \"maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r\" (UID: \"646ea6a5-9067-42ac-aa64-7863ccb3f553\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:12:13.527001 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.526976    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58k6\" (UniqueName: \"kubernetes.io/projected/646ea6a5-9067-42ac-aa64-7863ccb3f553-kube-api-access-d58k6\") pod \"maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r\" (UID: \"646ea6a5-9067-42ac-aa64-7863ccb3f553\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:12:13.616644 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.616554    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg" event={"ID":"2f8f6578-4632-459d-a15b-22e8d7c9ee5a","Type":"ContainerStarted","Data":"3e591402620c66405eecc9fcf8025535a3e415ad5c2edadd4f29bf62c38d0b96"}
Jul 16 23:12:13.639619 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.639561    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg" podStartSLOduration=1.639543851 podStartE2EDuration="1.639543851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:12:13.637223299 +0000 UTC m=+2344.385707372" watchObservedRunningTime="2026-07-16 23:12:13.639543851 +0000 UTC m=+2344.388028003"
Jul 16 23:12:13.661401 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.661361    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:12:13.811996 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.811968    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"]
Jul 16 23:12:13.814455 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:12:13.814429    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646ea6a5_9067_42ac_aa64_7863ccb3f553.slice/crio-21085e55428c5c1caebf2d88b8be1fbfc83a7c46a55a1882cde149dd90db4e9c WatchSource:0}: Error finding container 21085e55428c5c1caebf2d88b8be1fbfc83a7c46a55a1882cde149dd90db4e9c: Status 404 returned error can't find the container with id 21085e55428c5c1caebf2d88b8be1fbfc83a7c46a55a1882cde149dd90db4e9c
Jul 16 23:12:13.815911 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:13.815890    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 23:12:14.406756 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:14.406710    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:14.412370 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:14.412341    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:14.622736 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:14.622633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" event={"ID":"646ea6a5-9067-42ac-aa64-7863ccb3f553","Type":"ContainerStarted","Data":"b812ca593d0829fc21f15e85fbadebd9f088c19222467bb801cc8a0232d61802"}
Jul 16 23:12:14.622736 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:14.622688    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" event={"ID":"646ea6a5-9067-42ac-aa64-7863ccb3f553","Type":"ContainerStarted","Data":"21085e55428c5c1caebf2d88b8be1fbfc83a7c46a55a1882cde149dd90db4e9c"}
Jul 16 23:12:14.622736 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:14.622708    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:12:14.623024 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:14.623009    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:14.624064 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:14.624041    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:14.639197 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:14.639146    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podStartSLOduration=1.118677186 podStartE2EDuration="1.639128376s" totalImagesPullingTime="520.45119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:12:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:12:14.63796635 +0000 UTC m=+2345.386450422" watchObservedRunningTime="2026-07-16 23:12:14.639128376 +0000 UTC m=+2345.387612483"
Jul 16 23:12:15.628785 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:15.628753    2565 generic.go:356] "Generic (PLEG): container finished" podID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerID="b812ca593d0829fc21f15e85fbadebd9f088c19222467bb801cc8a0232d61802" exitCode=1
Jul 16 23:12:15.629195 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:15.628844    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" event={"ID":"646ea6a5-9067-42ac-aa64-7863ccb3f553","Type":"ContainerDied","Data":"b812ca593d0829fc21f15e85fbadebd9f088c19222467bb801cc8a0232d61802"}
Jul 16 23:12:15.629195 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:15.629083    2565 scope.go:117] "RemoveContainer" containerID="b812ca593d0829fc21f15e85fbadebd9f088c19222467bb801cc8a0232d61802"
Jul 16 23:12:16.636733 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:16.636692    2565 generic.go:356] "Generic (PLEG): container finished" podID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerID="155e29042bc7620dcf408ff51592b1c02ad2d7572762e9343dd1d15812ce8eec" exitCode=1
Jul 16 23:12:16.637142 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:16.636765    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" event={"ID":"646ea6a5-9067-42ac-aa64-7863ccb3f553","Type":"ContainerDied","Data":"155e29042bc7620dcf408ff51592b1c02ad2d7572762e9343dd1d15812ce8eec"}
Jul 16 23:12:16.637142 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:16.636815    2565 scope.go:117] "RemoveContainer" containerID="b812ca593d0829fc21f15e85fbadebd9f088c19222467bb801cc8a0232d61802"
Jul 16 23:12:16.637142 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:16.637126    2565 scope.go:117] "RemoveContainer" containerID="155e29042bc7620dcf408ff51592b1c02ad2d7572762e9343dd1d15812ce8eec"
Jul 16 23:12:16.637382 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:12:16.637352    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:12:17.647545 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:17.647509    2565 scope.go:117] "RemoveContainer" containerID="155e29042bc7620dcf408ff51592b1c02ad2d7572762e9343dd1d15812ce8eec"
Jul 16 23:12:17.648056 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:12:17.647800    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:12:19.623465 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:19.623427    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:12:19.623892 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:19.623879    2565 scope.go:117] "RemoveContainer" containerID="155e29042bc7620dcf408ff51592b1c02ad2d7572762e9343dd1d15812ce8eec"
Jul 16 23:12:19.624096 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:12:19.624069    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:12:23.662292 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:23.662230    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:12:23.662704 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:23.662676    2565 scope.go:117] "RemoveContainer" containerID="155e29042bc7620dcf408ff51592b1c02ad2d7572762e9343dd1d15812ce8eec"
Jul 16 23:12:23.662876 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:12:23.662854    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:12:24.690688 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:24.690655    2565 generic.go:356] "Generic (PLEG): container finished" podID="2f8f6578-4632-459d-a15b-22e8d7c9ee5a" containerID="3e591402620c66405eecc9fcf8025535a3e415ad5c2edadd4f29bf62c38d0b96" exitCode=0
Jul 16 23:12:24.691116 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:24.690725    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg" event={"ID":"2f8f6578-4632-459d-a15b-22e8d7c9ee5a","Type":"ContainerDied","Data":"3e591402620c66405eecc9fcf8025535a3e415ad5c2edadd4f29bf62c38d0b96"}
Jul 16 23:12:24.691161 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:24.691114    2565 scope.go:117] "RemoveContainer" containerID="3e591402620c66405eecc9fcf8025535a3e415ad5c2edadd4f29bf62c38d0b96"
Jul 16 23:12:24.692201 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:24.692175    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:12:24.692297 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:24.692242    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:12:24.692297 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:24.692288    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:12:25.697315 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:25.697275    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg" event={"ID":"2f8f6578-4632-459d-a15b-22e8d7c9ee5a","Type":"ContainerStarted","Data":"c5cf185646bf94d1e14a94902338cd73b65752ce99a5bd4aac3dbeed35496884"}
Jul 16 23:12:26.406544 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:26.406502    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:26.407746 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:26.407722    2565 patch_prober.go:28] interesting pod/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Jul 16 23:12:26.407796 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:26.407776    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg" podUID="2f8f6578-4632-459d-a15b-22e8d7c9ee5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Jul 16 23:12:27.406705 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:27.406664    2565 patch_prober.go:28] interesting pod/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Jul 16 23:12:27.407134 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:27.406717    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg" podUID="2f8f6578-4632-459d-a15b-22e8d7c9ee5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Jul 16 23:12:28.407123 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:28.407081    2565 patch_prober.go:28] interesting pod/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Jul 16 23:12:28.407584 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:28.407143    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg" podUID="2f8f6578-4632-459d-a15b-22e8d7c9ee5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Jul 16 23:12:29.407091 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:29.407053    2565 patch_prober.go:28] interesting pod/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body=
Jul 16 23:12:29.407449 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:29.407119    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg" podUID="2f8f6578-4632-459d-a15b-22e8d7c9ee5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused"
Jul 16 23:12:29.623649 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:29.623603    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:30.410808 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:30.410773    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:30.411860 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:30.411836    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:12:35.720776 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:35.720735    2565 scope.go:117] "RemoveContainer" containerID="155e29042bc7620dcf408ff51592b1c02ad2d7572762e9343dd1d15812ce8eec"
Jul 16 23:12:36.751518 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:36.751432    2565 generic.go:356] "Generic (PLEG): container finished" podID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerID="7ed653adb60e9287cd03ec7220299b95da8c8025c94cf590ef041e97d36b8b73" exitCode=1
Jul 16 23:12:36.751518 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:36.751499    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" event={"ID":"646ea6a5-9067-42ac-aa64-7863ccb3f553","Type":"ContainerDied","Data":"7ed653adb60e9287cd03ec7220299b95da8c8025c94cf590ef041e97d36b8b73"}
Jul 16 23:12:36.752052 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:36.751559    2565 scope.go:117] "RemoveContainer" containerID="155e29042bc7620dcf408ff51592b1c02ad2d7572762e9343dd1d15812ce8eec"
Jul 16 23:12:36.752052 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:36.751915    2565 scope.go:117] "RemoveContainer" containerID="7ed653adb60e9287cd03ec7220299b95da8c8025c94cf590ef041e97d36b8b73"
Jul 16 23:12:36.752165 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:12:36.752140    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:12:39.622979 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:39.622941    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:12:39.623427 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:39.623399    2565 scope.go:117] "RemoveContainer" containerID="7ed653adb60e9287cd03ec7220299b95da8c8025c94cf590ef041e97d36b8b73"
Jul 16 23:12:39.623596 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:12:39.623576    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:12:43.661734 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:43.661697    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:12:43.662149 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:43.662075    2565 scope.go:117] "RemoveContainer" containerID="7ed653adb60e9287cd03ec7220299b95da8c8025c94cf590ef041e97d36b8b73"
Jul 16 23:12:43.662289 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:12:43.662241    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:12:57.720996 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:57.720964    2565 scope.go:117] "RemoveContainer" containerID="7ed653adb60e9287cd03ec7220299b95da8c8025c94cf590ef041e97d36b8b73"
Jul 16 23:12:58.859681 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:58.859647    2565 generic.go:356] "Generic (PLEG): container finished" podID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerID="c1d91f8760f19e32b55d396d703f43ea7fa8b40eaaf82287758ae3b09105a587" exitCode=1
Jul 16 23:12:58.860103 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:58.859719    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" event={"ID":"646ea6a5-9067-42ac-aa64-7863ccb3f553","Type":"ContainerDied","Data":"c1d91f8760f19e32b55d396d703f43ea7fa8b40eaaf82287758ae3b09105a587"}
Jul 16 23:12:58.860103 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:58.859780    2565 scope.go:117] "RemoveContainer" containerID="7ed653adb60e9287cd03ec7220299b95da8c8025c94cf590ef041e97d36b8b73"
Jul 16 23:12:58.860203 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:58.860118    2565 scope.go:117] "RemoveContainer" containerID="c1d91f8760f19e32b55d396d703f43ea7fa8b40eaaf82287758ae3b09105a587"
Jul 16 23:12:58.860356 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:12:58.860328    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:12:59.623761 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:59.623721    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:12:59.865874 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:12:59.865848    2565 scope.go:117] "RemoveContainer" containerID="c1d91f8760f19e32b55d396d703f43ea7fa8b40eaaf82287758ae3b09105a587"
Jul 16 23:12:59.866287 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:12:59.866060    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:13:03.662308 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:13:03.662268    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:13:03.662718 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:13:03.662663    2565 scope.go:117] "RemoveContainer" containerID="c1d91f8760f19e32b55d396d703f43ea7fa8b40eaaf82287758ae3b09105a587"
Jul 16 23:13:03.662885 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:13:03.662865    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:13:17.720475 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:13:17.720442    2565 scope.go:117] "RemoveContainer" containerID="c1d91f8760f19e32b55d396d703f43ea7fa8b40eaaf82287758ae3b09105a587"
Jul 16 23:13:17.721003 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:13:17.720632    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:13:30.721120 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:13:30.721087    2565 scope.go:117] "RemoveContainer" containerID="c1d91f8760f19e32b55d396d703f43ea7fa8b40eaaf82287758ae3b09105a587"
Jul 16 23:13:30.721659 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:13:30.721279    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:13:41.720603 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:13:41.720516    2565 scope.go:117] "RemoveContainer" containerID="c1d91f8760f19e32b55d396d703f43ea7fa8b40eaaf82287758ae3b09105a587"
Jul 16 23:13:43.076318 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:13:43.076279    2565 generic.go:356] "Generic (PLEG): container finished" podID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerID="2891f16de921b7a58ccde33c239b2d0f55eafd8d88e8ca60184d9dde38031410" exitCode=1
Jul 16 23:13:43.076788 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:13:43.076335    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" event={"ID":"646ea6a5-9067-42ac-aa64-7863ccb3f553","Type":"ContainerDied","Data":"2891f16de921b7a58ccde33c239b2d0f55eafd8d88e8ca60184d9dde38031410"}
Jul 16 23:13:43.076788 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:13:43.076404    2565 scope.go:117] "RemoveContainer" containerID="c1d91f8760f19e32b55d396d703f43ea7fa8b40eaaf82287758ae3b09105a587"
Jul 16 23:13:43.076788 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:13:43.076688    2565 scope.go:117] "RemoveContainer" containerID="2891f16de921b7a58ccde33c239b2d0f55eafd8d88e8ca60184d9dde38031410"
Jul 16 23:13:43.076990 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:13:43.076949    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:13:43.662162 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:13:43.662123    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:13:44.083886 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:13:44.083858    2565 scope.go:117] "RemoveContainer" containerID="2891f16de921b7a58ccde33c239b2d0f55eafd8d88e8ca60184d9dde38031410"
Jul 16 23:13:44.084285 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:13:44.084032    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:13:44.623696 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:13:44.623660    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:13:45.088207 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:13:45.088180    2565 scope.go:117] "RemoveContainer" containerID="2891f16de921b7a58ccde33c239b2d0f55eafd8d88e8ca60184d9dde38031410"
Jul 16 23:13:45.088640 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:13:45.088391    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:13:59.723652 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:13:59.723618    2565 scope.go:117] "RemoveContainer" containerID="2891f16de921b7a58ccde33c239b2d0f55eafd8d88e8ca60184d9dde38031410"
Jul 16 23:13:59.724087 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:13:59.723842    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:14:10.720258 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:14:10.720226    2565 scope.go:117] "RemoveContainer" containerID="2891f16de921b7a58ccde33c239b2d0f55eafd8d88e8ca60184d9dde38031410"
Jul 16 23:14:10.720756 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:14:10.720462    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:14:25.720567 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:14:25.720532    2565 scope.go:117] "RemoveContainer" containerID="2891f16de921b7a58ccde33c239b2d0f55eafd8d88e8ca60184d9dde38031410"
Jul 16 23:14:25.721035 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:14:25.720767    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:14:37.721119 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:14:37.721078    2565 scope.go:117] "RemoveContainer" containerID="2891f16de921b7a58ccde33c239b2d0f55eafd8d88e8ca60184d9dde38031410"
Jul 16 23:14:37.721605 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:14:37.721294    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:14:52.720931 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:14:52.720898    2565 scope.go:117] "RemoveContainer" containerID="2891f16de921b7a58ccde33c239b2d0f55eafd8d88e8ca60184d9dde38031410"
Jul 16 23:14:52.721368 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:14:52.721100    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:15:03.720563 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:03.720480    2565 scope.go:117] "RemoveContainer" containerID="2891f16de921b7a58ccde33c239b2d0f55eafd8d88e8ca60184d9dde38031410"
Jul 16 23:15:04.476026 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:04.475992    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" event={"ID":"646ea6a5-9067-42ac-aa64-7863ccb3f553","Type":"ContainerStarted","Data":"02a139dfea6f807d2c1b9cc29c6f378e0e92b0548d4a21917f5dcba1a18f9c60"}
Jul 16 23:15:04.476217 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:04.476200    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:15:05.483605 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:05.483571    2565 generic.go:356] "Generic (PLEG): container finished" podID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerID="02a139dfea6f807d2c1b9cc29c6f378e0e92b0548d4a21917f5dcba1a18f9c60" exitCode=1
Jul 16 23:15:05.484070 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:05.483661    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" event={"ID":"646ea6a5-9067-42ac-aa64-7863ccb3f553","Type":"ContainerDied","Data":"02a139dfea6f807d2c1b9cc29c6f378e0e92b0548d4a21917f5dcba1a18f9c60"}
Jul 16 23:15:05.484070 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:05.483721    2565 scope.go:117] "RemoveContainer" containerID="2891f16de921b7a58ccde33c239b2d0f55eafd8d88e8ca60184d9dde38031410"
Jul 16 23:15:05.484070 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:05.483868    2565 scope.go:117] "RemoveContainer" containerID="02a139dfea6f807d2c1b9cc29c6f378e0e92b0548d4a21917f5dcba1a18f9c60"
Jul 16 23:15:05.484234 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:15:05.484079    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:15:06.490826 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:06.490799    2565 scope.go:117] "RemoveContainer" containerID="02a139dfea6f807d2c1b9cc29c6f378e0e92b0548d4a21917f5dcba1a18f9c60"
Jul 16 23:15:06.491204 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:15:06.490991    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:15:13.662259 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:13.662219    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:15:13.662765 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:13.662686    2565 scope.go:117] "RemoveContainer" containerID="02a139dfea6f807d2c1b9cc29c6f378e0e92b0548d4a21917f5dcba1a18f9c60"
Jul 16 23:15:13.662895 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:15:13.662873    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r_odh-ai-gateway-infra(646ea6a5-9067-42ac-aa64-7863ccb3f553)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553"
Jul 16 23:15:17.853707 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:17.853664    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"]
Jul 16 23:15:17.854228 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:17.853916    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg" podUID="2f8f6578-4632-459d-a15b-22e8d7c9ee5a" containerName="istio-proxy" containerID="cri-o://c5cf185646bf94d1e14a94902338cd73b65752ce99a5bd4aac3dbeed35496884" gracePeriod=30
Jul 16 23:15:23.113125 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.113098    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:15:23.192366 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.192323    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"]
Jul 16 23:15:23.207152 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.207120    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-workload-certs\") pod \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") "
Jul 16 23:15:23.207342 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.207174    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-workload-socket\") pod \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") "
Jul 16 23:15:23.207342 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.207212    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-data\") pod \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") "
Jul 16 23:15:23.207342 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.207242    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-podinfo\") pod \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") "
Jul 16 23:15:23.207342 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.207282    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrz62\" (UniqueName: \"kubernetes.io/projected/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-kube-api-access-zrz62\") pod \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") "
Jul 16 23:15:23.207342 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.207318    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-token\") pod \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") "
Jul 16 23:15:23.207618 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.207368    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istiod-ca-cert\") pod \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") "
Jul 16 23:15:23.207618 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.207457    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-envoy\") pod \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") "
Jul 16 23:15:23.207618 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.207506    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-credential-socket\") pod \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\" (UID: \"2f8f6578-4632-459d-a15b-22e8d7c9ee5a\") "
Jul 16 23:15:23.207618 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.207551    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2f8f6578-4632-459d-a15b-22e8d7c9ee5a" (UID: "2f8f6578-4632-459d-a15b-22e8d7c9ee5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:15:23.207618 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.207561    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2f8f6578-4632-459d-a15b-22e8d7c9ee5a" (UID: "2f8f6578-4632-459d-a15b-22e8d7c9ee5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:15:23.207875 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.207854    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2f8f6578-4632-459d-a15b-22e8d7c9ee5a" (UID: "2f8f6578-4632-459d-a15b-22e8d7c9ee5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:15:23.207936 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.207878    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-workload-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:15:23.207936 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.207899    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-workload-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:15:23.209387 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.209353    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-data" (OuterVolumeSpecName: "istio-data") pod "2f8f6578-4632-459d-a15b-22e8d7c9ee5a" (UID: "2f8f6578-4632-459d-a15b-22e8d7c9ee5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:15:23.210609 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.210562    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2f8f6578-4632-459d-a15b-22e8d7c9ee5a" (UID: "2f8f6578-4632-459d-a15b-22e8d7c9ee5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:15:23.211167 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.211126    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2f8f6578-4632-459d-a15b-22e8d7c9ee5a" (UID: "2f8f6578-4632-459d-a15b-22e8d7c9ee5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:15:23.211300 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.211159    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2f8f6578-4632-459d-a15b-22e8d7c9ee5a" (UID: "2f8f6578-4632-459d-a15b-22e8d7c9ee5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 23:15:23.213984 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.213947    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-token" (OuterVolumeSpecName: "istio-token") pod "2f8f6578-4632-459d-a15b-22e8d7c9ee5a" (UID: "2f8f6578-4632-459d-a15b-22e8d7c9ee5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:15:23.214221 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.214182    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-kube-api-access-zrz62" (OuterVolumeSpecName: "kube-api-access-zrz62") pod "2f8f6578-4632-459d-a15b-22e8d7c9ee5a" (UID: "2f8f6578-4632-459d-a15b-22e8d7c9ee5a"). InnerVolumeSpecName "kube-api-access-zrz62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:15:23.309302 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.309271    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istiod-ca-cert\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:15:23.309302 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.309299    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-envoy\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:15:23.309302 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.309308    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-credential-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:15:23.309545 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.309318    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-data\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:15:23.309545 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.309329    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-podinfo\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:15:23.309545 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.309337    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrz62\" (UniqueName: \"kubernetes.io/projected/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-kube-api-access-zrz62\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:15:23.309545 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.309347    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f8f6578-4632-459d-a15b-22e8d7c9ee5a-istio-token\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:15:23.323728 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.323704    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:15:23.410512 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.410473    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646ea6a5-9067-42ac-aa64-7863ccb3f553-openshift-service-ca\") pod \"646ea6a5-9067-42ac-aa64-7863ccb3f553\" (UID: \"646ea6a5-9067-42ac-aa64-7863ccb3f553\") "
Jul 16 23:15:23.410690 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.410526    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646ea6a5-9067-42ac-aa64-7863ccb3f553-maas-api-tls\") pod \"646ea6a5-9067-42ac-aa64-7863ccb3f553\" (UID: \"646ea6a5-9067-42ac-aa64-7863ccb3f553\") "
Jul 16 23:15:23.410690 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.410623    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d58k6\" (UniqueName: \"kubernetes.io/projected/646ea6a5-9067-42ac-aa64-7863ccb3f553-kube-api-access-d58k6\") pod \"646ea6a5-9067-42ac-aa64-7863ccb3f553\" (UID: \"646ea6a5-9067-42ac-aa64-7863ccb3f553\") "
Jul 16 23:15:23.410898 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.410874    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646ea6a5-9067-42ac-aa64-7863ccb3f553-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "646ea6a5-9067-42ac-aa64-7863ccb3f553" (UID: "646ea6a5-9067-42ac-aa64-7863ccb3f553"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:15:23.412761 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.412727    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646ea6a5-9067-42ac-aa64-7863ccb3f553-kube-api-access-d58k6" (OuterVolumeSpecName: "kube-api-access-d58k6") pod "646ea6a5-9067-42ac-aa64-7863ccb3f553" (UID: "646ea6a5-9067-42ac-aa64-7863ccb3f553"). InnerVolumeSpecName "kube-api-access-d58k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:15:23.412879 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.412795    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/646ea6a5-9067-42ac-aa64-7863ccb3f553-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "646ea6a5-9067-42ac-aa64-7863ccb3f553" (UID: "646ea6a5-9067-42ac-aa64-7863ccb3f553"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:15:23.512303 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.512196    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/646ea6a5-9067-42ac-aa64-7863ccb3f553-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:15:23.512303 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.512228    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/646ea6a5-9067-42ac-aa64-7863ccb3f553-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:15:23.512303 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.512242    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d58k6\" (UniqueName: \"kubernetes.io/projected/646ea6a5-9067-42ac-aa64-7863ccb3f553-kube-api-access-d58k6\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:15:23.574405 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.574370    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"
Jul 16 23:15:23.574618 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.574403    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r" event={"ID":"646ea6a5-9067-42ac-aa64-7863ccb3f553","Type":"ContainerDied","Data":"21085e55428c5c1caebf2d88b8be1fbfc83a7c46a55a1882cde149dd90db4e9c"}
Jul 16 23:15:23.574618 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.574479    2565 scope.go:117] "RemoveContainer" containerID="02a139dfea6f807d2c1b9cc29c6f378e0e92b0548d4a21917f5dcba1a18f9c60"
Jul 16 23:15:23.576157 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.576127    2565 generic.go:356] "Generic (PLEG): container finished" podID="2f8f6578-4632-459d-a15b-22e8d7c9ee5a" containerID="c5cf185646bf94d1e14a94902338cd73b65752ce99a5bd4aac3dbeed35496884" exitCode=0
Jul 16 23:15:23.576300 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.576173    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg" event={"ID":"2f8f6578-4632-459d-a15b-22e8d7c9ee5a","Type":"ContainerDied","Data":"c5cf185646bf94d1e14a94902338cd73b65752ce99a5bd4aac3dbeed35496884"}
Jul 16 23:15:23.576300 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.576176    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"
Jul 16 23:15:23.576300 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.576207    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg" event={"ID":"2f8f6578-4632-459d-a15b-22e8d7c9ee5a","Type":"ContainerDied","Data":"b3b3e5e5db43d5bb40e166f66369634be9088a41646729506f5f0ec41bd3c4d5"}
Jul 16 23:15:23.585631 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.585607    2565 scope.go:117] "RemoveContainer" containerID="c5cf185646bf94d1e14a94902338cd73b65752ce99a5bd4aac3dbeed35496884"
Jul 16 23:15:23.598309 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.598284    2565 scope.go:117] "RemoveContainer" containerID="3e591402620c66405eecc9fcf8025535a3e415ad5c2edadd4f29bf62c38d0b96"
Jul 16 23:15:23.606109 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.606088    2565 scope.go:117] "RemoveContainer" containerID="c5cf185646bf94d1e14a94902338cd73b65752ce99a5bd4aac3dbeed35496884"
Jul 16 23:15:23.606387 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:15:23.606369    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5cf185646bf94d1e14a94902338cd73b65752ce99a5bd4aac3dbeed35496884\": container with ID starting with c5cf185646bf94d1e14a94902338cd73b65752ce99a5bd4aac3dbeed35496884 not found: ID does not exist" containerID="c5cf185646bf94d1e14a94902338cd73b65752ce99a5bd4aac3dbeed35496884"
Jul 16 23:15:23.606453 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.606396    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5cf185646bf94d1e14a94902338cd73b65752ce99a5bd4aac3dbeed35496884"} err="failed to get container status \"c5cf185646bf94d1e14a94902338cd73b65752ce99a5bd4aac3dbeed35496884\": rpc error: code = NotFound desc = could not find container \"c5cf185646bf94d1e14a94902338cd73b65752ce99a5bd4aac3dbeed35496884\": container with ID starting with c5cf185646bf94d1e14a94902338cd73b65752ce99a5bd4aac3dbeed35496884 not found: ID does not exist"
Jul 16 23:15:23.606453 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.606411    2565 scope.go:117] "RemoveContainer" containerID="3e591402620c66405eecc9fcf8025535a3e415ad5c2edadd4f29bf62c38d0b96"
Jul 16 23:15:23.606671 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:15:23.606652    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e591402620c66405eecc9fcf8025535a3e415ad5c2edadd4f29bf62c38d0b96\": container with ID starting with 3e591402620c66405eecc9fcf8025535a3e415ad5c2edadd4f29bf62c38d0b96 not found: ID does not exist" containerID="3e591402620c66405eecc9fcf8025535a3e415ad5c2edadd4f29bf62c38d0b96"
Jul 16 23:15:23.606734 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.606677    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e591402620c66405eecc9fcf8025535a3e415ad5c2edadd4f29bf62c38d0b96"} err="failed to get container status \"3e591402620c66405eecc9fcf8025535a3e415ad5c2edadd4f29bf62c38d0b96\": rpc error: code = NotFound desc = could not find container \"3e591402620c66405eecc9fcf8025535a3e415ad5c2edadd4f29bf62c38d0b96\": container with ID starting with 3e591402620c66405eecc9fcf8025535a3e415ad5c2edadd4f29bf62c38d0b96 not found: ID does not exist"
Jul 16 23:15:23.623860 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.623804    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"]
Jul 16 23:15:23.626891 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.626859    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f263f2f2-openshift-default-b64dbcdf5-wzvqg"]
Jul 16 23:15:23.654662 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.654621    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"]
Jul 16 23:15:23.660643 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.660611    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f263f2f2-5d67cfc7f5-w8d6r"]
Jul 16 23:15:23.725940 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.725907    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f8f6578-4632-459d-a15b-22e8d7c9ee5a" path="/var/lib/kubelet/pods/2f8f6578-4632-459d-a15b-22e8d7c9ee5a/volumes"
Jul 16 23:15:23.726328 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:23.726313    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" path="/var/lib/kubelet/pods/646ea6a5-9067-42ac-aa64-7863ccb3f553/volumes"
Jul 16 23:15:31.656741 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.656706    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c"]
Jul 16 23:15:31.657220 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.657065    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f8f6578-4632-459d-a15b-22e8d7c9ee5a" containerName="istio-proxy"
Jul 16 23:15:31.657220 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.657076    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f8f6578-4632-459d-a15b-22e8d7c9ee5a" containerName="istio-proxy"
Jul 16 23:15:31.657220 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.657089    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerName="maas-api"
Jul 16 23:15:31.657220 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.657095    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerName="maas-api"
Jul 16 23:15:31.657220 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.657101    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerName="maas-api"
Jul 16 23:15:31.657220 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.657107    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerName="maas-api"
Jul 16 23:15:31.657220 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.657125    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerName="maas-api"
Jul 16 23:15:31.657220 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.657130    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerName="maas-api"
Jul 16 23:15:31.657220 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.657135    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerName="maas-api"
Jul 16 23:15:31.657220 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.657140    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerName="maas-api"
Jul 16 23:15:31.657220 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.657148    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerName="maas-api"
Jul 16 23:15:31.657220 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.657153    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerName="maas-api"
Jul 16 23:15:31.657220 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.657201    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f8f6578-4632-459d-a15b-22e8d7c9ee5a" containerName="istio-proxy"
Jul 16 23:15:31.657220 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.657209    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerName="maas-api"
Jul 16 23:15:31.657220 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.657214    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerName="maas-api"
Jul 16 23:15:31.657220 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.657220    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f8f6578-4632-459d-a15b-22e8d7c9ee5a" containerName="istio-proxy"
Jul 16 23:15:31.657220 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.657225    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerName="maas-api"
Jul 16 23:15:31.657769 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.657232    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerName="maas-api"
Jul 16 23:15:31.661630 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.661610    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c"
Jul 16 23:15:31.664020 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.663999    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:15:31.667513 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.667489    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c"]
Jul 16 23:15:31.784719 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.784667    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cee9860-4245-41f5-bfb3-7540c6abd15d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c\" (UID: \"4cee9860-4245-41f5-bfb3-7540c6abd15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c"
Jul 16 23:15:31.886150 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.886108    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cee9860-4245-41f5-bfb3-7540c6abd15d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c\" (UID: \"4cee9860-4245-41f5-bfb3-7540c6abd15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c"
Jul 16 23:15:31.886790 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.886768    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cee9860-4245-41f5-bfb3-7540c6abd15d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c\" (UID: \"4cee9860-4245-41f5-bfb3-7540c6abd15d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c"
Jul 16 23:15:31.974523 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:31.974434    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c"
Jul 16 23:15:32.313109 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:32.313024    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c"]
Jul 16 23:15:32.316232 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:15:32.316204    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cee9860_4245_41f5_bfb3_7540c6abd15d.slice/crio-516f224401fb2a61a5cff4bc0bd3ca62153bf6d1d00f3eb377fb95192a5ca7d2 WatchSource:0}: Error finding container 516f224401fb2a61a5cff4bc0bd3ca62153bf6d1d00f3eb377fb95192a5ca7d2: Status 404 returned error can't find the container with id 516f224401fb2a61a5cff4bc0bd3ca62153bf6d1d00f3eb377fb95192a5ca7d2
Jul 16 23:15:32.620063 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:32.620028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c" event={"ID":"4cee9860-4245-41f5-bfb3-7540c6abd15d","Type":"ContainerStarted","Data":"516f224401fb2a61a5cff4bc0bd3ca62153bf6d1d00f3eb377fb95192a5ca7d2"}
Jul 16 23:15:33.626164 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:33.626129    2565 generic.go:356] "Generic (PLEG): container finished" podID="4cee9860-4245-41f5-bfb3-7540c6abd15d" containerID="b19a83abd3af9982094300fbbfa34d7df3bca9482e8bbcf59566ca11e1ed6e22" exitCode=6
Jul 16 23:15:33.626636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:33.626213    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c" event={"ID":"4cee9860-4245-41f5-bfb3-7540c6abd15d","Type":"ContainerDied","Data":"b19a83abd3af9982094300fbbfa34d7df3bca9482e8bbcf59566ca11e1ed6e22"}
Jul 16 23:15:33.626636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:33.626367    2565 scope.go:117] "RemoveContainer" containerID="b19a83abd3af9982094300fbbfa34d7df3bca9482e8bbcf59566ca11e1ed6e22"
Jul 16 23:15:34.633392 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:34.633354    2565 generic.go:356] "Generic (PLEG): container finished" podID="4cee9860-4245-41f5-bfb3-7540c6abd15d" containerID="171ecbe476f77dcd1c2c39f35a2f53111fe3d0278ffcaf8a948d30b988bf8d3c" exitCode=6
Jul 16 23:15:34.633814 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:34.633434    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c" event={"ID":"4cee9860-4245-41f5-bfb3-7540c6abd15d","Type":"ContainerDied","Data":"171ecbe476f77dcd1c2c39f35a2f53111fe3d0278ffcaf8a948d30b988bf8d3c"}
Jul 16 23:15:34.633814 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:34.633489    2565 scope.go:117] "RemoveContainer" containerID="b19a83abd3af9982094300fbbfa34d7df3bca9482e8bbcf59566ca11e1ed6e22"
Jul 16 23:15:34.633814 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:34.633705    2565 scope.go:117] "RemoveContainer" containerID="171ecbe476f77dcd1c2c39f35a2f53111fe3d0278ffcaf8a948d30b988bf8d3c"
Jul 16 23:15:34.633969 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:15:34.633947    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c_odh-ai-gateway-infra(4cee9860-4245-41f5-bfb3-7540c6abd15d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c" podUID="4cee9860-4245-41f5-bfb3-7540c6abd15d"
Jul 16 23:15:35.640077 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:35.640049    2565 scope.go:117] "RemoveContainer" containerID="171ecbe476f77dcd1c2c39f35a2f53111fe3d0278ffcaf8a948d30b988bf8d3c"
Jul 16 23:15:35.640577 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:15:35.640320    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c_odh-ai-gateway-infra(4cee9860-4245-41f5-bfb3-7540c6abd15d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c" podUID="4cee9860-4245-41f5-bfb3-7540c6abd15d"
Jul 16 23:15:46.720323 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:46.720289    2565 scope.go:117] "RemoveContainer" containerID="171ecbe476f77dcd1c2c39f35a2f53111fe3d0278ffcaf8a948d30b988bf8d3c"
Jul 16 23:15:47.698031 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:47.697930    2565 generic.go:356] "Generic (PLEG): container finished" podID="4cee9860-4245-41f5-bfb3-7540c6abd15d" containerID="a143bd73237e1618c758b39dcac9e977c015e3ed8680f2df0fea6e71b357da57" exitCode=6
Jul 16 23:15:47.698031 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:47.698011    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c" event={"ID":"4cee9860-4245-41f5-bfb3-7540c6abd15d","Type":"ContainerDied","Data":"a143bd73237e1618c758b39dcac9e977c015e3ed8680f2df0fea6e71b357da57"}
Jul 16 23:15:47.698300 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:47.698081    2565 scope.go:117] "RemoveContainer" containerID="171ecbe476f77dcd1c2c39f35a2f53111fe3d0278ffcaf8a948d30b988bf8d3c"
Jul 16 23:15:47.698417 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:47.698403    2565 scope.go:117] "RemoveContainer" containerID="a143bd73237e1618c758b39dcac9e977c015e3ed8680f2df0fea6e71b357da57"
Jul 16 23:15:47.698647 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:15:47.698625    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c_odh-ai-gateway-infra(4cee9860-4245-41f5-bfb3-7540c6abd15d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c" podUID="4cee9860-4245-41f5-bfb3-7540c6abd15d"
Jul 16 23:15:48.723846 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:48.723816    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c"]
Jul 16 23:15:48.861135 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:48.861111    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c"
Jul 16 23:15:48.945348 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:48.945318    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cee9860-4245-41f5-bfb3-7540c6abd15d-maas-api-service-ca\") pod \"4cee9860-4245-41f5-bfb3-7540c6abd15d\" (UID: \"4cee9860-4245-41f5-bfb3-7540c6abd15d\") "
Jul 16 23:15:48.945704 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:48.945682    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cee9860-4245-41f5-bfb3-7540c6abd15d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4cee9860-4245-41f5-bfb3-7540c6abd15d" (UID: "4cee9860-4245-41f5-bfb3-7540c6abd15d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:15:49.046116 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:49.046042    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cee9860-4245-41f5-bfb3-7540c6abd15d-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:15:49.710434 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:49.710397    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c"
Jul 16 23:15:49.710612 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:49.710431    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c" event={"ID":"4cee9860-4245-41f5-bfb3-7540c6abd15d","Type":"ContainerDied","Data":"516f224401fb2a61a5cff4bc0bd3ca62153bf6d1d00f3eb377fb95192a5ca7d2"}
Jul 16 23:15:49.710612 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:49.710506    2565 scope.go:117] "RemoveContainer" containerID="a143bd73237e1618c758b39dcac9e977c015e3ed8680f2df0fea6e71b357da57"
Jul 16 23:15:49.730872 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:49.730848    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c"]
Jul 16 23:15:49.742999 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:49.742976    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-4l96c"]
Jul 16 23:15:50.634600 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.634565    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm"]
Jul 16 23:15:50.634934 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.634922    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerName="maas-api"
Jul 16 23:15:50.634983 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.634936    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerName="maas-api"
Jul 16 23:15:50.634983 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.634951    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f8f6578-4632-459d-a15b-22e8d7c9ee5a" containerName="istio-proxy"
Jul 16 23:15:50.634983 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.634956    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f8f6578-4632-459d-a15b-22e8d7c9ee5a" containerName="istio-proxy"
Jul 16 23:15:50.634983 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.634963    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cee9860-4245-41f5-bfb3-7540c6abd15d" containerName="revoke-keys"
Jul 16 23:15:50.634983 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.634969    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cee9860-4245-41f5-bfb3-7540c6abd15d" containerName="revoke-keys"
Jul 16 23:15:50.634983 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.634979    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cee9860-4245-41f5-bfb3-7540c6abd15d" containerName="revoke-keys"
Jul 16 23:15:50.634983 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.634985    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cee9860-4245-41f5-bfb3-7540c6abd15d" containerName="revoke-keys"
Jul 16 23:15:50.635192 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.635045    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerName="maas-api"
Jul 16 23:15:50.635192 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.635051    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cee9860-4245-41f5-bfb3-7540c6abd15d" containerName="revoke-keys"
Jul 16 23:15:50.635192 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.635059    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cee9860-4245-41f5-bfb3-7540c6abd15d" containerName="revoke-keys"
Jul 16 23:15:50.635192 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.635065    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cee9860-4245-41f5-bfb3-7540c6abd15d" containerName="revoke-keys"
Jul 16 23:15:50.635192 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.635070    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="646ea6a5-9067-42ac-aa64-7863ccb3f553" containerName="maas-api"
Jul 16 23:15:50.639118 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.639100    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm"
Jul 16 23:15:50.641444 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.641418    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:15:50.645728 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.645700    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm"]
Jul 16 23:15:50.761110 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.761069    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc6645d-4d5e-4728-9616-2b8b43670a4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm\" (UID: \"6dc6645d-4d5e-4728-9616-2b8b43670a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm"
Jul 16 23:15:50.862524 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.862479    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc6645d-4d5e-4728-9616-2b8b43670a4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm\" (UID: \"6dc6645d-4d5e-4728-9616-2b8b43670a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm"
Jul 16 23:15:50.863108 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.863088    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc6645d-4d5e-4728-9616-2b8b43670a4b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm\" (UID: \"6dc6645d-4d5e-4728-9616-2b8b43670a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm"
Jul 16 23:15:50.950575 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:50.950490    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm"
Jul 16 23:15:51.080786 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:51.080765    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm"]
Jul 16 23:15:51.083338 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:15:51.083301    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc6645d_4d5e_4728_9616_2b8b43670a4b.slice/crio-89566a53e548b9fb8a78712394e18e5f5e00ffb1bf517163d55b2ce0c67ec7fd WatchSource:0}: Error finding container 89566a53e548b9fb8a78712394e18e5f5e00ffb1bf517163d55b2ce0c67ec7fd: Status 404 returned error can't find the container with id 89566a53e548b9fb8a78712394e18e5f5e00ffb1bf517163d55b2ce0c67ec7fd
Jul 16 23:15:51.725895 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:51.725863    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cee9860-4245-41f5-bfb3-7540c6abd15d" path="/var/lib/kubelet/pods/4cee9860-4245-41f5-bfb3-7540c6abd15d/volumes"
Jul 16 23:15:51.726142 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:51.726125    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm" event={"ID":"6dc6645d-4d5e-4728-9616-2b8b43670a4b","Type":"ContainerStarted","Data":"0fef82ca68a0436cce96041e29f888d08c89aaee8d8f5195498676c67b478755"}
Jul 16 23:15:51.726232 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:51.726165    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm" event={"ID":"6dc6645d-4d5e-4728-9616-2b8b43670a4b","Type":"ContainerStarted","Data":"89566a53e548b9fb8a78712394e18e5f5e00ffb1bf517163d55b2ce0c67ec7fd"}
Jul 16 23:15:51.736278 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:51.736217    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm" podStartSLOduration=1.7362007720000001 podStartE2EDuration="1.736200772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:15:51.73543341 +0000 UTC m=+2562.483917480" watchObservedRunningTime="2026-07-16 23:15:51.736200772 +0000 UTC m=+2562.484684843"
Jul 16 23:15:52.727483 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:52.727445    2565 generic.go:356] "Generic (PLEG): container finished" podID="6dc6645d-4d5e-4728-9616-2b8b43670a4b" containerID="0fef82ca68a0436cce96041e29f888d08c89aaee8d8f5195498676c67b478755" exitCode=6
Jul 16 23:15:52.727949 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:52.727498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm" event={"ID":"6dc6645d-4d5e-4728-9616-2b8b43670a4b","Type":"ContainerDied","Data":"0fef82ca68a0436cce96041e29f888d08c89aaee8d8f5195498676c67b478755"}
Jul 16 23:15:52.727949 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:52.727711    2565 scope.go:117] "RemoveContainer" containerID="0fef82ca68a0436cce96041e29f888d08c89aaee8d8f5195498676c67b478755"
Jul 16 23:15:53.733833 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:53.733730    2565 generic.go:356] "Generic (PLEG): container finished" podID="6dc6645d-4d5e-4728-9616-2b8b43670a4b" containerID="f87fed75fcd062954528943d9a1af5a5aece8c82ff6a54b53866792a3db35db7" exitCode=6
Jul 16 23:15:53.733833 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:53.733815    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm" event={"ID":"6dc6645d-4d5e-4728-9616-2b8b43670a4b","Type":"ContainerDied","Data":"f87fed75fcd062954528943d9a1af5a5aece8c82ff6a54b53866792a3db35db7"}
Jul 16 23:15:53.734351 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:53.733873    2565 scope.go:117] "RemoveContainer" containerID="0fef82ca68a0436cce96041e29f888d08c89aaee8d8f5195498676c67b478755"
Jul 16 23:15:53.734351 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:53.734111    2565 scope.go:117] "RemoveContainer" containerID="f87fed75fcd062954528943d9a1af5a5aece8c82ff6a54b53866792a3db35db7"
Jul 16 23:15:53.734437 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:15:53.734361    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm_odh-ai-gateway-infra(6dc6645d-4d5e-4728-9616-2b8b43670a4b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm" podUID="6dc6645d-4d5e-4728-9616-2b8b43670a4b"
Jul 16 23:15:54.740618 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:54.740589    2565 scope.go:117] "RemoveContainer" containerID="f87fed75fcd062954528943d9a1af5a5aece8c82ff6a54b53866792a3db35db7"
Jul 16 23:15:54.741007 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:15:54.740778    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm_odh-ai-gateway-infra(6dc6645d-4d5e-4728-9616-2b8b43670a4b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm" podUID="6dc6645d-4d5e-4728-9616-2b8b43670a4b"
Jul 16 23:15:57.821344 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:57.821309    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b"]
Jul 16 23:15:57.821729 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:57.821705    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cee9860-4245-41f5-bfb3-7540c6abd15d" containerName="revoke-keys"
Jul 16 23:15:57.821729 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:57.821717    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cee9860-4245-41f5-bfb3-7540c6abd15d" containerName="revoke-keys"
Jul 16 23:15:57.826173 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:57.826149    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b"
Jul 16 23:15:57.832264 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:57.832224    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b"]
Jul 16 23:15:57.927420 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:57.927385    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b7b692-7082-46c7-829c-033281abeec4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b\" (UID: \"42b7b692-7082-46c7-829c-033281abeec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b"
Jul 16 23:15:58.028064 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:58.028025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b7b692-7082-46c7-829c-033281abeec4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b\" (UID: \"42b7b692-7082-46c7-829c-033281abeec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b"
Jul 16 23:15:58.028687 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:58.028665    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b7b692-7082-46c7-829c-033281abeec4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b\" (UID: \"42b7b692-7082-46c7-829c-033281abeec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b"
Jul 16 23:15:58.138016 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:58.137982    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b"
Jul 16 23:15:58.271318 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:58.271286    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b"]
Jul 16 23:15:58.274793 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:15:58.274767    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b7b692_7082_46c7_829c_033281abeec4.slice/crio-243eb449e6adc3e9a14647023fadadc16c274eb9ae8721a990b0e1efcf307c3d WatchSource:0}: Error finding container 243eb449e6adc3e9a14647023fadadc16c274eb9ae8721a990b0e1efcf307c3d: Status 404 returned error can't find the container with id 243eb449e6adc3e9a14647023fadadc16c274eb9ae8721a990b0e1efcf307c3d
Jul 16 23:15:58.760237 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:58.760141    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b" event={"ID":"42b7b692-7082-46c7-829c-033281abeec4","Type":"ContainerStarted","Data":"78b22121b3466eb23f5b5c55ea1986bb3fda4335872785282237941bb2b18a44"}
Jul 16 23:15:58.760237 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:58.760194    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b" event={"ID":"42b7b692-7082-46c7-829c-033281abeec4","Type":"ContainerStarted","Data":"243eb449e6adc3e9a14647023fadadc16c274eb9ae8721a990b0e1efcf307c3d"}
Jul 16 23:15:58.776409 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:58.776362    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b" podStartSLOduration=1.776347478 podStartE2EDuration="1.776347478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:15:58.775050093 +0000 UTC m=+2569.523534164" watchObservedRunningTime="2026-07-16 23:15:58.776347478 +0000 UTC m=+2569.524831527"
Jul 16 23:15:59.765704 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:59.765671    2565 generic.go:356] "Generic (PLEG): container finished" podID="42b7b692-7082-46c7-829c-033281abeec4" containerID="78b22121b3466eb23f5b5c55ea1986bb3fda4335872785282237941bb2b18a44" exitCode=6
Jul 16 23:15:59.766091 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:59.765764    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b" event={"ID":"42b7b692-7082-46c7-829c-033281abeec4","Type":"ContainerDied","Data":"78b22121b3466eb23f5b5c55ea1986bb3fda4335872785282237941bb2b18a44"}
Jul 16 23:15:59.766091 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:15:59.765909    2565 scope.go:117] "RemoveContainer" containerID="78b22121b3466eb23f5b5c55ea1986bb3fda4335872785282237941bb2b18a44"
Jul 16 23:16:00.772364 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:00.772235    2565 generic.go:356] "Generic (PLEG): container finished" podID="42b7b692-7082-46c7-829c-033281abeec4" containerID="5c384609205a2021649b9f284f158cacdfebd8fc714769d29177acf48172e2fa" exitCode=6
Jul 16 23:16:00.772364 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:00.772323    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b" event={"ID":"42b7b692-7082-46c7-829c-033281abeec4","Type":"ContainerDied","Data":"5c384609205a2021649b9f284f158cacdfebd8fc714769d29177acf48172e2fa"}
Jul 16 23:16:00.772836 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:00.772402    2565 scope.go:117] "RemoveContainer" containerID="78b22121b3466eb23f5b5c55ea1986bb3fda4335872785282237941bb2b18a44"
Jul 16 23:16:00.772836 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:00.772642    2565 scope.go:117] "RemoveContainer" containerID="5c384609205a2021649b9f284f158cacdfebd8fc714769d29177acf48172e2fa"
Jul 16 23:16:00.772940 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:16:00.772862    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b_odh-ai-gateway-infra(42b7b692-7082-46c7-829c-033281abeec4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b" podUID="42b7b692-7082-46c7-829c-033281abeec4"
Jul 16 23:16:01.778967 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:01.778937    2565 scope.go:117] "RemoveContainer" containerID="5c384609205a2021649b9f284f158cacdfebd8fc714769d29177acf48172e2fa"
Jul 16 23:16:01.779365 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:16:01.779119    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b_odh-ai-gateway-infra(42b7b692-7082-46c7-829c-033281abeec4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b" podUID="42b7b692-7082-46c7-829c-033281abeec4"
Jul 16 23:16:02.869450 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:02.869415    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d"]
Jul 16 23:16:02.874148 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:02.874127    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d"
Jul 16 23:16:02.879129 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:02.879103    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d"]
Jul 16 23:16:02.974863 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:02.974827    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0fb8e1-9ba3-4921-aba0-56d233c52b93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-q2f6d\" (UID: \"be0fb8e1-9ba3-4921-aba0-56d233c52b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d"
Jul 16 23:16:03.076143 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:03.076112    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0fb8e1-9ba3-4921-aba0-56d233c52b93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-q2f6d\" (UID: \"be0fb8e1-9ba3-4921-aba0-56d233c52b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d"
Jul 16 23:16:03.076763 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:03.076744    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0fb8e1-9ba3-4921-aba0-56d233c52b93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-q2f6d\" (UID: \"be0fb8e1-9ba3-4921-aba0-56d233c52b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d"
Jul 16 23:16:03.188537 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:03.188462    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d"
Jul 16 23:16:03.316632 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:03.316603    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d"]
Jul 16 23:16:03.318399 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:16:03.318375    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0fb8e1_9ba3_4921_aba0_56d233c52b93.slice/crio-96e240a05cf42d9c41609bf2b723631a66e8a5878a5bbd3ad18fe8b2e22c7c5e WatchSource:0}: Error finding container 96e240a05cf42d9c41609bf2b723631a66e8a5878a5bbd3ad18fe8b2e22c7c5e: Status 404 returned error can't find the container with id 96e240a05cf42d9c41609bf2b723631a66e8a5878a5bbd3ad18fe8b2e22c7c5e
Jul 16 23:16:03.789938 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:03.789895    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d" event={"ID":"be0fb8e1-9ba3-4921-aba0-56d233c52b93","Type":"ContainerStarted","Data":"b13945eaed82025edc2294dc01ef6bbca7f9f92b4572a8491099b1d1820bf5ca"}
Jul 16 23:16:03.790106 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:03.789962    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d" event={"ID":"be0fb8e1-9ba3-4921-aba0-56d233c52b93","Type":"ContainerStarted","Data":"96e240a05cf42d9c41609bf2b723631a66e8a5878a5bbd3ad18fe8b2e22c7c5e"}
Jul 16 23:16:03.807128 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:03.807037    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d" podStartSLOduration=1.807019999 podStartE2EDuration="1.807019999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:16:03.804701643 +0000 UTC m=+2574.553185713" watchObservedRunningTime="2026-07-16 23:16:03.807019999 +0000 UTC m=+2574.555504070"
Jul 16 23:16:04.802141 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:04.802107    2565 generic.go:356] "Generic (PLEG): container finished" podID="be0fb8e1-9ba3-4921-aba0-56d233c52b93" containerID="b13945eaed82025edc2294dc01ef6bbca7f9f92b4572a8491099b1d1820bf5ca" exitCode=6
Jul 16 23:16:04.802574 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:04.802185    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d" event={"ID":"be0fb8e1-9ba3-4921-aba0-56d233c52b93","Type":"ContainerDied","Data":"b13945eaed82025edc2294dc01ef6bbca7f9f92b4572a8491099b1d1820bf5ca"}
Jul 16 23:16:04.802574 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:04.802408    2565 scope.go:117] "RemoveContainer" containerID="b13945eaed82025edc2294dc01ef6bbca7f9f92b4572a8491099b1d1820bf5ca"
Jul 16 23:16:05.810283 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:05.810197    2565 generic.go:356] "Generic (PLEG): container finished" podID="be0fb8e1-9ba3-4921-aba0-56d233c52b93" containerID="e1b028b8b779c8ce21c925fc987185b8c38ca0d07ee6716150edc53c9c7a91bb" exitCode=6
Jul 16 23:16:05.810673 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:05.810279    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d" event={"ID":"be0fb8e1-9ba3-4921-aba0-56d233c52b93","Type":"ContainerDied","Data":"e1b028b8b779c8ce21c925fc987185b8c38ca0d07ee6716150edc53c9c7a91bb"}
Jul 16 23:16:05.810673 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:05.810333    2565 scope.go:117] "RemoveContainer" containerID="b13945eaed82025edc2294dc01ef6bbca7f9f92b4572a8491099b1d1820bf5ca"
Jul 16 23:16:05.810673 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:05.810580    2565 scope.go:117] "RemoveContainer" containerID="e1b028b8b779c8ce21c925fc987185b8c38ca0d07ee6716150edc53c9c7a91bb"
Jul 16 23:16:05.810784 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:16:05.810765    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-q2f6d_odh-ai-gateway-infra(be0fb8e1-9ba3-4921-aba0-56d233c52b93)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d" podUID="be0fb8e1-9ba3-4921-aba0-56d233c52b93"
Jul 16 23:16:06.816313 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:06.816287    2565 scope.go:117] "RemoveContainer" containerID="e1b028b8b779c8ce21c925fc987185b8c38ca0d07ee6716150edc53c9c7a91bb"
Jul 16 23:16:06.816793 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:16:06.816549    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-q2f6d_odh-ai-gateway-infra(be0fb8e1-9ba3-4921-aba0-56d233c52b93)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d" podUID="be0fb8e1-9ba3-4921-aba0-56d233c52b93"
Jul 16 23:16:07.726907 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:07.726877    2565 scope.go:117] "RemoveContainer" containerID="f87fed75fcd062954528943d9a1af5a5aece8c82ff6a54b53866792a3db35db7"
Jul 16 23:16:08.827955 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:08.827920    2565 generic.go:356] "Generic (PLEG): container finished" podID="6dc6645d-4d5e-4728-9616-2b8b43670a4b" containerID="144a66ae0508b5c19e2296b94e459172179de3e93ac2497b3e7b9ae26e2c8201" exitCode=6
Jul 16 23:16:08.828446 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:08.827994    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm" event={"ID":"6dc6645d-4d5e-4728-9616-2b8b43670a4b","Type":"ContainerDied","Data":"144a66ae0508b5c19e2296b94e459172179de3e93ac2497b3e7b9ae26e2c8201"}
Jul 16 23:16:08.828446 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:08.828058    2565 scope.go:117] "RemoveContainer" containerID="f87fed75fcd062954528943d9a1af5a5aece8c82ff6a54b53866792a3db35db7"
Jul 16 23:16:08.828446 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:08.828406    2565 scope.go:117] "RemoveContainer" containerID="144a66ae0508b5c19e2296b94e459172179de3e93ac2497b3e7b9ae26e2c8201"
Jul 16 23:16:08.828640 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:16:08.828607    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm_odh-ai-gateway-infra(6dc6645d-4d5e-4728-9616-2b8b43670a4b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm" podUID="6dc6645d-4d5e-4728-9616-2b8b43670a4b"
Jul 16 23:16:09.852812 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:09.852780    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm"]
Jul 16 23:16:09.993367 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:09.993345    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm"
Jul 16 23:16:10.039390 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:10.039357    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc6645d-4d5e-4728-9616-2b8b43670a4b-maas-api-service-ca\") pod \"6dc6645d-4d5e-4728-9616-2b8b43670a4b\" (UID: \"6dc6645d-4d5e-4728-9616-2b8b43670a4b\") "
Jul 16 23:16:10.039762 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:10.039740    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc6645d-4d5e-4728-9616-2b8b43670a4b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6dc6645d-4d5e-4728-9616-2b8b43670a4b" (UID: "6dc6645d-4d5e-4728-9616-2b8b43670a4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:16:10.140728 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:10.140644    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc6645d-4d5e-4728-9616-2b8b43670a4b-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:16:10.840960 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:10.840925    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm" event={"ID":"6dc6645d-4d5e-4728-9616-2b8b43670a4b","Type":"ContainerDied","Data":"89566a53e548b9fb8a78712394e18e5f5e00ffb1bf517163d55b2ce0c67ec7fd"}
Jul 16 23:16:10.840960 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:10.840957    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm"
Jul 16 23:16:10.841309 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:10.840986    2565 scope.go:117] "RemoveContainer" containerID="144a66ae0508b5c19e2296b94e459172179de3e93ac2497b3e7b9ae26e2c8201"
Jul 16 23:16:10.863830 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:10.863796    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm"]
Jul 16 23:16:10.867847 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:10.867822    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-t5rcm"]
Jul 16 23:16:11.725451 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:11.725419    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dc6645d-4d5e-4728-9616-2b8b43670a4b" path="/var/lib/kubelet/pods/6dc6645d-4d5e-4728-9616-2b8b43670a4b/volumes"
Jul 16 23:16:16.720992 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:16.720957    2565 scope.go:117] "RemoveContainer" containerID="5c384609205a2021649b9f284f158cacdfebd8fc714769d29177acf48172e2fa"
Jul 16 23:16:17.876307 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:17.876272    2565 generic.go:356] "Generic (PLEG): container finished" podID="42b7b692-7082-46c7-829c-033281abeec4" containerID="5a98b0e183af88853cffb2abe2495cdb8e9a452af1c8e15f8c46cf08dbeb9e17" exitCode=6
Jul 16 23:16:17.876759 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:17.876324    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b" event={"ID":"42b7b692-7082-46c7-829c-033281abeec4","Type":"ContainerDied","Data":"5a98b0e183af88853cffb2abe2495cdb8e9a452af1c8e15f8c46cf08dbeb9e17"}
Jul 16 23:16:17.876759 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:17.876379    2565 scope.go:117] "RemoveContainer" containerID="5c384609205a2021649b9f284f158cacdfebd8fc714769d29177acf48172e2fa"
Jul 16 23:16:17.876759 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:17.876700    2565 scope.go:117] "RemoveContainer" containerID="5a98b0e183af88853cffb2abe2495cdb8e9a452af1c8e15f8c46cf08dbeb9e17"
Jul 16 23:16:17.876930 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:16:17.876909    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b_odh-ai-gateway-infra(42b7b692-7082-46c7-829c-033281abeec4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b" podUID="42b7b692-7082-46c7-829c-033281abeec4"
Jul 16 23:16:18.901492 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:18.901451    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b"]
Jul 16 23:16:19.043619 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:19.043585    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b"
Jul 16 23:16:19.124393 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:19.124356    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b7b692-7082-46c7-829c-033281abeec4-maas-api-service-ca\") pod \"42b7b692-7082-46c7-829c-033281abeec4\" (UID: \"42b7b692-7082-46c7-829c-033281abeec4\") "
Jul 16 23:16:19.124723 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:19.124698    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b7b692-7082-46c7-829c-033281abeec4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42b7b692-7082-46c7-829c-033281abeec4" (UID: "42b7b692-7082-46c7-829c-033281abeec4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:16:19.226040 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:19.225955    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b7b692-7082-46c7-829c-033281abeec4-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:16:19.722707 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:19.722674    2565 scope.go:117] "RemoveContainer" containerID="e1b028b8b779c8ce21c925fc987185b8c38ca0d07ee6716150edc53c9c7a91bb"
Jul 16 23:16:19.888631 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:19.888603    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b"
Jul 16 23:16:19.888791 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:19.888648    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b" event={"ID":"42b7b692-7082-46c7-829c-033281abeec4","Type":"ContainerDied","Data":"243eb449e6adc3e9a14647023fadadc16c274eb9ae8721a990b0e1efcf307c3d"}
Jul 16 23:16:19.888791 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:19.888713    2565 scope.go:117] "RemoveContainer" containerID="5a98b0e183af88853cffb2abe2495cdb8e9a452af1c8e15f8c46cf08dbeb9e17"
Jul 16 23:16:19.904827 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:19.904805    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b"]
Jul 16 23:16:19.910024 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:19.910001    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-5vv7b"]
Jul 16 23:16:20.895131 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:20.895095    2565 generic.go:356] "Generic (PLEG): container finished" podID="be0fb8e1-9ba3-4921-aba0-56d233c52b93" containerID="da999103cab0a67e86d6195bfd4a08986a14f919fba54b7453792079d45f24b0" exitCode=6
Jul 16 23:16:20.895364 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:20.895179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d" event={"ID":"be0fb8e1-9ba3-4921-aba0-56d233c52b93","Type":"ContainerDied","Data":"da999103cab0a67e86d6195bfd4a08986a14f919fba54b7453792079d45f24b0"}
Jul 16 23:16:20.895364 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:20.895262    2565 scope.go:117] "RemoveContainer" containerID="e1b028b8b779c8ce21c925fc987185b8c38ca0d07ee6716150edc53c9c7a91bb"
Jul 16 23:16:20.895595 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:20.895571    2565 scope.go:117] "RemoveContainer" containerID="da999103cab0a67e86d6195bfd4a08986a14f919fba54b7453792079d45f24b0"
Jul 16 23:16:20.895866 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:16:20.895841    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-q2f6d_odh-ai-gateway-infra(be0fb8e1-9ba3-4921-aba0-56d233c52b93)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d" podUID="be0fb8e1-9ba3-4921-aba0-56d233c52b93"
Jul 16 23:16:21.699653 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.699612    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2"]
Jul 16 23:16:21.700048 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.699997    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc6645d-4d5e-4728-9616-2b8b43670a4b" containerName="revoke-keys"
Jul 16 23:16:21.700048 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.700008    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc6645d-4d5e-4728-9616-2b8b43670a4b" containerName="revoke-keys"
Jul 16 23:16:21.700048 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.700017    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b7b692-7082-46c7-829c-033281abeec4" containerName="revoke-keys"
Jul 16 23:16:21.700048 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.700022    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b7b692-7082-46c7-829c-033281abeec4" containerName="revoke-keys"
Jul 16 23:16:21.700048 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.700037    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc6645d-4d5e-4728-9616-2b8b43670a4b" containerName="revoke-keys"
Jul 16 23:16:21.700048 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.700042    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc6645d-4d5e-4728-9616-2b8b43670a4b" containerName="revoke-keys"
Jul 16 23:16:21.700292 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.700054    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b7b692-7082-46c7-829c-033281abeec4" containerName="revoke-keys"
Jul 16 23:16:21.700292 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.700060    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b7b692-7082-46c7-829c-033281abeec4" containerName="revoke-keys"
Jul 16 23:16:21.700292 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.700116    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc6645d-4d5e-4728-9616-2b8b43670a4b" containerName="revoke-keys"
Jul 16 23:16:21.700292 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.700123    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc6645d-4d5e-4728-9616-2b8b43670a4b" containerName="revoke-keys"
Jul 16 23:16:21.700292 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.700131    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b7b692-7082-46c7-829c-033281abeec4" containerName="revoke-keys"
Jul 16 23:16:21.700292 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.700136    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc6645d-4d5e-4728-9616-2b8b43670a4b" containerName="revoke-keys"
Jul 16 23:16:21.700292 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.700143    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b7b692-7082-46c7-829c-033281abeec4" containerName="revoke-keys"
Jul 16 23:16:21.700292 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.700149    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b7b692-7082-46c7-829c-033281abeec4" containerName="revoke-keys"
Jul 16 23:16:21.704397 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.704375    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2"
Jul 16 23:16:21.709242 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.709176    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2"]
Jul 16 23:16:21.728208 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.728165    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42b7b692-7082-46c7-829c-033281abeec4" path="/var/lib/kubelet/pods/42b7b692-7082-46c7-829c-033281abeec4/volumes"
Jul 16 23:16:21.853747 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.853700    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb4a646-ade3-4a45-b20a-732a2118d999-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2\" (UID: \"aeb4a646-ade3-4a45-b20a-732a2118d999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2"
Jul 16 23:16:21.918319 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.918284    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d"]
Jul 16 23:16:21.954285 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.954171    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb4a646-ade3-4a45-b20a-732a2118d999-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2\" (UID: \"aeb4a646-ade3-4a45-b20a-732a2118d999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2"
Jul 16 23:16:21.954822 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:21.954803    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb4a646-ade3-4a45-b20a-732a2118d999-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2\" (UID: \"aeb4a646-ade3-4a45-b20a-732a2118d999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2"
Jul 16 23:16:22.017838 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:22.017805    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2"
Jul 16 23:16:22.061055 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:22.061024    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d"
Jul 16 23:16:22.156036 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:22.155994    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0fb8e1-9ba3-4921-aba0-56d233c52b93-maas-api-service-ca\") pod \"be0fb8e1-9ba3-4921-aba0-56d233c52b93\" (UID: \"be0fb8e1-9ba3-4921-aba0-56d233c52b93\") "
Jul 16 23:16:22.156751 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:22.156719    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be0fb8e1-9ba3-4921-aba0-56d233c52b93-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be0fb8e1-9ba3-4921-aba0-56d233c52b93" (UID: "be0fb8e1-9ba3-4921-aba0-56d233c52b93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:16:22.159117 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:22.158899    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2"]
Jul 16 23:16:22.161309 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:16:22.161284    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb4a646_ade3_4a45_b20a_732a2118d999.slice/crio-8264e6bfbc81a65e7a0c92dbc596e65c02a1147c7f5eda5879019746e9bb074b WatchSource:0}: Error finding container 8264e6bfbc81a65e7a0c92dbc596e65c02a1147c7f5eda5879019746e9bb074b: Status 404 returned error can't find the container with id 8264e6bfbc81a65e7a0c92dbc596e65c02a1147c7f5eda5879019746e9bb074b
Jul 16 23:16:22.257920 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:22.257832    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0fb8e1-9ba3-4921-aba0-56d233c52b93-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:16:22.912618 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:22.911564    2565 generic.go:356] "Generic (PLEG): container finished" podID="aeb4a646-ade3-4a45-b20a-732a2118d999" containerID="d541a9d2d22890a9e8a50bc49f17825a44aa5d3bfee9ad51baaf2a7571cb986b" exitCode=6
Jul 16 23:16:22.912618 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:22.911730    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2" event={"ID":"aeb4a646-ade3-4a45-b20a-732a2118d999","Type":"ContainerDied","Data":"d541a9d2d22890a9e8a50bc49f17825a44aa5d3bfee9ad51baaf2a7571cb986b"}
Jul 16 23:16:22.912618 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:22.911780    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2" event={"ID":"aeb4a646-ade3-4a45-b20a-732a2118d999","Type":"ContainerStarted","Data":"8264e6bfbc81a65e7a0c92dbc596e65c02a1147c7f5eda5879019746e9bb074b"}
Jul 16 23:16:22.912618 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:22.912081    2565 scope.go:117] "RemoveContainer" containerID="d541a9d2d22890a9e8a50bc49f17825a44aa5d3bfee9ad51baaf2a7571cb986b"
Jul 16 23:16:22.914557 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:22.914535    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d"
Jul 16 23:16:22.914698 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:22.914530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d" event={"ID":"be0fb8e1-9ba3-4921-aba0-56d233c52b93","Type":"ContainerDied","Data":"96e240a05cf42d9c41609bf2b723631a66e8a5878a5bbd3ad18fe8b2e22c7c5e"}
Jul 16 23:16:22.914698 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:22.914674    2565 scope.go:117] "RemoveContainer" containerID="da999103cab0a67e86d6195bfd4a08986a14f919fba54b7453792079d45f24b0"
Jul 16 23:16:22.938784 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:22.938732    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d"]
Jul 16 23:16:22.942832 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:22.942795    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-q2f6d"]
Jul 16 23:16:23.726453 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:23.726418    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be0fb8e1-9ba3-4921-aba0-56d233c52b93" path="/var/lib/kubelet/pods/be0fb8e1-9ba3-4921-aba0-56d233c52b93/volumes"
Jul 16 23:16:23.922148 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:23.922055    2565 generic.go:356] "Generic (PLEG): container finished" podID="aeb4a646-ade3-4a45-b20a-732a2118d999" containerID="3cfa28a9760f298c57bb1477dffc0d426bac930e7c470a62b4b5268864a0fab5" exitCode=6
Jul 16 23:16:23.922699 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:23.922140    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2" event={"ID":"aeb4a646-ade3-4a45-b20a-732a2118d999","Type":"ContainerDied","Data":"3cfa28a9760f298c57bb1477dffc0d426bac930e7c470a62b4b5268864a0fab5"}
Jul 16 23:16:23.922699 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:23.922211    2565 scope.go:117] "RemoveContainer" containerID="d541a9d2d22890a9e8a50bc49f17825a44aa5d3bfee9ad51baaf2a7571cb986b"
Jul 16 23:16:23.922699 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:23.922467    2565 scope.go:117] "RemoveContainer" containerID="3cfa28a9760f298c57bb1477dffc0d426bac930e7c470a62b4b5268864a0fab5"
Jul 16 23:16:23.922859 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:16:23.922725    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2_odh-ai-gateway-infra(aeb4a646-ade3-4a45-b20a-732a2118d999)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2" podUID="aeb4a646-ade3-4a45-b20a-732a2118d999"
Jul 16 23:16:24.928149 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:24.928125    2565 scope.go:117] "RemoveContainer" containerID="3cfa28a9760f298c57bb1477dffc0d426bac930e7c470a62b4b5268864a0fab5"
Jul 16 23:16:24.928576 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:16:24.928366    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2_odh-ai-gateway-infra(aeb4a646-ade3-4a45-b20a-732a2118d999)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2" podUID="aeb4a646-ade3-4a45-b20a-732a2118d999"
Jul 16 23:16:36.720294 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:36.720193    2565 scope.go:117] "RemoveContainer" containerID="3cfa28a9760f298c57bb1477dffc0d426bac930e7c470a62b4b5268864a0fab5"
Jul 16 23:16:36.985888 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:36.985805    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2" event={"ID":"aeb4a646-ade3-4a45-b20a-732a2118d999","Type":"ContainerStarted","Data":"f05d8c7e5537c6fb131146448d38cc4e9f97fe41c303a165362a1d299ee3c9cf"}
Jul 16 23:16:37.001572 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:37.001515    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2" podStartSLOduration=16.001499829 podStartE2EDuration="16.001499829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:16:37.000199298 +0000 UTC m=+2607.748683399" watchObservedRunningTime="2026-07-16 23:16:37.001499829 +0000 UTC m=+2607.749983911"
Jul 16 23:16:37.991769 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:37.991737    2565 generic.go:356] "Generic (PLEG): container finished" podID="aeb4a646-ade3-4a45-b20a-732a2118d999" containerID="f05d8c7e5537c6fb131146448d38cc4e9f97fe41c303a165362a1d299ee3c9cf" exitCode=6
Jul 16 23:16:37.992201 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:37.991815    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2" event={"ID":"aeb4a646-ade3-4a45-b20a-732a2118d999","Type":"ContainerDied","Data":"f05d8c7e5537c6fb131146448d38cc4e9f97fe41c303a165362a1d299ee3c9cf"}
Jul 16 23:16:37.992201 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:37.991881    2565 scope.go:117] "RemoveContainer" containerID="3cfa28a9760f298c57bb1477dffc0d426bac930e7c470a62b4b5268864a0fab5"
Jul 16 23:16:37.992342 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:37.992211    2565 scope.go:117] "RemoveContainer" containerID="f05d8c7e5537c6fb131146448d38cc4e9f97fe41c303a165362a1d299ee3c9cf"
Jul 16 23:16:37.992488 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:16:37.992462    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2_odh-ai-gateway-infra(aeb4a646-ade3-4a45-b20a-732a2118d999)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2" podUID="aeb4a646-ade3-4a45-b20a-732a2118d999"
Jul 16 23:16:38.013008 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:38.012980    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2"]
Jul 16 23:16:39.140983 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:39.140954    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2"
Jul 16 23:16:39.215762 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:39.215726    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb4a646-ade3-4a45-b20a-732a2118d999-maas-api-service-ca\") pod \"aeb4a646-ade3-4a45-b20a-732a2118d999\" (UID: \"aeb4a646-ade3-4a45-b20a-732a2118d999\") "
Jul 16 23:16:39.216104 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:39.216080    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb4a646-ade3-4a45-b20a-732a2118d999-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aeb4a646-ade3-4a45-b20a-732a2118d999" (UID: "aeb4a646-ade3-4a45-b20a-732a2118d999"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:16:39.316976 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:39.316876    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeb4a646-ade3-4a45-b20a-732a2118d999-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:16:40.004365 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:40.004243    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2"
Jul 16 23:16:40.004365 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:40.004278    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2" event={"ID":"aeb4a646-ade3-4a45-b20a-732a2118d999","Type":"ContainerDied","Data":"8264e6bfbc81a65e7a0c92dbc596e65c02a1147c7f5eda5879019746e9bb074b"}
Jul 16 23:16:40.004365 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:40.004363    2565 scope.go:117] "RemoveContainer" containerID="f05d8c7e5537c6fb131146448d38cc4e9f97fe41c303a165362a1d299ee3c9cf"
Jul 16 23:16:40.019827 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:40.019796    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2"]
Jul 16 23:16:40.025885 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:40.025859    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f9zr2"]
Jul 16 23:16:41.726032 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:41.725988    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeb4a646-ade3-4a45-b20a-732a2118d999" path="/var/lib/kubelet/pods/aeb4a646-ade3-4a45-b20a-732a2118d999/volumes"
Jul 16 23:16:50.675735 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.675694    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4"]
Jul 16 23:16:50.676271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.676038    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be0fb8e1-9ba3-4921-aba0-56d233c52b93" containerName="revoke-keys"
Jul 16 23:16:50.676271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.676049    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="be0fb8e1-9ba3-4921-aba0-56d233c52b93" containerName="revoke-keys"
Jul 16 23:16:50.676271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.676060    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be0fb8e1-9ba3-4921-aba0-56d233c52b93" containerName="revoke-keys"
Jul 16 23:16:50.676271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.676066    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="be0fb8e1-9ba3-4921-aba0-56d233c52b93" containerName="revoke-keys"
Jul 16 23:16:50.676271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.676076    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb4a646-ade3-4a45-b20a-732a2118d999" containerName="revoke-keys"
Jul 16 23:16:50.676271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.676083    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb4a646-ade3-4a45-b20a-732a2118d999" containerName="revoke-keys"
Jul 16 23:16:50.676271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.676091    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc6645d-4d5e-4728-9616-2b8b43670a4b" containerName="revoke-keys"
Jul 16 23:16:50.676271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.676096    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc6645d-4d5e-4728-9616-2b8b43670a4b" containerName="revoke-keys"
Jul 16 23:16:50.676271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.676107    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb4a646-ade3-4a45-b20a-732a2118d999" containerName="revoke-keys"
Jul 16 23:16:50.676271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.676112    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb4a646-ade3-4a45-b20a-732a2118d999" containerName="revoke-keys"
Jul 16 23:16:50.676271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.676119    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb4a646-ade3-4a45-b20a-732a2118d999" containerName="revoke-keys"
Jul 16 23:16:50.676271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.676124    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb4a646-ade3-4a45-b20a-732a2118d999" containerName="revoke-keys"
Jul 16 23:16:50.676271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.676135    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b7b692-7082-46c7-829c-033281abeec4" containerName="revoke-keys"
Jul 16 23:16:50.676271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.676141    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b7b692-7082-46c7-829c-033281abeec4" containerName="revoke-keys"
Jul 16 23:16:50.676271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.676199    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeb4a646-ade3-4a45-b20a-732a2118d999" containerName="revoke-keys"
Jul 16 23:16:50.676271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.676207    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeb4a646-ade3-4a45-b20a-732a2118d999" containerName="revoke-keys"
Jul 16 23:16:50.676271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.676213    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="be0fb8e1-9ba3-4921-aba0-56d233c52b93" containerName="revoke-keys"
Jul 16 23:16:50.676271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.676218    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="be0fb8e1-9ba3-4921-aba0-56d233c52b93" containerName="revoke-keys"
Jul 16 23:16:50.679219 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.679201    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4"
Jul 16 23:16:50.681512 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.681491    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:16:50.690476 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.690454    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4"]
Jul 16 23:16:50.723218 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.723182    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701151f8-d63f-4d88-9b9f-93a8dfce20b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4\" (UID: \"701151f8-d63f-4d88-9b9f-93a8dfce20b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4"
Jul 16 23:16:50.823905 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.823862    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701151f8-d63f-4d88-9b9f-93a8dfce20b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4\" (UID: \"701151f8-d63f-4d88-9b9f-93a8dfce20b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4"
Jul 16 23:16:50.824551 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.824527    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701151f8-d63f-4d88-9b9f-93a8dfce20b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4\" (UID: \"701151f8-d63f-4d88-9b9f-93a8dfce20b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4"
Jul 16 23:16:50.988820 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:50.988724    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4"
Jul 16 23:16:51.136625 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:51.136589    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4"]
Jul 16 23:16:51.140533 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:16:51.140506    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701151f8_d63f_4d88_9b9f_93a8dfce20b7.slice/crio-9d0d24dc2e05fdba89dcb948e664c63a9b725b979cea587622ac821a4ddce8ab WatchSource:0}: Error finding container 9d0d24dc2e05fdba89dcb948e664c63a9b725b979cea587622ac821a4ddce8ab: Status 404 returned error can't find the container with id 9d0d24dc2e05fdba89dcb948e664c63a9b725b979cea587622ac821a4ddce8ab
Jul 16 23:16:52.064655 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:52.064560    2565 generic.go:356] "Generic (PLEG): container finished" podID="701151f8-d63f-4d88-9b9f-93a8dfce20b7" containerID="eb7d5dfb1b1fd2e269b18d16c12ef39438f19b07a269ac7654410121620c05d0" exitCode=6
Jul 16 23:16:52.064655 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:52.064598    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4" event={"ID":"701151f8-d63f-4d88-9b9f-93a8dfce20b7","Type":"ContainerDied","Data":"eb7d5dfb1b1fd2e269b18d16c12ef39438f19b07a269ac7654410121620c05d0"}
Jul 16 23:16:52.064655 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:52.064636    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4" event={"ID":"701151f8-d63f-4d88-9b9f-93a8dfce20b7","Type":"ContainerStarted","Data":"9d0d24dc2e05fdba89dcb948e664c63a9b725b979cea587622ac821a4ddce8ab"}
Jul 16 23:16:52.065095 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:52.064821    2565 scope.go:117] "RemoveContainer" containerID="eb7d5dfb1b1fd2e269b18d16c12ef39438f19b07a269ac7654410121620c05d0"
Jul 16 23:16:53.071763 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:53.071668    2565 generic.go:356] "Generic (PLEG): container finished" podID="701151f8-d63f-4d88-9b9f-93a8dfce20b7" containerID="6abb68e069ff7cc58ce40458f0bd377883788d871987db70f19eb68a439b4241" exitCode=6
Jul 16 23:16:53.071763 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:53.071743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4" event={"ID":"701151f8-d63f-4d88-9b9f-93a8dfce20b7","Type":"ContainerDied","Data":"6abb68e069ff7cc58ce40458f0bd377883788d871987db70f19eb68a439b4241"}
Jul 16 23:16:53.072301 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:53.071803    2565 scope.go:117] "RemoveContainer" containerID="eb7d5dfb1b1fd2e269b18d16c12ef39438f19b07a269ac7654410121620c05d0"
Jul 16 23:16:53.072301 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:53.072006    2565 scope.go:117] "RemoveContainer" containerID="6abb68e069ff7cc58ce40458f0bd377883788d871987db70f19eb68a439b4241"
Jul 16 23:16:53.072301 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:16:53.072281    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4_odh-ai-gateway-infra(701151f8-d63f-4d88-9b9f-93a8dfce20b7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4" podUID="701151f8-d63f-4d88-9b9f-93a8dfce20b7"
Jul 16 23:16:54.078696 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:54.078668    2565 scope.go:117] "RemoveContainer" containerID="6abb68e069ff7cc58ce40458f0bd377883788d871987db70f19eb68a439b4241"
Jul 16 23:16:54.079094 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:16:54.078860    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4_odh-ai-gateway-infra(701151f8-d63f-4d88-9b9f-93a8dfce20b7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4" podUID="701151f8-d63f-4d88-9b9f-93a8dfce20b7"
Jul 16 23:16:57.865361 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:57.865321    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5"]
Jul 16 23:16:57.865836 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:57.865718    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be0fb8e1-9ba3-4921-aba0-56d233c52b93" containerName="revoke-keys"
Jul 16 23:16:57.865836 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:57.865735    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="be0fb8e1-9ba3-4921-aba0-56d233c52b93" containerName="revoke-keys"
Jul 16 23:16:57.865836 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:57.865825    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="be0fb8e1-9ba3-4921-aba0-56d233c52b93" containerName="revoke-keys"
Jul 16 23:16:57.865939 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:57.865839    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeb4a646-ade3-4a45-b20a-732a2118d999" containerName="revoke-keys"
Jul 16 23:16:57.870175 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:57.870155    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5"
Jul 16 23:16:57.877182 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:57.877154    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5"]
Jul 16 23:16:57.995601 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:57.995569    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8163ed74-2aaa-4375-bc08-14e388ef29d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5\" (UID: \"8163ed74-2aaa-4375-bc08-14e388ef29d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5"
Jul 16 23:16:58.096205 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:58.096170    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8163ed74-2aaa-4375-bc08-14e388ef29d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5\" (UID: \"8163ed74-2aaa-4375-bc08-14e388ef29d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5"
Jul 16 23:16:58.096876 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:58.096853    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8163ed74-2aaa-4375-bc08-14e388ef29d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5\" (UID: \"8163ed74-2aaa-4375-bc08-14e388ef29d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5"
Jul 16 23:16:58.181833 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:58.181744    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5"
Jul 16 23:16:58.310816 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:58.310792    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5"]
Jul 16 23:16:58.313341 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:16:58.313313    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8163ed74_2aaa_4375_bc08_14e388ef29d8.slice/crio-4861644cc4478768e9292d0cf993f1385ad52be0bd2abaa8b3a009753ceeed20 WatchSource:0}: Error finding container 4861644cc4478768e9292d0cf993f1385ad52be0bd2abaa8b3a009753ceeed20: Status 404 returned error can't find the container with id 4861644cc4478768e9292d0cf993f1385ad52be0bd2abaa8b3a009753ceeed20
Jul 16 23:16:59.103846 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:59.103811    2565 generic.go:356] "Generic (PLEG): container finished" podID="8163ed74-2aaa-4375-bc08-14e388ef29d8" containerID="54dc076b2abb2dd510a891aeae167881c7d5622279ecc0e6782d2fa1ae141cc7" exitCode=6
Jul 16 23:16:59.104283 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:59.103862    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5" event={"ID":"8163ed74-2aaa-4375-bc08-14e388ef29d8","Type":"ContainerDied","Data":"54dc076b2abb2dd510a891aeae167881c7d5622279ecc0e6782d2fa1ae141cc7"}
Jul 16 23:16:59.104283 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:59.103919    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5" event={"ID":"8163ed74-2aaa-4375-bc08-14e388ef29d8","Type":"ContainerStarted","Data":"4861644cc4478768e9292d0cf993f1385ad52be0bd2abaa8b3a009753ceeed20"}
Jul 16 23:16:59.104283 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:16:59.104135    2565 scope.go:117] "RemoveContainer" containerID="54dc076b2abb2dd510a891aeae167881c7d5622279ecc0e6782d2fa1ae141cc7"
Jul 16 23:17:00.111325 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:00.111199    2565 generic.go:356] "Generic (PLEG): container finished" podID="8163ed74-2aaa-4375-bc08-14e388ef29d8" containerID="ce2c9b2be2e8a85e6641269fab7be55aa763bc1954b20d2ada2066871f4b0000" exitCode=6
Jul 16 23:17:00.111325 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:00.111283    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5" event={"ID":"8163ed74-2aaa-4375-bc08-14e388ef29d8","Type":"ContainerDied","Data":"ce2c9b2be2e8a85e6641269fab7be55aa763bc1954b20d2ada2066871f4b0000"}
Jul 16 23:17:00.111795 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:00.111391    2565 scope.go:117] "RemoveContainer" containerID="54dc076b2abb2dd510a891aeae167881c7d5622279ecc0e6782d2fa1ae141cc7"
Jul 16 23:17:00.111795 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:00.111607    2565 scope.go:117] "RemoveContainer" containerID="ce2c9b2be2e8a85e6641269fab7be55aa763bc1954b20d2ada2066871f4b0000"
Jul 16 23:17:00.111870 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:17:00.111832    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5_odh-ai-gateway-infra(8163ed74-2aaa-4375-bc08-14e388ef29d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5" podUID="8163ed74-2aaa-4375-bc08-14e388ef29d8"
Jul 16 23:17:01.118387 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:01.118359    2565 scope.go:117] "RemoveContainer" containerID="ce2c9b2be2e8a85e6641269fab7be55aa763bc1954b20d2ada2066871f4b0000"
Jul 16 23:17:01.118783 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:17:01.118574    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5_odh-ai-gateway-infra(8163ed74-2aaa-4375-bc08-14e388ef29d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5" podUID="8163ed74-2aaa-4375-bc08-14e388ef29d8"
Jul 16 23:17:02.928885 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:02.928849    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn"]
Jul 16 23:17:02.933431 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:02.933411    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn"
Jul 16 23:17:02.939721 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:02.939695    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn"]
Jul 16 23:17:03.041429 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:03.041393    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cce688f-fa7b-475e-87ec-db8eb87dfc90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-x5dsn\" (UID: \"2cce688f-fa7b-475e-87ec-db8eb87dfc90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn"
Jul 16 23:17:03.142464 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:03.142420    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cce688f-fa7b-475e-87ec-db8eb87dfc90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-x5dsn\" (UID: \"2cce688f-fa7b-475e-87ec-db8eb87dfc90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn"
Jul 16 23:17:03.143036 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:03.143015    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cce688f-fa7b-475e-87ec-db8eb87dfc90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-x5dsn\" (UID: \"2cce688f-fa7b-475e-87ec-db8eb87dfc90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn"
Jul 16 23:17:03.244832 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:03.244746    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn"
Jul 16 23:17:03.377424 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:03.377395    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn"]
Jul 16 23:17:03.379649 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:17:03.379625    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cce688f_fa7b_475e_87ec_db8eb87dfc90.slice/crio-a9393be638ea3f08edfd5322d8460a14b71417aaaf941c1780be3e20ea0701a7 WatchSource:0}: Error finding container a9393be638ea3f08edfd5322d8460a14b71417aaaf941c1780be3e20ea0701a7: Status 404 returned error can't find the container with id a9393be638ea3f08edfd5322d8460a14b71417aaaf941c1780be3e20ea0701a7
Jul 16 23:17:04.134640 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:04.134603    2565 generic.go:356] "Generic (PLEG): container finished" podID="2cce688f-fa7b-475e-87ec-db8eb87dfc90" containerID="4189eb1d1a1bb3ff33a205a865347c3dcf9f22f0b741c647380c64edb5c94a2a" exitCode=6
Jul 16 23:17:04.135015 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:04.134687    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn" event={"ID":"2cce688f-fa7b-475e-87ec-db8eb87dfc90","Type":"ContainerDied","Data":"4189eb1d1a1bb3ff33a205a865347c3dcf9f22f0b741c647380c64edb5c94a2a"}
Jul 16 23:17:04.135015 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:04.134735    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn" event={"ID":"2cce688f-fa7b-475e-87ec-db8eb87dfc90","Type":"ContainerStarted","Data":"a9393be638ea3f08edfd5322d8460a14b71417aaaf941c1780be3e20ea0701a7"}
Jul 16 23:17:04.135015 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:04.134849    2565 scope.go:117] "RemoveContainer" containerID="4189eb1d1a1bb3ff33a205a865347c3dcf9f22f0b741c647380c64edb5c94a2a"
Jul 16 23:17:05.141540 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:05.141442    2565 generic.go:356] "Generic (PLEG): container finished" podID="2cce688f-fa7b-475e-87ec-db8eb87dfc90" containerID="27f02eb68c9e082889edc3b185d67f0c793fe4105a79555e8c29c388dd9371e9" exitCode=6
Jul 16 23:17:05.141953 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:05.141529    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn" event={"ID":"2cce688f-fa7b-475e-87ec-db8eb87dfc90","Type":"ContainerDied","Data":"27f02eb68c9e082889edc3b185d67f0c793fe4105a79555e8c29c388dd9371e9"}
Jul 16 23:17:05.141953 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:05.141588    2565 scope.go:117] "RemoveContainer" containerID="4189eb1d1a1bb3ff33a205a865347c3dcf9f22f0b741c647380c64edb5c94a2a"
Jul 16 23:17:05.141953 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:05.141837    2565 scope.go:117] "RemoveContainer" containerID="27f02eb68c9e082889edc3b185d67f0c793fe4105a79555e8c29c388dd9371e9"
Jul 16 23:17:05.142087 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:17:05.142067    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-x5dsn_odh-ai-gateway-infra(2cce688f-fa7b-475e-87ec-db8eb87dfc90)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn" podUID="2cce688f-fa7b-475e-87ec-db8eb87dfc90"
Jul 16 23:17:05.843016 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:05.842982    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"]
Jul 16 23:17:05.847724 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:05.847698    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:05.850092 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:05.850071    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4b61afbb-openshift-default-dockercfg-4tzv6\""
Jul 16 23:17:05.859905 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:05.859877    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"]
Jul 16 23:17:05.967549 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:05.967512    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-workload-socket\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:05.967549 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:05.967553    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7116503f-e44a-47d1-91e1-dcee065ab312-istiod-ca-cert\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:05.967769 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:05.967578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-workload-certs\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:05.967769 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:05.967594    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-istio-envoy\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:05.967769 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:05.967618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-istio-data\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:05.967769 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:05.967698    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxlqz\" (UniqueName: \"kubernetes.io/projected/7116503f-e44a-47d1-91e1-dcee065ab312-kube-api-access-cxlqz\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:05.967905 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:05.967787    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7116503f-e44a-47d1-91e1-dcee065ab312-istio-token\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:05.967905 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:05.967831    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7116503f-e44a-47d1-91e1-dcee065ab312-istio-podinfo\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:05.967905 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:05.967877    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-credential-socket\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.068597 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.068548    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7116503f-e44a-47d1-91e1-dcee065ab312-istio-token\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.068786 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.068625    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7116503f-e44a-47d1-91e1-dcee065ab312-istio-podinfo\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.068786 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.068656    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-credential-socket\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.068786 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.068720    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-workload-socket\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.068786 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.068750    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7116503f-e44a-47d1-91e1-dcee065ab312-istiod-ca-cert\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.068786 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.068780    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-workload-certs\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.069034 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.068801    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-istio-envoy\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.069034 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.068824    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-istio-data\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.069034 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.068853    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxlqz\" (UniqueName: \"kubernetes.io/projected/7116503f-e44a-47d1-91e1-dcee065ab312-kube-api-access-cxlqz\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.069206 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.069182    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-credential-socket\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.069440 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.069416    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-workload-socket\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.069535 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.069434    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-istio-data\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.069535 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.069472    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-workload-certs\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.069667 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.069647    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7116503f-e44a-47d1-91e1-dcee065ab312-istiod-ca-cert\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.071045 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.071022    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-istio-envoy\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.071339 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.071322    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7116503f-e44a-47d1-91e1-dcee065ab312-istio-podinfo\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.076132 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.076104    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7116503f-e44a-47d1-91e1-dcee065ab312-istio-token\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.076219 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.076191    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxlqz\" (UniqueName: \"kubernetes.io/projected/7116503f-e44a-47d1-91e1-dcee065ab312-kube-api-access-cxlqz\") pod \"e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") " pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.148818 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.148733    2565 scope.go:117] "RemoveContainer" containerID="27f02eb68c9e082889edc3b185d67f0c793fe4105a79555e8c29c388dd9371e9"
Jul 16 23:17:06.149197 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:17:06.148953    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-x5dsn_odh-ai-gateway-infra(2cce688f-fa7b-475e-87ec-db8eb87dfc90)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn" podUID="2cce688f-fa7b-475e-87ec-db8eb87dfc90"
Jul 16 23:17:06.157767 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.157735    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:06.298590 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.298477    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"]
Jul 16 23:17:06.303666 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:17:06.303618    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7116503f_e44a_47d1_91e1_dcee065ab312.slice/crio-5b8eb7b51046c25c171b08345fe298ce6a4a611d3e6a02f9402792ec5f4fa608 WatchSource:0}: Error finding container 5b8eb7b51046c25c171b08345fe298ce6a4a611d3e6a02f9402792ec5f4fa608: Status 404 returned error can't find the container with id 5b8eb7b51046c25c171b08345fe298ce6a4a611d3e6a02f9402792ec5f4fa608
Jul 16 23:17:06.305736 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.305702    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:17:06.305832 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.305769    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:17:06.305832 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.305800    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:17:06.720661 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:06.720635    2565 scope.go:117] "RemoveContainer" containerID="6abb68e069ff7cc58ce40458f0bd377883788d871987db70f19eb68a439b4241"
Jul 16 23:17:07.154959 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:07.154920    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4" event={"ID":"701151f8-d63f-4d88-9b9f-93a8dfce20b7","Type":"ContainerStarted","Data":"1f30d2ad0099bf06b600030f8bd541df95a8885f88426b97d36d0d02a861ee86"}
Jul 16 23:17:07.156608 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:07.156580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4" event={"ID":"7116503f-e44a-47d1-91e1-dcee065ab312","Type":"ContainerStarted","Data":"52d3ed1b54f5320fcbc2278051b45736a5018ec7711a332affc52dc88ba30067"}
Jul 16 23:17:07.156743 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:07.156633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4" event={"ID":"7116503f-e44a-47d1-91e1-dcee065ab312","Type":"ContainerStarted","Data":"5b8eb7b51046c25c171b08345fe298ce6a4a611d3e6a02f9402792ec5f4fa608"}
Jul 16 23:17:07.157981 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:07.157961    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:07.163096 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:07.163073    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:07.174960 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:07.174909    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4" podStartSLOduration=17.174894422 podStartE2EDuration="17.174894422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:17:07.173538026 +0000 UTC m=+2637.922022100" watchObservedRunningTime="2026-07-16 23:17:07.174894422 +0000 UTC m=+2637.923378539"
Jul 16 23:17:07.193759 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:07.193715    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4" podStartSLOduration=2.19370066 podStartE2EDuration="2.19370066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:17:07.192857532 +0000 UTC m=+2637.941341604" watchObservedRunningTime="2026-07-16 23:17:07.19370066 +0000 UTC m=+2637.942184730"
Jul 16 23:17:08.163352 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:08.163318    2565 generic.go:356] "Generic (PLEG): container finished" podID="701151f8-d63f-4d88-9b9f-93a8dfce20b7" containerID="1f30d2ad0099bf06b600030f8bd541df95a8885f88426b97d36d0d02a861ee86" exitCode=6
Jul 16 23:17:08.163802 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:08.163395    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4" event={"ID":"701151f8-d63f-4d88-9b9f-93a8dfce20b7","Type":"ContainerDied","Data":"1f30d2ad0099bf06b600030f8bd541df95a8885f88426b97d36d0d02a861ee86"}
Jul 16 23:17:08.163802 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:08.163449    2565 scope.go:117] "RemoveContainer" containerID="6abb68e069ff7cc58ce40458f0bd377883788d871987db70f19eb68a439b4241"
Jul 16 23:17:08.163932 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:08.163855    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:08.163932 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:08.163878    2565 scope.go:117] "RemoveContainer" containerID="1f30d2ad0099bf06b600030f8bd541df95a8885f88426b97d36d0d02a861ee86"
Jul 16 23:17:08.164115 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:17:08.164095    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4_odh-ai-gateway-infra(701151f8-d63f-4d88-9b9f-93a8dfce20b7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4" podUID="701151f8-d63f-4d88-9b9f-93a8dfce20b7"
Jul 16 23:17:08.165134 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:08.165119    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:08.184659 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:08.184630    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4"]
Jul 16 23:17:09.225827 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.225790    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"]
Jul 16 23:17:09.230586 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.230558    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"
Jul 16 23:17:09.232802 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.232774    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4b61afbb-serving-cert\""
Jul 16 23:17:09.239010 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.238711    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"]
Jul 16 23:17:09.317161 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.317126    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4"
Jul 16 23:17:09.402641 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.402603    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701151f8-d63f-4d88-9b9f-93a8dfce20b7-maas-api-service-ca\") pod \"701151f8-d63f-4d88-9b9f-93a8dfce20b7\" (UID: \"701151f8-d63f-4d88-9b9f-93a8dfce20b7\") "
Jul 16 23:17:09.402932 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.402910    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/701151f8-d63f-4d88-9b9f-93a8dfce20b7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "701151f8-d63f-4d88-9b9f-93a8dfce20b7" (UID: "701151f8-d63f-4d88-9b9f-93a8dfce20b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:17:09.403001 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.402978    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r8dz\" (UniqueName: \"kubernetes.io/projected/2f4c5ed9-a38e-4880-aa55-11339fd8b678-kube-api-access-8r8dz\") pod \"maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8\" (UID: \"2f4c5ed9-a38e-4880-aa55-11339fd8b678\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"
Jul 16 23:17:09.403051 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.403003    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4c5ed9-a38e-4880-aa55-11339fd8b678-maas-api-tls\") pod \"maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8\" (UID: \"2f4c5ed9-a38e-4880-aa55-11339fd8b678\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"
Jul 16 23:17:09.403139 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.403116    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4c5ed9-a38e-4880-aa55-11339fd8b678-openshift-service-ca\") pod \"maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8\" (UID: \"2f4c5ed9-a38e-4880-aa55-11339fd8b678\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"
Jul 16 23:17:09.403272 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.403185    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/701151f8-d63f-4d88-9b9f-93a8dfce20b7-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:17:09.504722 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.504618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r8dz\" (UniqueName: \"kubernetes.io/projected/2f4c5ed9-a38e-4880-aa55-11339fd8b678-kube-api-access-8r8dz\") pod \"maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8\" (UID: \"2f4c5ed9-a38e-4880-aa55-11339fd8b678\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"
Jul 16 23:17:09.504722 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.504668    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4c5ed9-a38e-4880-aa55-11339fd8b678-maas-api-tls\") pod \"maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8\" (UID: \"2f4c5ed9-a38e-4880-aa55-11339fd8b678\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"
Jul 16 23:17:09.504964 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.504726    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4c5ed9-a38e-4880-aa55-11339fd8b678-openshift-service-ca\") pod \"maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8\" (UID: \"2f4c5ed9-a38e-4880-aa55-11339fd8b678\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"
Jul 16 23:17:09.505475 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.505448    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4c5ed9-a38e-4880-aa55-11339fd8b678-openshift-service-ca\") pod \"maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8\" (UID: \"2f4c5ed9-a38e-4880-aa55-11339fd8b678\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"
Jul 16 23:17:09.507230 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.507205    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4c5ed9-a38e-4880-aa55-11339fd8b678-maas-api-tls\") pod \"maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8\" (UID: \"2f4c5ed9-a38e-4880-aa55-11339fd8b678\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"
Jul 16 23:17:09.518952 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.518923    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r8dz\" (UniqueName: \"kubernetes.io/projected/2f4c5ed9-a38e-4880-aa55-11339fd8b678-kube-api-access-8r8dz\") pod \"maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8\" (UID: \"2f4c5ed9-a38e-4880-aa55-11339fd8b678\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"
Jul 16 23:17:09.543485 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.543449    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"
Jul 16 23:17:09.688468 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:09.688296    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"]
Jul 16 23:17:09.690653 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:17:09.690625    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4c5ed9_a38e_4880_aa55_11339fd8b678.slice/crio-34dac89971838a813e35be7a969f77083206228d29c539bf02fa1c4aab46ab42 WatchSource:0}: Error finding container 34dac89971838a813e35be7a969f77083206228d29c539bf02fa1c4aab46ab42: Status 404 returned error can't find the container with id 34dac89971838a813e35be7a969f77083206228d29c539bf02fa1c4aab46ab42
Jul 16 23:17:10.175292 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:10.175268    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4"
Jul 16 23:17:10.175401 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:10.175284    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4" event={"ID":"701151f8-d63f-4d88-9b9f-93a8dfce20b7","Type":"ContainerDied","Data":"9d0d24dc2e05fdba89dcb948e664c63a9b725b979cea587622ac821a4ddce8ab"}
Jul 16 23:17:10.175401 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:10.175352    2565 scope.go:117] "RemoveContainer" containerID="1f30d2ad0099bf06b600030f8bd541df95a8885f88426b97d36d0d02a861ee86"
Jul 16 23:17:10.176633 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:10.176601    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8" event={"ID":"2f4c5ed9-a38e-4880-aa55-11339fd8b678","Type":"ContainerStarted","Data":"34dac89971838a813e35be7a969f77083206228d29c539bf02fa1c4aab46ab42"}
Jul 16 23:17:10.202071 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:10.202037    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4"]
Jul 16 23:17:10.208424 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:10.208397    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-bmcf4"]
Jul 16 23:17:11.183917 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:11.183881    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8" event={"ID":"2f4c5ed9-a38e-4880-aa55-11339fd8b678","Type":"ContainerStarted","Data":"75b4774405c54325cf29b824d4e98e68cd164953b3f3e0d2dd11c5774368e9fe"}
Jul 16 23:17:11.184394 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:11.183982    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"
Jul 16 23:17:11.199507 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:11.199427    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8" podStartSLOduration=1.73887071 podStartE2EDuration="2.199410448s" totalImagesPullingTime="460.539738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:17:11.198478092 +0000 UTC m=+2641.946962174" watchObservedRunningTime="2026-07-16 23:17:11.199410448 +0000 UTC m=+2641.947894550"
Jul 16 23:17:11.726622 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:11.726589    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="701151f8-d63f-4d88-9b9f-93a8dfce20b7" path="/var/lib/kubelet/pods/701151f8-d63f-4d88-9b9f-93a8dfce20b7/volumes"
Jul 16 23:17:13.720670 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:13.720638    2565 scope.go:117] "RemoveContainer" containerID="ce2c9b2be2e8a85e6641269fab7be55aa763bc1954b20d2ada2066871f4b0000"
Jul 16 23:17:14.200385 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:14.200345    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5" event={"ID":"8163ed74-2aaa-4375-bc08-14e388ef29d8","Type":"ContainerStarted","Data":"6ddba18afc76b8b0c54f52eac70b84b39e4c1cb9a2d0d14505d81f99eec01968"}
Jul 16 23:17:14.218927 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:14.218876    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5" podStartSLOduration=17.218859228 podStartE2EDuration="17.218859228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:17:14.216443029 +0000 UTC m=+2644.964927106" watchObservedRunningTime="2026-07-16 23:17:14.218859228 +0000 UTC m=+2644.967343298"
Jul 16 23:17:15.213572 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:15.213538    2565 generic.go:356] "Generic (PLEG): container finished" podID="8163ed74-2aaa-4375-bc08-14e388ef29d8" containerID="6ddba18afc76b8b0c54f52eac70b84b39e4c1cb9a2d0d14505d81f99eec01968" exitCode=6
Jul 16 23:17:15.214040 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:15.213600    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5" event={"ID":"8163ed74-2aaa-4375-bc08-14e388ef29d8","Type":"ContainerDied","Data":"6ddba18afc76b8b0c54f52eac70b84b39e4c1cb9a2d0d14505d81f99eec01968"}
Jul 16 23:17:15.214040 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:15.213668    2565 scope.go:117] "RemoveContainer" containerID="ce2c9b2be2e8a85e6641269fab7be55aa763bc1954b20d2ada2066871f4b0000"
Jul 16 23:17:15.214040 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:15.213977    2565 scope.go:117] "RemoveContainer" containerID="6ddba18afc76b8b0c54f52eac70b84b39e4c1cb9a2d0d14505d81f99eec01968"
Jul 16 23:17:15.214237 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:17:15.214213    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5_odh-ai-gateway-infra(8163ed74-2aaa-4375-bc08-14e388ef29d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5" podUID="8163ed74-2aaa-4375-bc08-14e388ef29d8"
Jul 16 23:17:15.228109 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:15.228073    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5"]
Jul 16 23:17:16.401168 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:16.401143    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5"
Jul 16 23:17:16.471173 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:16.471068    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8163ed74-2aaa-4375-bc08-14e388ef29d8-maas-api-service-ca\") pod \"8163ed74-2aaa-4375-bc08-14e388ef29d8\" (UID: \"8163ed74-2aaa-4375-bc08-14e388ef29d8\") "
Jul 16 23:17:16.471494 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:16.471467    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8163ed74-2aaa-4375-bc08-14e388ef29d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8163ed74-2aaa-4375-bc08-14e388ef29d8" (UID: "8163ed74-2aaa-4375-bc08-14e388ef29d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:17:16.571972 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:16.571939    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8163ed74-2aaa-4375-bc08-14e388ef29d8-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:17:17.195704 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:17.195675    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"
Jul 16 23:17:17.228694 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:17.228666    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5"
Jul 16 23:17:17.228694 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:17.228672    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5" event={"ID":"8163ed74-2aaa-4375-bc08-14e388ef29d8","Type":"ContainerDied","Data":"4861644cc4478768e9292d0cf993f1385ad52be0bd2abaa8b3a009753ceeed20"}
Jul 16 23:17:17.228921 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:17.228746    2565 scope.go:117] "RemoveContainer" containerID="6ddba18afc76b8b0c54f52eac70b84b39e4c1cb9a2d0d14505d81f99eec01968"
Jul 16 23:17:17.276363 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:17.276322    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5"]
Jul 16 23:17:17.280520 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:17.280494    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-qcmt5"]
Jul 16 23:17:17.726830 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:17.726800    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8163ed74-2aaa-4375-bc08-14e388ef29d8" path="/var/lib/kubelet/pods/8163ed74-2aaa-4375-bc08-14e388ef29d8/volumes"
Jul 16 23:17:20.721281 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:20.721221    2565 scope.go:117] "RemoveContainer" containerID="27f02eb68c9e082889edc3b185d67f0c793fe4105a79555e8c29c388dd9371e9"
Jul 16 23:17:21.254134 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:21.254098    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn" event={"ID":"2cce688f-fa7b-475e-87ec-db8eb87dfc90","Type":"ContainerStarted","Data":"04246dc33055760d1e66164eb0595a23a1b07f331bf7f7a89396dbfd9e630122"}
Jul 16 23:17:21.269867 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:21.269806    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn" podStartSLOduration=19.269786229 podStartE2EDuration="19.269786229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:17:21.269187049 +0000 UTC m=+2652.017671119" watchObservedRunningTime="2026-07-16 23:17:21.269786229 +0000 UTC m=+2652.018270301"
Jul 16 23:17:22.262539 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:22.262505    2565 generic.go:356] "Generic (PLEG): container finished" podID="2cce688f-fa7b-475e-87ec-db8eb87dfc90" containerID="04246dc33055760d1e66164eb0595a23a1b07f331bf7f7a89396dbfd9e630122" exitCode=6
Jul 16 23:17:22.262989 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:22.262566    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn" event={"ID":"2cce688f-fa7b-475e-87ec-db8eb87dfc90","Type":"ContainerDied","Data":"04246dc33055760d1e66164eb0595a23a1b07f331bf7f7a89396dbfd9e630122"}
Jul 16 23:17:22.262989 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:22.262617    2565 scope.go:117] "RemoveContainer" containerID="27f02eb68c9e082889edc3b185d67f0c793fe4105a79555e8c29c388dd9371e9"
Jul 16 23:17:22.263087 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:22.263018    2565 scope.go:117] "RemoveContainer" containerID="04246dc33055760d1e66164eb0595a23a1b07f331bf7f7a89396dbfd9e630122"
Jul 16 23:17:22.263303 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:17:22.263279    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-x5dsn_odh-ai-gateway-infra(2cce688f-fa7b-475e-87ec-db8eb87dfc90)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn" podUID="2cce688f-fa7b-475e-87ec-db8eb87dfc90"
Jul 16 23:17:22.280064 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:22.280035    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn"]
Jul 16 23:17:23.165829 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:23.165791    2565 patch_prober.go:28] interesting pod/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 16 23:17:23.166040 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:23.165843    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4" podUID="7116503f-e44a-47d1-91e1-dcee065ab312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 16 23:17:23.270356 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:23.270329    2565 generic.go:356] "Generic (PLEG): container finished" podID="7116503f-e44a-47d1-91e1-dcee065ab312" containerID="52d3ed1b54f5320fcbc2278051b45736a5018ec7711a332affc52dc88ba30067" exitCode=0
Jul 16 23:17:23.270649 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:23.270411    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4" event={"ID":"7116503f-e44a-47d1-91e1-dcee065ab312","Type":"ContainerDied","Data":"52d3ed1b54f5320fcbc2278051b45736a5018ec7711a332affc52dc88ba30067"}
Jul 16 23:17:23.270941 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:23.270921    2565 scope.go:117] "RemoveContainer" containerID="52d3ed1b54f5320fcbc2278051b45736a5018ec7711a332affc52dc88ba30067"
Jul 16 23:17:23.272062 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:23.272036    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:17:23.272200 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:23.272106    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:17:23.272200 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:23.272141    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:17:23.407320 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:23.407291    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn"
Jul 16 23:17:23.431766 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:23.431678    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cce688f-fa7b-475e-87ec-db8eb87dfc90-maas-api-service-ca\") pod \"2cce688f-fa7b-475e-87ec-db8eb87dfc90\" (UID: \"2cce688f-fa7b-475e-87ec-db8eb87dfc90\") "
Jul 16 23:17:23.432112 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:23.432086    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cce688f-fa7b-475e-87ec-db8eb87dfc90-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2cce688f-fa7b-475e-87ec-db8eb87dfc90" (UID: "2cce688f-fa7b-475e-87ec-db8eb87dfc90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:17:23.533381 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:23.533341    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cce688f-fa7b-475e-87ec-db8eb87dfc90-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:17:24.276769 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:24.276740    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn"
Jul 16 23:17:24.277287 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:24.276740    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn" event={"ID":"2cce688f-fa7b-475e-87ec-db8eb87dfc90","Type":"ContainerDied","Data":"a9393be638ea3f08edfd5322d8460a14b71417aaaf941c1780be3e20ea0701a7"}
Jul 16 23:17:24.277287 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:24.276889    2565 scope.go:117] "RemoveContainer" containerID="04246dc33055760d1e66164eb0595a23a1b07f331bf7f7a89396dbfd9e630122"
Jul 16 23:17:24.278833 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:24.278785    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4" event={"ID":"7116503f-e44a-47d1-91e1-dcee065ab312","Type":"ContainerStarted","Data":"05c353b817a02814b15fb56d3e2372958e41366fe49d7435a3358032a9cd23d7"}
Jul 16 23:17:24.291526 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:24.291500    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn"]
Jul 16 23:17:24.296942 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:24.296911    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-x5dsn"]
Jul 16 23:17:25.158743 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:25.158691    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:25.158989 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:25.158960    2565 patch_prober.go:28] interesting pod/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 16 23:17:25.159060 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:25.159034    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4" podUID="7116503f-e44a-47d1-91e1-dcee065ab312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 16 23:17:25.726757 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:25.726721    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cce688f-fa7b-475e-87ec-db8eb87dfc90" path="/var/lib/kubelet/pods/2cce688f-fa7b-475e-87ec-db8eb87dfc90/volumes"
Jul 16 23:17:26.158919 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:26.158869    2565 patch_prober.go:28] interesting pod/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 16 23:17:26.159143 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:26.158961    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4" podUID="7116503f-e44a-47d1-91e1-dcee065ab312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 16 23:17:27.158199 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:27.158157    2565 patch_prober.go:28] interesting pod/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 16 23:17:27.158676 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:27.158324    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4" podUID="7116503f-e44a-47d1-91e1-dcee065ab312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 16 23:17:28.162312 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:28.162269    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:28.162735 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:28.162667    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:28.163437 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:28.163418    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:31.486919 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:31.486883    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"]
Jul 16 23:17:31.487506 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:31.487105    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8" podUID="2f4c5ed9-a38e-4880-aa55-11339fd8b678" containerName="maas-api" containerID="cri-o://75b4774405c54325cf29b824d4e98e68cd164953b3f3e0d2dd11c5774368e9fe" gracePeriod=30
Jul 16 23:17:31.745432 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:31.745360    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"
Jul 16 23:17:31.816642 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:31.816602    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r8dz\" (UniqueName: \"kubernetes.io/projected/2f4c5ed9-a38e-4880-aa55-11339fd8b678-kube-api-access-8r8dz\") pod \"2f4c5ed9-a38e-4880-aa55-11339fd8b678\" (UID: \"2f4c5ed9-a38e-4880-aa55-11339fd8b678\") "
Jul 16 23:17:31.816859 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:31.816653    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4c5ed9-a38e-4880-aa55-11339fd8b678-maas-api-tls\") pod \"2f4c5ed9-a38e-4880-aa55-11339fd8b678\" (UID: \"2f4c5ed9-a38e-4880-aa55-11339fd8b678\") "
Jul 16 23:17:31.816859 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:31.816690    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4c5ed9-a38e-4880-aa55-11339fd8b678-openshift-service-ca\") pod \"2f4c5ed9-a38e-4880-aa55-11339fd8b678\" (UID: \"2f4c5ed9-a38e-4880-aa55-11339fd8b678\") "
Jul 16 23:17:31.817100 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:31.817072    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4c5ed9-a38e-4880-aa55-11339fd8b678-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2f4c5ed9-a38e-4880-aa55-11339fd8b678" (UID: "2f4c5ed9-a38e-4880-aa55-11339fd8b678"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:17:31.818734 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:31.818709    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4c5ed9-a38e-4880-aa55-11339fd8b678-kube-api-access-8r8dz" (OuterVolumeSpecName: "kube-api-access-8r8dz") pod "2f4c5ed9-a38e-4880-aa55-11339fd8b678" (UID: "2f4c5ed9-a38e-4880-aa55-11339fd8b678"). InnerVolumeSpecName "kube-api-access-8r8dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:17:31.818837 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:31.818795    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4c5ed9-a38e-4880-aa55-11339fd8b678-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2f4c5ed9-a38e-4880-aa55-11339fd8b678" (UID: "2f4c5ed9-a38e-4880-aa55-11339fd8b678"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:17:31.917810 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:31.917766    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r8dz\" (UniqueName: \"kubernetes.io/projected/2f4c5ed9-a38e-4880-aa55-11339fd8b678-kube-api-access-8r8dz\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:17:31.917810 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:31.917800    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4c5ed9-a38e-4880-aa55-11339fd8b678-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:17:31.917810 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:31.917811    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4c5ed9-a38e-4880-aa55-11339fd8b678-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:17:32.321508 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:32.321470    2565 generic.go:356] "Generic (PLEG): container finished" podID="2f4c5ed9-a38e-4880-aa55-11339fd8b678" containerID="75b4774405c54325cf29b824d4e98e68cd164953b3f3e0d2dd11c5774368e9fe" exitCode=0
Jul 16 23:17:32.321681 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:32.321513    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"
Jul 16 23:17:32.321681 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:32.321519    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8" event={"ID":"2f4c5ed9-a38e-4880-aa55-11339fd8b678","Type":"ContainerDied","Data":"75b4774405c54325cf29b824d4e98e68cd164953b3f3e0d2dd11c5774368e9fe"}
Jul 16 23:17:32.321681 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:32.321585    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8" event={"ID":"2f4c5ed9-a38e-4880-aa55-11339fd8b678","Type":"ContainerDied","Data":"34dac89971838a813e35be7a969f77083206228d29c539bf02fa1c4aab46ab42"}
Jul 16 23:17:32.321681 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:32.321615    2565 scope.go:117] "RemoveContainer" containerID="75b4774405c54325cf29b824d4e98e68cd164953b3f3e0d2dd11c5774368e9fe"
Jul 16 23:17:32.331828 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:32.331810    2565 scope.go:117] "RemoveContainer" containerID="75b4774405c54325cf29b824d4e98e68cd164953b3f3e0d2dd11c5774368e9fe"
Jul 16 23:17:32.332119 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:17:32.332097    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b4774405c54325cf29b824d4e98e68cd164953b3f3e0d2dd11c5774368e9fe\": container with ID starting with 75b4774405c54325cf29b824d4e98e68cd164953b3f3e0d2dd11c5774368e9fe not found: ID does not exist" containerID="75b4774405c54325cf29b824d4e98e68cd164953b3f3e0d2dd11c5774368e9fe"
Jul 16 23:17:32.332258 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:32.332124    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b4774405c54325cf29b824d4e98e68cd164953b3f3e0d2dd11c5774368e9fe"} err="failed to get container status \"75b4774405c54325cf29b824d4e98e68cd164953b3f3e0d2dd11c5774368e9fe\": rpc error: code = NotFound desc = could not find container \"75b4774405c54325cf29b824d4e98e68cd164953b3f3e0d2dd11c5774368e9fe\": container with ID starting with 75b4774405c54325cf29b824d4e98e68cd164953b3f3e0d2dd11c5774368e9fe not found: ID does not exist"
Jul 16 23:17:32.342685 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:32.342651    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"]
Jul 16 23:17:32.345092 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:32.345063    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4b61afbb-b4d485dbb-rdwp8"]
Jul 16 23:17:33.726647 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:33.726602    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f4c5ed9-a38e-4880-aa55-11339fd8b678" path="/var/lib/kubelet/pods/2f4c5ed9-a38e-4880-aa55-11339fd8b678/volumes"
Jul 16 23:17:40.713657 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.713612    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74"]
Jul 16 23:17:40.714076 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.713994    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cce688f-fa7b-475e-87ec-db8eb87dfc90" containerName="revoke-keys"
Jul 16 23:17:40.714076 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714005    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cce688f-fa7b-475e-87ec-db8eb87dfc90" containerName="revoke-keys"
Jul 16 23:17:40.714076 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714020    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8163ed74-2aaa-4375-bc08-14e388ef29d8" containerName="revoke-keys"
Jul 16 23:17:40.714076 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714025    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8163ed74-2aaa-4375-bc08-14e388ef29d8" containerName="revoke-keys"
Jul 16 23:17:40.714076 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714036    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8163ed74-2aaa-4375-bc08-14e388ef29d8" containerName="revoke-keys"
Jul 16 23:17:40.714076 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714041    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8163ed74-2aaa-4375-bc08-14e388ef29d8" containerName="revoke-keys"
Jul 16 23:17:40.714076 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714049    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cce688f-fa7b-475e-87ec-db8eb87dfc90" containerName="revoke-keys"
Jul 16 23:17:40.714076 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714054    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cce688f-fa7b-475e-87ec-db8eb87dfc90" containerName="revoke-keys"
Jul 16 23:17:40.714076 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714065    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701151f8-d63f-4d88-9b9f-93a8dfce20b7" containerName="revoke-keys"
Jul 16 23:17:40.714076 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714070    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="701151f8-d63f-4d88-9b9f-93a8dfce20b7" containerName="revoke-keys"
Jul 16 23:17:40.714076 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714076    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701151f8-d63f-4d88-9b9f-93a8dfce20b7" containerName="revoke-keys"
Jul 16 23:17:40.714076 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714081    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="701151f8-d63f-4d88-9b9f-93a8dfce20b7" containerName="revoke-keys"
Jul 16 23:17:40.714467 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714088    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8163ed74-2aaa-4375-bc08-14e388ef29d8" containerName="revoke-keys"
Jul 16 23:17:40.714467 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714093    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8163ed74-2aaa-4375-bc08-14e388ef29d8" containerName="revoke-keys"
Jul 16 23:17:40.714467 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714100    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701151f8-d63f-4d88-9b9f-93a8dfce20b7" containerName="revoke-keys"
Jul 16 23:17:40.714467 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714105    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="701151f8-d63f-4d88-9b9f-93a8dfce20b7" containerName="revoke-keys"
Jul 16 23:17:40.714467 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714112    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f4c5ed9-a38e-4880-aa55-11339fd8b678" containerName="maas-api"
Jul 16 23:17:40.714467 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714117    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f4c5ed9-a38e-4880-aa55-11339fd8b678" containerName="maas-api"
Jul 16 23:17:40.714467 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714168    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="701151f8-d63f-4d88-9b9f-93a8dfce20b7" containerName="revoke-keys"
Jul 16 23:17:40.714467 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714175    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f4c5ed9-a38e-4880-aa55-11339fd8b678" containerName="maas-api"
Jul 16 23:17:40.714467 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714183    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cce688f-fa7b-475e-87ec-db8eb87dfc90" containerName="revoke-keys"
Jul 16 23:17:40.714467 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714189    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="701151f8-d63f-4d88-9b9f-93a8dfce20b7" containerName="revoke-keys"
Jul 16 23:17:40.714467 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714195    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8163ed74-2aaa-4375-bc08-14e388ef29d8" containerName="revoke-keys"
Jul 16 23:17:40.714467 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714201    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="701151f8-d63f-4d88-9b9f-93a8dfce20b7" containerName="revoke-keys"
Jul 16 23:17:40.714467 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714207    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cce688f-fa7b-475e-87ec-db8eb87dfc90" containerName="revoke-keys"
Jul 16 23:17:40.714467 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.714212    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8163ed74-2aaa-4375-bc08-14e388ef29d8" containerName="revoke-keys"
Jul 16 23:17:40.717321 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.717302    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74"
Jul 16 23:17:40.719345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.719319    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:17:40.723428 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.723387    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74"]
Jul 16 23:17:40.801775 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.801728    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74\" (UID: \"aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74"
Jul 16 23:17:40.903224 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.903179    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74\" (UID: \"aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74"
Jul 16 23:17:40.903870 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:40.903845    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74\" (UID: \"aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74"
Jul 16 23:17:41.030049 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:41.029958    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74"
Jul 16 23:17:41.166853 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:41.166826    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74"]
Jul 16 23:17:41.169361 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:17:41.169316    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa37c1a3_cbec_4e94_a2ac_33e6a67a32a6.slice/crio-9e8fdf60591a1b3dd32da47d12df618fcf9387f204109a18a458da5a06333057 WatchSource:0}: Error finding container 9e8fdf60591a1b3dd32da47d12df618fcf9387f204109a18a458da5a06333057: Status 404 returned error can't find the container with id 9e8fdf60591a1b3dd32da47d12df618fcf9387f204109a18a458da5a06333057
Jul 16 23:17:41.368037 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:41.367989    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74" event={"ID":"aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6","Type":"ContainerStarted","Data":"9e8fdf60591a1b3dd32da47d12df618fcf9387f204109a18a458da5a06333057"}
Jul 16 23:17:42.374516 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:42.374474    2565 generic.go:356] "Generic (PLEG): container finished" podID="aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6" containerID="e4d52a449c9c718f50e3b67b03895d28ccd73b848260b7fbd5a2d60a21013b15" exitCode=6
Jul 16 23:17:42.374926 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:42.374557    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74" event={"ID":"aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6","Type":"ContainerDied","Data":"e4d52a449c9c718f50e3b67b03895d28ccd73b848260b7fbd5a2d60a21013b15"}
Jul 16 23:17:42.374926 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:42.374773    2565 scope.go:117] "RemoveContainer" containerID="e4d52a449c9c718f50e3b67b03895d28ccd73b848260b7fbd5a2d60a21013b15"
Jul 16 23:17:43.381104 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:43.381072    2565 generic.go:356] "Generic (PLEG): container finished" podID="aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6" containerID="5decc1f9a719a957c2226e76a3f5389014e04de18b86a43a6621f06671a25b39" exitCode=6
Jul 16 23:17:43.381555 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:43.381119    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74" event={"ID":"aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6","Type":"ContainerDied","Data":"5decc1f9a719a957c2226e76a3f5389014e04de18b86a43a6621f06671a25b39"}
Jul 16 23:17:43.381555 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:43.381166    2565 scope.go:117] "RemoveContainer" containerID="e4d52a449c9c718f50e3b67b03895d28ccd73b848260b7fbd5a2d60a21013b15"
Jul 16 23:17:43.381555 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:43.381366    2565 scope.go:117] "RemoveContainer" containerID="5decc1f9a719a957c2226e76a3f5389014e04de18b86a43a6621f06671a25b39"
Jul 16 23:17:43.381671 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:17:43.381611    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74_odh-ai-gateway-infra(aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74" podUID="aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6"
Jul 16 23:17:44.387697 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:44.387667    2565 scope.go:117] "RemoveContainer" containerID="5decc1f9a719a957c2226e76a3f5389014e04de18b86a43a6621f06671a25b39"
Jul 16 23:17:44.388081 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:17:44.387857    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74_odh-ai-gateway-infra(aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74" podUID="aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6"
Jul 16 23:17:46.584123 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:46.584087    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"]
Jul 16 23:17:46.594708 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:46.594653    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4" podUID="7116503f-e44a-47d1-91e1-dcee065ab312" containerName="istio-proxy" containerID="cri-o://05c353b817a02814b15fb56d3e2372958e41366fe49d7435a3358032a9cd23d7" gracePeriod=30
Jul 16 23:17:47.905374 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:47.905343    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5"]
Jul 16 23:17:47.905747 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:47.905711    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cce688f-fa7b-475e-87ec-db8eb87dfc90" containerName="revoke-keys"
Jul 16 23:17:47.905747 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:47.905722    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cce688f-fa7b-475e-87ec-db8eb87dfc90" containerName="revoke-keys"
Jul 16 23:17:47.905836 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:47.905782    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cce688f-fa7b-475e-87ec-db8eb87dfc90" containerName="revoke-keys"
Jul 16 23:17:47.905836 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:47.905792    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8163ed74-2aaa-4375-bc08-14e388ef29d8" containerName="revoke-keys"
Jul 16 23:17:47.907563 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:47.907546    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5"
Jul 16 23:17:47.915873 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:47.915850    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5"]
Jul 16 23:17:47.977347 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:47.977310    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1744f8-214d-423c-bc1f-418766290cb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5\" (UID: \"3f1744f8-214d-423c-bc1f-418766290cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5"
Jul 16 23:17:48.078545 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:48.078509    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1744f8-214d-423c-bc1f-418766290cb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5\" (UID: \"3f1744f8-214d-423c-bc1f-418766290cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5"
Jul 16 23:17:48.079103 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:48.079082    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1744f8-214d-423c-bc1f-418766290cb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5\" (UID: \"3f1744f8-214d-423c-bc1f-418766290cb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5"
Jul 16 23:17:48.218662 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:48.218577    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5"
Jul 16 23:17:48.354424 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:48.354396    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5"]
Jul 16 23:17:48.356826 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:17:48.356799    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1744f8_214d_423c_bc1f_418766290cb8.slice/crio-20f08e6ba7729c3948db674f5e3f2aec52f7bf2bf55e0a4f825370779b38316a WatchSource:0}: Error finding container 20f08e6ba7729c3948db674f5e3f2aec52f7bf2bf55e0a4f825370779b38316a: Status 404 returned error can't find the container with id 20f08e6ba7729c3948db674f5e3f2aec52f7bf2bf55e0a4f825370779b38316a
Jul 16 23:17:48.416561 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:48.416524    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5" event={"ID":"3f1744f8-214d-423c-bc1f-418766290cb8","Type":"ContainerStarted","Data":"20f08e6ba7729c3948db674f5e3f2aec52f7bf2bf55e0a4f825370779b38316a"}
Jul 16 23:17:49.423334 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:49.423301    2565 generic.go:356] "Generic (PLEG): container finished" podID="3f1744f8-214d-423c-bc1f-418766290cb8" containerID="2ec125d447cc2377b6339058c143124a117ef2b905c38faeb7fe8566640b6cd1" exitCode=6
Jul 16 23:17:49.423770 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:49.423360    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5" event={"ID":"3f1744f8-214d-423c-bc1f-418766290cb8","Type":"ContainerDied","Data":"2ec125d447cc2377b6339058c143124a117ef2b905c38faeb7fe8566640b6cd1"}
Jul 16 23:17:49.423770 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:49.423598    2565 scope.go:117] "RemoveContainer" containerID="2ec125d447cc2377b6339058c143124a117ef2b905c38faeb7fe8566640b6cd1"
Jul 16 23:17:50.429748 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:50.429659    2565 generic.go:356] "Generic (PLEG): container finished" podID="3f1744f8-214d-423c-bc1f-418766290cb8" containerID="3837cd48c42049177b9df493fd09384c91c104b1d277bd72c67d2f98e78bc96d" exitCode=6
Jul 16 23:17:50.430172 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:50.429746    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5" event={"ID":"3f1744f8-214d-423c-bc1f-418766290cb8","Type":"ContainerDied","Data":"3837cd48c42049177b9df493fd09384c91c104b1d277bd72c67d2f98e78bc96d"}
Jul 16 23:17:50.430172 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:50.429808    2565 scope.go:117] "RemoveContainer" containerID="2ec125d447cc2377b6339058c143124a117ef2b905c38faeb7fe8566640b6cd1"
Jul 16 23:17:50.430172 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:50.430026    2565 scope.go:117] "RemoveContainer" containerID="3837cd48c42049177b9df493fd09384c91c104b1d277bd72c67d2f98e78bc96d"
Jul 16 23:17:50.430341 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:17:50.430317    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5_odh-ai-gateway-infra(3f1744f8-214d-423c-bc1f-418766290cb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5" podUID="3f1744f8-214d-423c-bc1f-418766290cb8"
Jul 16 23:17:51.436537 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.436509    2565 scope.go:117] "RemoveContainer" containerID="3837cd48c42049177b9df493fd09384c91c104b1d277bd72c67d2f98e78bc96d"
Jul 16 23:17:51.436920 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:17:51.436706    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5_odh-ai-gateway-infra(3f1744f8-214d-423c-bc1f-418766290cb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5" podUID="3f1744f8-214d-423c-bc1f-418766290cb8"
Jul 16 23:17:51.841348 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.841325    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:51.914278 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.914215    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7116503f-e44a-47d1-91e1-dcee065ab312-istiod-ca-cert\") pod \"7116503f-e44a-47d1-91e1-dcee065ab312\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") "
Jul 16 23:17:51.914490 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.914300    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-workload-socket\") pod \"7116503f-e44a-47d1-91e1-dcee065ab312\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") "
Jul 16 23:17:51.914490 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.914411    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-workload-certs\") pod \"7116503f-e44a-47d1-91e1-dcee065ab312\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") "
Jul 16 23:17:51.914623 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.914529    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxlqz\" (UniqueName: \"kubernetes.io/projected/7116503f-e44a-47d1-91e1-dcee065ab312-kube-api-access-cxlqz\") pod \"7116503f-e44a-47d1-91e1-dcee065ab312\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") "
Jul 16 23:17:51.914623 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.914577    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-istio-data\") pod \"7116503f-e44a-47d1-91e1-dcee065ab312\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") "
Jul 16 23:17:51.914623 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.914603    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7116503f-e44a-47d1-91e1-dcee065ab312-istio-token\") pod \"7116503f-e44a-47d1-91e1-dcee065ab312\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") "
Jul 16 23:17:51.914771 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.914615    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7116503f-e44a-47d1-91e1-dcee065ab312" (UID: "7116503f-e44a-47d1-91e1-dcee065ab312"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:17:51.914771 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.914626    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-credential-socket\") pod \"7116503f-e44a-47d1-91e1-dcee065ab312\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") "
Jul 16 23:17:51.914771 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.914658    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7116503f-e44a-47d1-91e1-dcee065ab312" (UID: "7116503f-e44a-47d1-91e1-dcee065ab312"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:17:51.914771 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.914694    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-istio-envoy\") pod \"7116503f-e44a-47d1-91e1-dcee065ab312\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") "
Jul 16 23:17:51.914771 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.914724    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7116503f-e44a-47d1-91e1-dcee065ab312-istio-podinfo\") pod \"7116503f-e44a-47d1-91e1-dcee065ab312\" (UID: \"7116503f-e44a-47d1-91e1-dcee065ab312\") "
Jul 16 23:17:51.914771 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.914691    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7116503f-e44a-47d1-91e1-dcee065ab312-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7116503f-e44a-47d1-91e1-dcee065ab312" (UID: "7116503f-e44a-47d1-91e1-dcee065ab312"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:17:51.915069 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.914866    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7116503f-e44a-47d1-91e1-dcee065ab312" (UID: "7116503f-e44a-47d1-91e1-dcee065ab312"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:17:51.915069 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.915052    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7116503f-e44a-47d1-91e1-dcee065ab312-istiod-ca-cert\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:17:51.915163 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.915069    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-workload-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:17:51.915163 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.915084    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-workload-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:17:51.915163 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.915098    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-credential-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:17:51.915491 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.915462    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-istio-data" (OuterVolumeSpecName: "istio-data") pod "7116503f-e44a-47d1-91e1-dcee065ab312" (UID: "7116503f-e44a-47d1-91e1-dcee065ab312"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:17:51.916840 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.916818    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7116503f-e44a-47d1-91e1-dcee065ab312-kube-api-access-cxlqz" (OuterVolumeSpecName: "kube-api-access-cxlqz") pod "7116503f-e44a-47d1-91e1-dcee065ab312" (UID: "7116503f-e44a-47d1-91e1-dcee065ab312"). InnerVolumeSpecName "kube-api-access-cxlqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:17:51.916922 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.916866    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7116503f-e44a-47d1-91e1-dcee065ab312-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7116503f-e44a-47d1-91e1-dcee065ab312" (UID: "7116503f-e44a-47d1-91e1-dcee065ab312"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 23:17:51.917139 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.917122    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7116503f-e44a-47d1-91e1-dcee065ab312" (UID: "7116503f-e44a-47d1-91e1-dcee065ab312"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:17:51.917200 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:51.917178    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7116503f-e44a-47d1-91e1-dcee065ab312-istio-token" (OuterVolumeSpecName: "istio-token") pod "7116503f-e44a-47d1-91e1-dcee065ab312" (UID: "7116503f-e44a-47d1-91e1-dcee065ab312"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:17:52.016056 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:52.015953    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxlqz\" (UniqueName: \"kubernetes.io/projected/7116503f-e44a-47d1-91e1-dcee065ab312-kube-api-access-cxlqz\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:17:52.016056 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:52.015986    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-istio-data\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:17:52.016056 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:52.015999    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7116503f-e44a-47d1-91e1-dcee065ab312-istio-token\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:17:52.016056 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:52.016009    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7116503f-e44a-47d1-91e1-dcee065ab312-istio-envoy\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:17:52.016056 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:52.016022    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7116503f-e44a-47d1-91e1-dcee065ab312-istio-podinfo\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:17:52.442312 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:52.442279    2565 generic.go:356] "Generic (PLEG): container finished" podID="7116503f-e44a-47d1-91e1-dcee065ab312" containerID="05c353b817a02814b15fb56d3e2372958e41366fe49d7435a3358032a9cd23d7" exitCode=0
Jul 16 23:17:52.442736 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:52.442332    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"
Jul 16 23:17:52.442736 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:52.442363    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4" event={"ID":"7116503f-e44a-47d1-91e1-dcee065ab312","Type":"ContainerDied","Data":"05c353b817a02814b15fb56d3e2372958e41366fe49d7435a3358032a9cd23d7"}
Jul 16 23:17:52.442736 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:52.442413    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4" event={"ID":"7116503f-e44a-47d1-91e1-dcee065ab312","Type":"ContainerDied","Data":"5b8eb7b51046c25c171b08345fe298ce6a4a611d3e6a02f9402792ec5f4fa608"}
Jul 16 23:17:52.442736 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:52.442436    2565 scope.go:117] "RemoveContainer" containerID="05c353b817a02814b15fb56d3e2372958e41366fe49d7435a3358032a9cd23d7"
Jul 16 23:17:52.451982 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:52.451968    2565 scope.go:117] "RemoveContainer" containerID="52d3ed1b54f5320fcbc2278051b45736a5018ec7711a332affc52dc88ba30067"
Jul 16 23:17:52.460201 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:52.460182    2565 scope.go:117] "RemoveContainer" containerID="05c353b817a02814b15fb56d3e2372958e41366fe49d7435a3358032a9cd23d7"
Jul 16 23:17:52.460494 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:17:52.460475    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05c353b817a02814b15fb56d3e2372958e41366fe49d7435a3358032a9cd23d7\": container with ID starting with 05c353b817a02814b15fb56d3e2372958e41366fe49d7435a3358032a9cd23d7 not found: ID does not exist" containerID="05c353b817a02814b15fb56d3e2372958e41366fe49d7435a3358032a9cd23d7"
Jul 16 23:17:52.460564 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:52.460503    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c353b817a02814b15fb56d3e2372958e41366fe49d7435a3358032a9cd23d7"} err="failed to get container status \"05c353b817a02814b15fb56d3e2372958e41366fe49d7435a3358032a9cd23d7\": rpc error: code = NotFound desc = could not find container \"05c353b817a02814b15fb56d3e2372958e41366fe49d7435a3358032a9cd23d7\": container with ID starting with 05c353b817a02814b15fb56d3e2372958e41366fe49d7435a3358032a9cd23d7 not found: ID does not exist"
Jul 16 23:17:52.460564 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:52.460522    2565 scope.go:117] "RemoveContainer" containerID="52d3ed1b54f5320fcbc2278051b45736a5018ec7711a332affc52dc88ba30067"
Jul 16 23:17:52.460770 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:17:52.460751    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52d3ed1b54f5320fcbc2278051b45736a5018ec7711a332affc52dc88ba30067\": container with ID starting with 52d3ed1b54f5320fcbc2278051b45736a5018ec7711a332affc52dc88ba30067 not found: ID does not exist" containerID="52d3ed1b54f5320fcbc2278051b45736a5018ec7711a332affc52dc88ba30067"
Jul 16 23:17:52.460821 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:52.460780    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52d3ed1b54f5320fcbc2278051b45736a5018ec7711a332affc52dc88ba30067"} err="failed to get container status \"52d3ed1b54f5320fcbc2278051b45736a5018ec7711a332affc52dc88ba30067\": rpc error: code = NotFound desc = could not find container \"52d3ed1b54f5320fcbc2278051b45736a5018ec7711a332affc52dc88ba30067\": container with ID starting with 52d3ed1b54f5320fcbc2278051b45736a5018ec7711a332affc52dc88ba30067 not found: ID does not exist"
Jul 16 23:17:52.466473 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:52.466451    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"]
Jul 16 23:17:52.470499 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:52.470478    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4b61afbb-openshift-default-ffb5475c9-762h4"]
Jul 16 23:17:53.726490 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:53.726458    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7116503f-e44a-47d1-91e1-dcee065ab312" path="/var/lib/kubelet/pods/7116503f-e44a-47d1-91e1-dcee065ab312/volumes"
Jul 16 23:17:57.720626 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:57.720595    2565 scope.go:117] "RemoveContainer" containerID="5decc1f9a719a957c2226e76a3f5389014e04de18b86a43a6621f06671a25b39"
Jul 16 23:17:58.475954 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:58.475926    2565 generic.go:356] "Generic (PLEG): container finished" podID="aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6" containerID="d74ea7bf9f97b2c40f931f4837fb04c85729f66de254c357858a014b0834a63e" exitCode=6
Jul 16 23:17:58.476105 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:58.475968    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74" event={"ID":"aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6","Type":"ContainerDied","Data":"d74ea7bf9f97b2c40f931f4837fb04c85729f66de254c357858a014b0834a63e"}
Jul 16 23:17:58.476105 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:58.476016    2565 scope.go:117] "RemoveContainer" containerID="5decc1f9a719a957c2226e76a3f5389014e04de18b86a43a6621f06671a25b39"
Jul 16 23:17:58.476327 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:58.476308    2565 scope.go:117] "RemoveContainer" containerID="d74ea7bf9f97b2c40f931f4837fb04c85729f66de254c357858a014b0834a63e"
Jul 16 23:17:58.476553 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:17:58.476532    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74_odh-ai-gateway-infra(aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74" podUID="aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6"
Jul 16 23:17:59.500324 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:59.500290    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74"]
Jul 16 23:17:59.636805 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:59.636777    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74"
Jul 16 23:17:59.787852 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:59.787756    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6-maas-api-service-ca\") pod \"aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6\" (UID: \"aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6\") "
Jul 16 23:17:59.788128 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:59.788103    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6" (UID: "aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:17:59.889001 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:17:59.888966    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:18:00.489187 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:00.489155    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74"
Jul 16 23:18:00.489373 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:00.489154    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74" event={"ID":"aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6","Type":"ContainerDied","Data":"9e8fdf60591a1b3dd32da47d12df618fcf9387f204109a18a458da5a06333057"}
Jul 16 23:18:00.489373 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:00.489325    2565 scope.go:117] "RemoveContainer" containerID="d74ea7bf9f97b2c40f931f4837fb04c85729f66de254c357858a014b0834a63e"
Jul 16 23:18:00.511601 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:00.511571    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74"]
Jul 16 23:18:00.515230 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:00.515205    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-4kv74"]
Jul 16 23:18:01.720906 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:01.720869    2565 scope.go:117] "RemoveContainer" containerID="3837cd48c42049177b9df493fd09384c91c104b1d277bd72c67d2f98e78bc96d"
Jul 16 23:18:01.726915 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:01.726893    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6" path="/var/lib/kubelet/pods/aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6/volumes"
Jul 16 23:18:01.773390 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:01.773312    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr"]
Jul 16 23:18:01.773715 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:01.773701    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6" containerName="revoke-keys"
Jul 16 23:18:01.773768 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:01.773716    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6" containerName="revoke-keys"
Jul 16 23:18:01.773768 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:01.773744    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7116503f-e44a-47d1-91e1-dcee065ab312" containerName="istio-proxy"
Jul 16 23:18:01.773768 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:01.773750    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7116503f-e44a-47d1-91e1-dcee065ab312" containerName="istio-proxy"
Jul 16 23:18:01.773768 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:01.773756    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7116503f-e44a-47d1-91e1-dcee065ab312" containerName="istio-proxy"
Jul 16 23:18:01.773768 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:01.773762    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7116503f-e44a-47d1-91e1-dcee065ab312" containerName="istio-proxy"
Jul 16 23:18:01.773926 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:01.773770    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6" containerName="revoke-keys"
Jul 16 23:18:01.773926 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:01.773775    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6" containerName="revoke-keys"
Jul 16 23:18:01.773926 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:01.773838    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6" containerName="revoke-keys"
Jul 16 23:18:01.773926 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:01.773846    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7116503f-e44a-47d1-91e1-dcee065ab312" containerName="istio-proxy"
Jul 16 23:18:01.773926 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:01.773854    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7116503f-e44a-47d1-91e1-dcee065ab312" containerName="istio-proxy"
Jul 16 23:18:01.773926 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:01.773860    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6" containerName="revoke-keys"
Jul 16 23:18:01.776590 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:01.776568    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr"
Jul 16 23:18:01.785163 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:01.785132    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr"]
Jul 16 23:18:01.909164 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:01.909120    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e85d93-d964-4d5a-992e-3cd806171172-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr\" (UID: \"09e85d93-d964-4d5a-992e-3cd806171172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr"
Jul 16 23:18:02.009926 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:02.009811    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e85d93-d964-4d5a-992e-3cd806171172-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr\" (UID: \"09e85d93-d964-4d5a-992e-3cd806171172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr"
Jul 16 23:18:02.010536 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:02.010509    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e85d93-d964-4d5a-992e-3cd806171172-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr\" (UID: \"09e85d93-d964-4d5a-992e-3cd806171172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr"
Jul 16 23:18:02.088607 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:02.088568    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr"
Jul 16 23:18:02.217330 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:02.217311    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr"]
Jul 16 23:18:02.219966 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:18:02.219936    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e85d93_d964_4d5a_992e_3cd806171172.slice/crio-949edaa469d9a3ffc269fc2626d3f97ea6ae39e1603e07b37e3751cdc7dd4cb3 WatchSource:0}: Error finding container 949edaa469d9a3ffc269fc2626d3f97ea6ae39e1603e07b37e3751cdc7dd4cb3: Status 404 returned error can't find the container with id 949edaa469d9a3ffc269fc2626d3f97ea6ae39e1603e07b37e3751cdc7dd4cb3
Jul 16 23:18:02.502264 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:02.502222    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr" event={"ID":"09e85d93-d964-4d5a-992e-3cd806171172","Type":"ContainerStarted","Data":"949edaa469d9a3ffc269fc2626d3f97ea6ae39e1603e07b37e3751cdc7dd4cb3"}
Jul 16 23:18:02.503878 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:02.503852    2565 generic.go:356] "Generic (PLEG): container finished" podID="3f1744f8-214d-423c-bc1f-418766290cb8" containerID="f42f74605191aef10c38d325c2e1b40d9b62aa216bee23d1cda83c5e063d44ac" exitCode=6
Jul 16 23:18:02.504014 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:02.503927    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5" event={"ID":"3f1744f8-214d-423c-bc1f-418766290cb8","Type":"ContainerDied","Data":"f42f74605191aef10c38d325c2e1b40d9b62aa216bee23d1cda83c5e063d44ac"}
Jul 16 23:18:02.504014 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:02.503983    2565 scope.go:117] "RemoveContainer" containerID="3837cd48c42049177b9df493fd09384c91c104b1d277bd72c67d2f98e78bc96d"
Jul 16 23:18:02.504301 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:02.504285    2565 scope.go:117] "RemoveContainer" containerID="f42f74605191aef10c38d325c2e1b40d9b62aa216bee23d1cda83c5e063d44ac"
Jul 16 23:18:02.504556 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:18:02.504533    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5_odh-ai-gateway-infra(3f1744f8-214d-423c-bc1f-418766290cb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5" podUID="3f1744f8-214d-423c-bc1f-418766290cb8"
Jul 16 23:18:02.969648 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:02.969618    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6"]
Jul 16 23:18:02.970001 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:02.969988    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6" containerName="revoke-keys"
Jul 16 23:18:02.970040 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:02.970002    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6" containerName="revoke-keys"
Jul 16 23:18:02.970075 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:02.970055    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa37c1a3-cbec-4e94-a2ac-33e6a67a32a6" containerName="revoke-keys"
Jul 16 23:18:02.971928 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:02.971912    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6"
Jul 16 23:18:02.979118 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:02.979094    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6"]
Jul 16 23:18:03.119520 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:03.119479    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a27d58-354d-4284-b3e4-2ec5e0c8ae95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-rjjn6\" (UID: \"96a27d58-354d-4284-b3e4-2ec5e0c8ae95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6"
Jul 16 23:18:03.220591 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:03.220557    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a27d58-354d-4284-b3e4-2ec5e0c8ae95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-rjjn6\" (UID: \"96a27d58-354d-4284-b3e4-2ec5e0c8ae95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6"
Jul 16 23:18:03.221179 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:03.221160    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a27d58-354d-4284-b3e4-2ec5e0c8ae95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-rjjn6\" (UID: \"96a27d58-354d-4284-b3e4-2ec5e0c8ae95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6"
Jul 16 23:18:03.306909 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:03.306830    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6"
Jul 16 23:18:03.438098 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:03.438069    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6"]
Jul 16 23:18:03.439157 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:18:03.439132    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a27d58_354d_4284_b3e4_2ec5e0c8ae95.slice/crio-0fa188bd7cc5efe93575e5f9d5fdabe91e526da3ebd1818100c1f1fba9c060be WatchSource:0}: Error finding container 0fa188bd7cc5efe93575e5f9d5fdabe91e526da3ebd1818100c1f1fba9c060be: Status 404 returned error can't find the container with id 0fa188bd7cc5efe93575e5f9d5fdabe91e526da3ebd1818100c1f1fba9c060be
Jul 16 23:18:03.510461 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:03.510429    2565 generic.go:356] "Generic (PLEG): container finished" podID="09e85d93-d964-4d5a-992e-3cd806171172" containerID="d33787a59f432aa21237606f8af1cd9c96ac208e623868daf27bcb53d22c3b7b" exitCode=6
Jul 16 23:18:03.510612 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:03.510509    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr" event={"ID":"09e85d93-d964-4d5a-992e-3cd806171172","Type":"ContainerDied","Data":"d33787a59f432aa21237606f8af1cd9c96ac208e623868daf27bcb53d22c3b7b"}
Jul 16 23:18:03.510721 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:03.510702    2565 scope.go:117] "RemoveContainer" containerID="d33787a59f432aa21237606f8af1cd9c96ac208e623868daf27bcb53d22c3b7b"
Jul 16 23:18:03.511938 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:03.511916    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6" event={"ID":"96a27d58-354d-4284-b3e4-2ec5e0c8ae95","Type":"ContainerStarted","Data":"0fa188bd7cc5efe93575e5f9d5fdabe91e526da3ebd1818100c1f1fba9c060be"}
Jul 16 23:18:03.535181 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:03.535143    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5"]
Jul 16 23:18:03.647493 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:03.647464    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5"
Jul 16 23:18:03.825952 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:03.825922    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1744f8-214d-423c-bc1f-418766290cb8-maas-api-service-ca\") pod \"3f1744f8-214d-423c-bc1f-418766290cb8\" (UID: \"3f1744f8-214d-423c-bc1f-418766290cb8\") "
Jul 16 23:18:03.826329 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:03.826304    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1744f8-214d-423c-bc1f-418766290cb8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f1744f8-214d-423c-bc1f-418766290cb8" (UID: "3f1744f8-214d-423c-bc1f-418766290cb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:18:03.927042 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:03.927010    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1744f8-214d-423c-bc1f-418766290cb8-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:18:04.520816 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:04.520730    2565 generic.go:356] "Generic (PLEG): container finished" podID="09e85d93-d964-4d5a-992e-3cd806171172" containerID="b1fd66faa7b9132fc19291a28dcb10399dcf0be3a32b064ddc84c85673a1314f" exitCode=6
Jul 16 23:18:04.521313 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:04.520817    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr" event={"ID":"09e85d93-d964-4d5a-992e-3cd806171172","Type":"ContainerDied","Data":"b1fd66faa7b9132fc19291a28dcb10399dcf0be3a32b064ddc84c85673a1314f"}
Jul 16 23:18:04.521313 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:04.520885    2565 scope.go:117] "RemoveContainer" containerID="d33787a59f432aa21237606f8af1cd9c96ac208e623868daf27bcb53d22c3b7b"
Jul 16 23:18:04.521313 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:04.521112    2565 scope.go:117] "RemoveContainer" containerID="b1fd66faa7b9132fc19291a28dcb10399dcf0be3a32b064ddc84c85673a1314f"
Jul 16 23:18:04.521473 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:18:04.521360    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr_odh-ai-gateway-infra(09e85d93-d964-4d5a-992e-3cd806171172)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr" podUID="09e85d93-d964-4d5a-992e-3cd806171172"
Jul 16 23:18:04.522395 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:04.522373    2565 generic.go:356] "Generic (PLEG): container finished" podID="96a27d58-354d-4284-b3e4-2ec5e0c8ae95" containerID="769b9adaf3248cc5d5562e1cf86a67055d707d90094c78deedd1df2cce8b598d" exitCode=6
Jul 16 23:18:04.522502 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:04.522446    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6" event={"ID":"96a27d58-354d-4284-b3e4-2ec5e0c8ae95","Type":"ContainerDied","Data":"769b9adaf3248cc5d5562e1cf86a67055d707d90094c78deedd1df2cce8b598d"}
Jul 16 23:18:04.522654 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:04.522636    2565 scope.go:117] "RemoveContainer" containerID="769b9adaf3248cc5d5562e1cf86a67055d707d90094c78deedd1df2cce8b598d"
Jul 16 23:18:04.523681 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:04.523660    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5" event={"ID":"3f1744f8-214d-423c-bc1f-418766290cb8","Type":"ContainerDied","Data":"20f08e6ba7729c3948db674f5e3f2aec52f7bf2bf55e0a4f825370779b38316a"}
Jul 16 23:18:04.523681 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:04.523673    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5"
Jul 16 23:18:04.533826 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:04.533803    2565 scope.go:117] "RemoveContainer" containerID="f42f74605191aef10c38d325c2e1b40d9b62aa216bee23d1cda83c5e063d44ac"
Jul 16 23:18:04.550324 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:04.550300    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5"]
Jul 16 23:18:04.552921 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:04.552900    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-xq7c5"]
Jul 16 23:18:05.532025 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:05.531938    2565 scope.go:117] "RemoveContainer" containerID="b1fd66faa7b9132fc19291a28dcb10399dcf0be3a32b064ddc84c85673a1314f"
Jul 16 23:18:05.532513 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:18:05.532163    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr_odh-ai-gateway-infra(09e85d93-d964-4d5a-992e-3cd806171172)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr" podUID="09e85d93-d964-4d5a-992e-3cd806171172"
Jul 16 23:18:05.533603 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:05.533578    2565 generic.go:356] "Generic (PLEG): container finished" podID="96a27d58-354d-4284-b3e4-2ec5e0c8ae95" containerID="4aa1321c836cdf3a1068c650101ea62357f05a0b99628842fdbff8dc549a9267" exitCode=6
Jul 16 23:18:05.533712 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:05.533612    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6" event={"ID":"96a27d58-354d-4284-b3e4-2ec5e0c8ae95","Type":"ContainerDied","Data":"4aa1321c836cdf3a1068c650101ea62357f05a0b99628842fdbff8dc549a9267"}
Jul 16 23:18:05.533712 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:05.533654    2565 scope.go:117] "RemoveContainer" containerID="769b9adaf3248cc5d5562e1cf86a67055d707d90094c78deedd1df2cce8b598d"
Jul 16 23:18:05.533802 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:05.533774    2565 scope.go:117] "RemoveContainer" containerID="4aa1321c836cdf3a1068c650101ea62357f05a0b99628842fdbff8dc549a9267"
Jul 16 23:18:05.534005 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:18:05.533986    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-rjjn6_odh-ai-gateway-infra(96a27d58-354d-4284-b3e4-2ec5e0c8ae95)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6" podUID="96a27d58-354d-4284-b3e4-2ec5e0c8ae95"
Jul 16 23:18:05.726380 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:05.726349    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f1744f8-214d-423c-bc1f-418766290cb8" path="/var/lib/kubelet/pods/3f1744f8-214d-423c-bc1f-418766290cb8/volumes"
Jul 16 23:18:05.996763 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:05.996726    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"]
Jul 16 23:18:05.997111 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:05.997097    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f1744f8-214d-423c-bc1f-418766290cb8" containerName="revoke-keys"
Jul 16 23:18:05.997166 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:05.997114    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f1744f8-214d-423c-bc1f-418766290cb8" containerName="revoke-keys"
Jul 16 23:18:05.997166 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:05.997139    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f1744f8-214d-423c-bc1f-418766290cb8" containerName="revoke-keys"
Jul 16 23:18:05.997166 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:05.997144    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f1744f8-214d-423c-bc1f-418766290cb8" containerName="revoke-keys"
Jul 16 23:18:05.997276 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:05.997206    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f1744f8-214d-423c-bc1f-418766290cb8" containerName="revoke-keys"
Jul 16 23:18:05.997276 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:05.997215    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f1744f8-214d-423c-bc1f-418766290cb8" containerName="revoke-keys"
Jul 16 23:18:06.001619 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.001602    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.003755 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.003733    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9752734e-openshift-default-dockercfg-xmcvs\""
Jul 16 23:18:06.011813 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.011788    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"]
Jul 16 23:18:06.149278 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.149208    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-credential-socket\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.149465 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.149343    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-envoy\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.149465 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.149379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-workload-socket\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.149465 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.149398    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-podinfo\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.149465 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.149414    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istiod-ca-cert\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.149643 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.149529    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-workload-certs\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.149643 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.149567    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-token\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.149643 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.149586    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57lbn\" (UniqueName: \"kubernetes.io/projected/e8e2713c-4c29-49ea-81bc-97edd584d3fd-kube-api-access-57lbn\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.149643 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.149630    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-data\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.250389 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.250275    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-envoy\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.250389 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.250329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-workload-socket\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.250389 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.250360    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-podinfo\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.250389 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.250381    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istiod-ca-cert\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.250726 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.250454    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-workload-certs\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.250726 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.250494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-token\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.250726 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.250516    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57lbn\" (UniqueName: \"kubernetes.io/projected/e8e2713c-4c29-49ea-81bc-97edd584d3fd-kube-api-access-57lbn\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.250726 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.250548    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-data\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.250726 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.250604    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-credential-socket\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.251059 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.250763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-workload-socket\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.251059 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.250772    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-workload-certs\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.251059 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.250978    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-credential-socket\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.251059 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.251034    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-data\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.251240 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.251216    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istiod-ca-cert\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.252670 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.252644    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-envoy\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.252872 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.252854    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-podinfo\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.258021 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.257993    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-token\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.258127 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.258062    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57lbn\" (UniqueName: \"kubernetes.io/projected/e8e2713c-4c29-49ea-81bc-97edd584d3fd-kube-api-access-57lbn\") pod \"e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") " pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.312742 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.312703    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:06.460945 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.460906    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"]
Jul 16 23:18:06.464628 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:18:06.464601    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e2713c_4c29_49ea_81bc_97edd584d3fd.slice/crio-8248b88865b4e655c456062dbf94e31ce24208feed98d60ccf230b4f25f55861 WatchSource:0}: Error finding container 8248b88865b4e655c456062dbf94e31ce24208feed98d60ccf230b4f25f55861: Status 404 returned error can't find the container with id 8248b88865b4e655c456062dbf94e31ce24208feed98d60ccf230b4f25f55861
Jul 16 23:18:06.466964 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.466926    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:18:06.467049 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.467009    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:18:06.467087 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.467052    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:18:06.540902 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.540874    2565 scope.go:117] "RemoveContainer" containerID="4aa1321c836cdf3a1068c650101ea62357f05a0b99628842fdbff8dc549a9267"
Jul 16 23:18:06.541296 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:18:06.541090    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-rjjn6_odh-ai-gateway-infra(96a27d58-354d-4284-b3e4-2ec5e0c8ae95)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6" podUID="96a27d58-354d-4284-b3e4-2ec5e0c8ae95"
Jul 16 23:18:06.542308 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.542282    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg" event={"ID":"e8e2713c-4c29-49ea-81bc-97edd584d3fd","Type":"ContainerStarted","Data":"8c8d44cd8b33610c26be61cfa6d3e63d0dc2928857b77a42e1ffbd8282f8af1a"}
Jul 16 23:18:06.542427 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.542344    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg" event={"ID":"e8e2713c-4c29-49ea-81bc-97edd584d3fd","Type":"ContainerStarted","Data":"8248b88865b4e655c456062dbf94e31ce24208feed98d60ccf230b4f25f55861"}
Jul 16 23:18:06.575711 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:06.574078    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg" podStartSLOduration=1.574055591 podStartE2EDuration="1.574055591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:18:06.570641909 +0000 UTC m=+2697.319125980" watchObservedRunningTime="2026-07-16 23:18:06.574055591 +0000 UTC m=+2697.322539663"
Jul 16 23:18:07.313577 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:07.313539    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:07.318691 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:07.318666    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:07.547325 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:07.547295    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:07.548225 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:07.548201    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:08.545739 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:08.545708    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"]
Jul 16 23:18:08.546126 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:08.546113    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f1744f8-214d-423c-bc1f-418766290cb8" containerName="revoke-keys"
Jul 16 23:18:08.546179 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:08.546128    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f1744f8-214d-423c-bc1f-418766290cb8" containerName="revoke-keys"
Jul 16 23:18:08.546234 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:08.546222    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f1744f8-214d-423c-bc1f-418766290cb8" containerName="revoke-keys"
Jul 16 23:18:08.549749 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:08.549719    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"
Jul 16 23:18:08.551891 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:08.551867    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9752734e-serving-cert\""
Jul 16 23:18:08.562264 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:08.562224    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"]
Jul 16 23:18:08.571951 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:08.571927    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-maas-api-tls\") pod \"maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9\" (UID: \"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"
Jul 16 23:18:08.572080 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:08.571982    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx827\" (UniqueName: \"kubernetes.io/projected/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-kube-api-access-sx827\") pod \"maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9\" (UID: \"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"
Jul 16 23:18:08.572080 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:08.572007    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-openshift-service-ca\") pod \"maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9\" (UID: \"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"
Jul 16 23:18:08.672905 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:08.672848    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-maas-api-tls\") pod \"maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9\" (UID: \"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"
Jul 16 23:18:08.673111 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:08.672991    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx827\" (UniqueName: \"kubernetes.io/projected/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-kube-api-access-sx827\") pod \"maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9\" (UID: \"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"
Jul 16 23:18:08.673111 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:18:08.673024    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-serving-cert: secret "maas-api-e2e-mt-9752734e-serving-cert" not found
Jul 16 23:18:08.673111 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:18:08.673106    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-maas-api-tls podName:443c802b-22b2-41f6-b0aa-86e8a2b4cbbf nodeName:}" failed. No retries permitted until 2026-07-16 23:18:09.17308069 +0000 UTC m=+2699.921564753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-maas-api-tls") pod "maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9" (UID: "443c802b-22b2-41f6-b0aa-86e8a2b4cbbf") : secret "maas-api-e2e-mt-9752734e-serving-cert" not found
Jul 16 23:18:08.673392 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:08.673029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-openshift-service-ca\") pod \"maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9\" (UID: \"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"
Jul 16 23:18:08.673853 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:08.673821    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-openshift-service-ca\") pod \"maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9\" (UID: \"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"
Jul 16 23:18:08.682041 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:08.682005    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx827\" (UniqueName: \"kubernetes.io/projected/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-kube-api-access-sx827\") pod \"maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9\" (UID: \"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"
Jul 16 23:18:09.177615 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:09.177581    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-maas-api-tls\") pod \"maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9\" (UID: \"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"
Jul 16 23:18:09.180103 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:09.180072    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-maas-api-tls\") pod \"maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9\" (UID: \"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"
Jul 16 23:18:09.462861 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:09.462771    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"
Jul 16 23:18:09.598486 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:09.598464    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"]
Jul 16 23:18:09.601023 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:18:09.600993    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443c802b_22b2_41f6_b0aa_86e8a2b4cbbf.slice/crio-089a5b1c1d2cf6ad836c35078af5f5f4cf64a916f2f81db27a70c96171f6624d WatchSource:0}: Error finding container 089a5b1c1d2cf6ad836c35078af5f5f4cf64a916f2f81db27a70c96171f6624d: Status 404 returned error can't find the container with id 089a5b1c1d2cf6ad836c35078af5f5f4cf64a916f2f81db27a70c96171f6624d
Jul 16 23:18:09.602400 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:09.602382    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 23:18:10.564775 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:10.564741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9" event={"ID":"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf","Type":"ContainerStarted","Data":"c2ee94c362daa482ced6c1c013654525b41a303d1f47d056ee4f6533a78c42f0"}
Jul 16 23:18:10.564775 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:10.564791    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9" event={"ID":"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf","Type":"ContainerStarted","Data":"089a5b1c1d2cf6ad836c35078af5f5f4cf64a916f2f81db27a70c96171f6624d"}
Jul 16 23:18:10.565013 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:10.564885    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"
Jul 16 23:18:10.580914 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:10.580862    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9" podStartSLOduration=2.228036846 podStartE2EDuration="2.580846593s" totalImagesPullingTime="352.809747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:18:10.579923976 +0000 UTC m=+2701.328408046" watchObservedRunningTime="2026-07-16 23:18:10.580846593 +0000 UTC m=+2701.329330706"
Jul 16 23:18:16.575925 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:16.575894    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"
Jul 16 23:18:16.720561 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:16.720524    2565 scope.go:117] "RemoveContainer" containerID="4aa1321c836cdf3a1068c650101ea62357f05a0b99628842fdbff8dc549a9267"
Jul 16 23:18:16.720771 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:16.720611    2565 scope.go:117] "RemoveContainer" containerID="b1fd66faa7b9132fc19291a28dcb10399dcf0be3a32b064ddc84c85673a1314f"
Jul 16 23:18:17.604747 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:17.604710    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr" event={"ID":"09e85d93-d964-4d5a-992e-3cd806171172","Type":"ContainerStarted","Data":"63719962bb2a21708c74a1415801d50e3c7d0ee452f9b292047b3e2f8fd3d849"}
Jul 16 23:18:17.606660 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:17.606636    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6" event={"ID":"96a27d58-354d-4284-b3e4-2ec5e0c8ae95","Type":"ContainerStarted","Data":"8f743d349f9cb20697a2ad7cee4b8501fd346617fbb8db3cff673471d02c734f"}
Jul 16 23:18:17.620724 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:17.620665    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr" podStartSLOduration=16.620645144 podStartE2EDuration="16.620645144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:18:17.618120544 +0000 UTC m=+2708.366604616" watchObservedRunningTime="2026-07-16 23:18:17.620645144 +0000 UTC m=+2708.369129218"
Jul 16 23:18:18.613199 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:18.613160    2565 generic.go:356] "Generic (PLEG): container finished" podID="09e85d93-d964-4d5a-992e-3cd806171172" containerID="63719962bb2a21708c74a1415801d50e3c7d0ee452f9b292047b3e2f8fd3d849" exitCode=6
Jul 16 23:18:18.613637 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:18.613234    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr" event={"ID":"09e85d93-d964-4d5a-992e-3cd806171172","Type":"ContainerDied","Data":"63719962bb2a21708c74a1415801d50e3c7d0ee452f9b292047b3e2f8fd3d849"}
Jul 16 23:18:18.613637 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:18.613350    2565 scope.go:117] "RemoveContainer" containerID="b1fd66faa7b9132fc19291a28dcb10399dcf0be3a32b064ddc84c85673a1314f"
Jul 16 23:18:18.613637 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:18.613612    2565 scope.go:117] "RemoveContainer" containerID="63719962bb2a21708c74a1415801d50e3c7d0ee452f9b292047b3e2f8fd3d849"
Jul 16 23:18:18.613915 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:18:18.613890    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr_odh-ai-gateway-infra(09e85d93-d964-4d5a-992e-3cd806171172)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr" podUID="09e85d93-d964-4d5a-992e-3cd806171172"
Jul 16 23:18:18.615201 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:18.615181    2565 generic.go:356] "Generic (PLEG): container finished" podID="96a27d58-354d-4284-b3e4-2ec5e0c8ae95" containerID="8f743d349f9cb20697a2ad7cee4b8501fd346617fbb8db3cff673471d02c734f" exitCode=6
Jul 16 23:18:18.615309 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:18.615283    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6" event={"ID":"96a27d58-354d-4284-b3e4-2ec5e0c8ae95","Type":"ContainerDied","Data":"8f743d349f9cb20697a2ad7cee4b8501fd346617fbb8db3cff673471d02c734f"}
Jul 16 23:18:18.615557 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:18.615540    2565 scope.go:117] "RemoveContainer" containerID="8f743d349f9cb20697a2ad7cee4b8501fd346617fbb8db3cff673471d02c734f"
Jul 16 23:18:18.615773 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:18:18.615751    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-rjjn6_odh-ai-gateway-infra(96a27d58-354d-4284-b3e4-2ec5e0c8ae95)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6" podUID="96a27d58-354d-4284-b3e4-2ec5e0c8ae95"
Jul 16 23:18:18.624379 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:18.624362    2565 scope.go:117] "RemoveContainer" containerID="4aa1321c836cdf3a1068c650101ea62357f05a0b99628842fdbff8dc549a9267"
Jul 16 23:18:18.629959 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:18.629914    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6" podStartSLOduration=16.629898536 podStartE2EDuration="16.629898536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:18:17.631199218 +0000 UTC m=+2708.379683289" watchObservedRunningTime="2026-07-16 23:18:18.629898536 +0000 UTC m=+2709.378382607"
Jul 16 23:18:18.631758 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:18.631738    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr"]
Jul 16 23:18:18.646333 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:18.646292    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6"]
Jul 16 23:18:19.798465 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:19.798438    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr"
Jul 16 23:18:19.824775 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:19.824751    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6"
Jul 16 23:18:19.888484 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:19.888394    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e85d93-d964-4d5a-992e-3cd806171172-maas-api-service-ca\") pod \"09e85d93-d964-4d5a-992e-3cd806171172\" (UID: \"09e85d93-d964-4d5a-992e-3cd806171172\") "
Jul 16 23:18:19.888484 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:19.888449    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a27d58-354d-4284-b3e4-2ec5e0c8ae95-maas-api-service-ca\") pod \"96a27d58-354d-4284-b3e4-2ec5e0c8ae95\" (UID: \"96a27d58-354d-4284-b3e4-2ec5e0c8ae95\") "
Jul 16 23:18:19.888824 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:19.888793    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a27d58-354d-4284-b3e4-2ec5e0c8ae95-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96a27d58-354d-4284-b3e4-2ec5e0c8ae95" (UID: "96a27d58-354d-4284-b3e4-2ec5e0c8ae95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:18:19.888824 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:19.888813    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e85d93-d964-4d5a-992e-3cd806171172-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "09e85d93-d964-4d5a-992e-3cd806171172" (UID: "09e85d93-d964-4d5a-992e-3cd806171172"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:18:19.989512 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:19.989480    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a27d58-354d-4284-b3e4-2ec5e0c8ae95-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:18:19.989512 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:19.989511    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e85d93-d964-4d5a-992e-3cd806171172-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:18:20.629068 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:20.629043    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr"
Jul 16 23:18:20.629284 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:20.629042    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr" event={"ID":"09e85d93-d964-4d5a-992e-3cd806171172","Type":"ContainerDied","Data":"949edaa469d9a3ffc269fc2626d3f97ea6ae39e1603e07b37e3751cdc7dd4cb3"}
Jul 16 23:18:20.629284 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:20.629170    2565 scope.go:117] "RemoveContainer" containerID="63719962bb2a21708c74a1415801d50e3c7d0ee452f9b292047b3e2f8fd3d849"
Jul 16 23:18:20.630547 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:20.630422    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6"
Jul 16 23:18:20.630547 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:20.630443    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6" event={"ID":"96a27d58-354d-4284-b3e4-2ec5e0c8ae95","Type":"ContainerDied","Data":"0fa188bd7cc5efe93575e5f9d5fdabe91e526da3ebd1818100c1f1fba9c060be"}
Jul 16 23:18:20.640889 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:20.640872    2565 scope.go:117] "RemoveContainer" containerID="8f743d349f9cb20697a2ad7cee4b8501fd346617fbb8db3cff673471d02c734f"
Jul 16 23:18:20.677359 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:20.677331    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr"]
Jul 16 23:18:20.679827 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:20.679801    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-cctqr"]
Jul 16 23:18:20.688356 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:20.688325    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6"]
Jul 16 23:18:20.693541 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:20.693518    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-rjjn6"]
Jul 16 23:18:21.728410 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:21.728367    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09e85d93-d964-4d5a-992e-3cd806171172" path="/var/lib/kubelet/pods/09e85d93-d964-4d5a-992e-3cd806171172/volumes"
Jul 16 23:18:21.729127 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:21.729106    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96a27d58-354d-4284-b3e4-2ec5e0c8ae95" path="/var/lib/kubelet/pods/96a27d58-354d-4284-b3e4-2ec5e0c8ae95/volumes"
Jul 16 23:18:23.577653 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:23.576869    2565 patch_prober.go:28] interesting pod/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 16 23:18:23.577653 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:23.576939    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg" podUID="e8e2713c-4c29-49ea-81bc-97edd584d3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 16 23:18:23.649209 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:23.649177    2565 generic.go:356] "Generic (PLEG): container finished" podID="e8e2713c-4c29-49ea-81bc-97edd584d3fd" containerID="8c8d44cd8b33610c26be61cfa6d3e63d0dc2928857b77a42e1ffbd8282f8af1a" exitCode=0
Jul 16 23:18:23.649395 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:23.649282    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg" event={"ID":"e8e2713c-4c29-49ea-81bc-97edd584d3fd","Type":"ContainerDied","Data":"8c8d44cd8b33610c26be61cfa6d3e63d0dc2928857b77a42e1ffbd8282f8af1a"}
Jul 16 23:18:23.649702 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:23.649686    2565 scope.go:117] "RemoveContainer" containerID="8c8d44cd8b33610c26be61cfa6d3e63d0dc2928857b77a42e1ffbd8282f8af1a"
Jul 16 23:18:23.650796 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:23.650764    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:18:23.650871 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:23.650839    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:18:23.650915 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:23.650874    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:18:24.660109 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:24.660067    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg" event={"ID":"e8e2713c-4c29-49ea-81bc-97edd584d3fd","Type":"ContainerStarted","Data":"907df6805fb1c6b9df1446c43f095ff581cf68d36fd2e9c0e1170b1a067032ce"}
Jul 16 23:18:25.313033 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:25.312994    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:25.313309 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:25.313285    2565 patch_prober.go:28] interesting pod/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Jul 16 23:18:25.313373 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:25.313350    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg" podUID="e8e2713c-4c29-49ea-81bc-97edd584d3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Jul 16 23:18:26.313601 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:26.313515    2565 patch_prober.go:28] interesting pod/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Jul 16 23:18:26.313601 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:26.313570    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg" podUID="e8e2713c-4c29-49ea-81bc-97edd584d3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Jul 16 23:18:27.313509 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:27.313470    2565 patch_prober.go:28] interesting pod/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Jul 16 23:18:27.313697 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:27.313527    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg" podUID="e8e2713c-4c29-49ea-81bc-97edd584d3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Jul 16 23:18:28.313902 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:28.313861    2565 patch_prober.go:28] interesting pod/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Jul 16 23:18:28.314473 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:28.313945    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg" podUID="e8e2713c-4c29-49ea-81bc-97edd584d3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Jul 16 23:18:29.317204 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:29.317170    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:29.317663 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:29.317561    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:29.318194 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:29.318176    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:18:37.944152 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:37.944115    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n"]
Jul 16 23:18:37.944671 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:37.944504    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e85d93-d964-4d5a-992e-3cd806171172" containerName="revoke-keys"
Jul 16 23:18:37.944671 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:37.944516    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e85d93-d964-4d5a-992e-3cd806171172" containerName="revoke-keys"
Jul 16 23:18:37.944671 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:37.944526    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e85d93-d964-4d5a-992e-3cd806171172" containerName="revoke-keys"
Jul 16 23:18:37.944671 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:37.944532    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e85d93-d964-4d5a-992e-3cd806171172" containerName="revoke-keys"
Jul 16 23:18:37.944671 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:37.944539    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a27d58-354d-4284-b3e4-2ec5e0c8ae95" containerName="revoke-keys"
Jul 16 23:18:37.944671 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:37.944545    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a27d58-354d-4284-b3e4-2ec5e0c8ae95" containerName="revoke-keys"
Jul 16 23:18:37.944671 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:37.944552    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a27d58-354d-4284-b3e4-2ec5e0c8ae95" containerName="revoke-keys"
Jul 16 23:18:37.944671 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:37.944557    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a27d58-354d-4284-b3e4-2ec5e0c8ae95" containerName="revoke-keys"
Jul 16 23:18:37.944671 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:37.944626    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a27d58-354d-4284-b3e4-2ec5e0c8ae95" containerName="revoke-keys"
Jul 16 23:18:37.944671 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:37.944635    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="09e85d93-d964-4d5a-992e-3cd806171172" containerName="revoke-keys"
Jul 16 23:18:37.944671 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:37.944640    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="09e85d93-d964-4d5a-992e-3cd806171172" containerName="revoke-keys"
Jul 16 23:18:37.944671 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:37.944645    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a27d58-354d-4284-b3e4-2ec5e0c8ae95" containerName="revoke-keys"
Jul 16 23:18:37.944671 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:37.944650    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="09e85d93-d964-4d5a-992e-3cd806171172" containerName="revoke-keys"
Jul 16 23:18:37.944671 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:37.944655    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a27d58-354d-4284-b3e4-2ec5e0c8ae95" containerName="revoke-keys"
Jul 16 23:18:37.947877 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:37.947861    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n"
Jul 16 23:18:37.950074 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:37.950053    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:18:37.954066 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:37.954023    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n"]
Jul 16 23:18:38.066225 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:38.066189    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/746bbe78-e7e7-431d-afc7-214a81973926-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n\" (UID: \"746bbe78-e7e7-431d-afc7-214a81973926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n"
Jul 16 23:18:38.166970 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:38.166930    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/746bbe78-e7e7-431d-afc7-214a81973926-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n\" (UID: \"746bbe78-e7e7-431d-afc7-214a81973926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n"
Jul 16 23:18:38.167669 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:38.167643    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/746bbe78-e7e7-431d-afc7-214a81973926-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n\" (UID: \"746bbe78-e7e7-431d-afc7-214a81973926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n"
Jul 16 23:18:38.259165 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:38.259084    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n"
Jul 16 23:18:38.392390 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:38.392366    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n"]
Jul 16 23:18:38.394784 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:18:38.394750    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746bbe78_e7e7_431d_afc7_214a81973926.slice/crio-0660b06194bbdf6f57d6c433e9bf8265c9e31f227c883c7cc59d19f6334e3671 WatchSource:0}: Error finding container 0660b06194bbdf6f57d6c433e9bf8265c9e31f227c883c7cc59d19f6334e3671: Status 404 returned error can't find the container with id 0660b06194bbdf6f57d6c433e9bf8265c9e31f227c883c7cc59d19f6334e3671
Jul 16 23:18:38.731743 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:38.731700    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n" event={"ID":"746bbe78-e7e7-431d-afc7-214a81973926","Type":"ContainerStarted","Data":"d8e650e05470d9a33d0275d630e5ba484f93d59ea1aa62f1d37d677327d2256d"}
Jul 16 23:18:38.731996 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:38.731789    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n" event={"ID":"746bbe78-e7e7-431d-afc7-214a81973926","Type":"ContainerStarted","Data":"0660b06194bbdf6f57d6c433e9bf8265c9e31f227c883c7cc59d19f6334e3671"}
Jul 16 23:18:38.758112 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:38.758056    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n" podStartSLOduration=1.758041832 podStartE2EDuration="1.758041832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:18:38.755293989 +0000 UTC m=+2729.503778061" watchObservedRunningTime="2026-07-16 23:18:38.758041832 +0000 UTC m=+2729.506525903"
Jul 16 23:18:39.737076 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:39.737042    2565 generic.go:356] "Generic (PLEG): container finished" podID="746bbe78-e7e7-431d-afc7-214a81973926" containerID="d8e650e05470d9a33d0275d630e5ba484f93d59ea1aa62f1d37d677327d2256d" exitCode=6
Jul 16 23:18:39.737505 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:39.737118    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n" event={"ID":"746bbe78-e7e7-431d-afc7-214a81973926","Type":"ContainerDied","Data":"d8e650e05470d9a33d0275d630e5ba484f93d59ea1aa62f1d37d677327d2256d"}
Jul 16 23:18:39.737505 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:39.737300    2565 scope.go:117] "RemoveContainer" containerID="d8e650e05470d9a33d0275d630e5ba484f93d59ea1aa62f1d37d677327d2256d"
Jul 16 23:18:40.743460 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:40.743424    2565 generic.go:356] "Generic (PLEG): container finished" podID="746bbe78-e7e7-431d-afc7-214a81973926" containerID="48e3d778359a48a2ed95090ba4c3e02b55f0b4c6912fc357b361c2e82c66a99e" exitCode=6
Jul 16 23:18:40.743932 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:40.743501    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n" event={"ID":"746bbe78-e7e7-431d-afc7-214a81973926","Type":"ContainerDied","Data":"48e3d778359a48a2ed95090ba4c3e02b55f0b4c6912fc357b361c2e82c66a99e"}
Jul 16 23:18:40.743932 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:40.743559    2565 scope.go:117] "RemoveContainer" containerID="d8e650e05470d9a33d0275d630e5ba484f93d59ea1aa62f1d37d677327d2256d"
Jul 16 23:18:40.743932 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:40.743714    2565 scope.go:117] "RemoveContainer" containerID="48e3d778359a48a2ed95090ba4c3e02b55f0b4c6912fc357b361c2e82c66a99e"
Jul 16 23:18:40.744116 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:18:40.743960    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n_odh-ai-gateway-infra(746bbe78-e7e7-431d-afc7-214a81973926)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n" podUID="746bbe78-e7e7-431d-afc7-214a81973926"
Jul 16 23:18:41.749531 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:41.749501    2565 scope.go:117] "RemoveContainer" containerID="48e3d778359a48a2ed95090ba4c3e02b55f0b4c6912fc357b361c2e82c66a99e"
Jul 16 23:18:41.749937 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:18:41.749721    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n_odh-ai-gateway-infra(746bbe78-e7e7-431d-afc7-214a81973926)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n" podUID="746bbe78-e7e7-431d-afc7-214a81973926"
Jul 16 23:18:45.897360 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:45.897324    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9752734e-js9t8"]
Jul 16 23:18:45.897900 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:45.897882    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e85d93-d964-4d5a-992e-3cd806171172" containerName="revoke-keys"
Jul 16 23:18:45.897985 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:45.897903    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e85d93-d964-4d5a-992e-3cd806171172" containerName="revoke-keys"
Jul 16 23:18:45.897985 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:45.897916    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a27d58-354d-4284-b3e4-2ec5e0c8ae95" containerName="revoke-keys"
Jul 16 23:18:45.897985 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:45.897924    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a27d58-354d-4284-b3e4-2ec5e0c8ae95" containerName="revoke-keys"
Jul 16 23:18:45.902392 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:45.902371    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9752734e-js9t8"
Jul 16 23:18:45.907504 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:45.907477    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9752734e-js9t8"]
Jul 16 23:18:45.946190 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:45.946152    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25aa5168-250d-4b3e-9de8-72282b82a95a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9752734e-js9t8\" (UID: \"25aa5168-250d-4b3e-9de8-72282b82a95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9752734e-js9t8"
Jul 16 23:18:46.047049 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:46.047003    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25aa5168-250d-4b3e-9de8-72282b82a95a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9752734e-js9t8\" (UID: \"25aa5168-250d-4b3e-9de8-72282b82a95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9752734e-js9t8"
Jul 16 23:18:46.047675 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:46.047655    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25aa5168-250d-4b3e-9de8-72282b82a95a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9752734e-js9t8\" (UID: \"25aa5168-250d-4b3e-9de8-72282b82a95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9752734e-js9t8"
Jul 16 23:18:46.214168 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:46.214056    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9752734e-js9t8"
Jul 16 23:18:46.367054 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:46.366865    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9752734e-js9t8"]
Jul 16 23:18:46.369786 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:18:46.369758    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25aa5168_250d_4b3e_9de8_72282b82a95a.slice/crio-e565f447f6f592726c12173bbe67bcde0adb7ae697ae10e65dff8d59baa45268 WatchSource:0}: Error finding container e565f447f6f592726c12173bbe67bcde0adb7ae697ae10e65dff8d59baa45268: Status 404 returned error can't find the container with id e565f447f6f592726c12173bbe67bcde0adb7ae697ae10e65dff8d59baa45268
Jul 16 23:18:46.774347 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:46.774310    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9752734e-js9t8" event={"ID":"25aa5168-250d-4b3e-9de8-72282b82a95a","Type":"ContainerStarted","Data":"42c270c53d84c5285d8f178d9595b98e8e3521e11a50a7da707a61976f5ac729"}
Jul 16 23:18:46.774539 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:46.774367    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9752734e-js9t8" event={"ID":"25aa5168-250d-4b3e-9de8-72282b82a95a","Type":"ContainerStarted","Data":"e565f447f6f592726c12173bbe67bcde0adb7ae697ae10e65dff8d59baa45268"}
Jul 16 23:18:46.792228 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:46.792182    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9752734e-js9t8" podStartSLOduration=1.792167909 podStartE2EDuration="1.792167909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:18:46.78968043 +0000 UTC m=+2737.538164500" watchObservedRunningTime="2026-07-16 23:18:46.792167909 +0000 UTC m=+2737.540651980"
Jul 16 23:18:47.780014 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:47.779984    2565 generic.go:356] "Generic (PLEG): container finished" podID="25aa5168-250d-4b3e-9de8-72282b82a95a" containerID="42c270c53d84c5285d8f178d9595b98e8e3521e11a50a7da707a61976f5ac729" exitCode=0
Jul 16 23:18:47.780465 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:47.780071    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9752734e-js9t8" event={"ID":"25aa5168-250d-4b3e-9de8-72282b82a95a","Type":"ContainerDied","Data":"42c270c53d84c5285d8f178d9595b98e8e3521e11a50a7da707a61976f5ac729"}
Jul 16 23:18:48.918531 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:48.918503    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9752734e-js9t8"
Jul 16 23:18:48.972052 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:48.972018    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25aa5168-250d-4b3e-9de8-72282b82a95a-maas-api-service-ca\") pod \"25aa5168-250d-4b3e-9de8-72282b82a95a\" (UID: \"25aa5168-250d-4b3e-9de8-72282b82a95a\") "
Jul 16 23:18:48.972421 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:48.972395    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25aa5168-250d-4b3e-9de8-72282b82a95a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "25aa5168-250d-4b3e-9de8-72282b82a95a" (UID: "25aa5168-250d-4b3e-9de8-72282b82a95a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:18:49.073618 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:49.073535    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25aa5168-250d-4b3e-9de8-72282b82a95a-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:18:49.791397 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:49.791358    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9752734e-js9t8" event={"ID":"25aa5168-250d-4b3e-9de8-72282b82a95a","Type":"ContainerDied","Data":"e565f447f6f592726c12173bbe67bcde0adb7ae697ae10e65dff8d59baa45268"}
Jul 16 23:18:49.791591 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:49.791412    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e565f447f6f592726c12173bbe67bcde0adb7ae697ae10e65dff8d59baa45268"
Jul 16 23:18:49.791591 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:49.791374    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9752734e-js9t8"
Jul 16 23:18:55.720349 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:55.720305    2565 scope.go:117] "RemoveContainer" containerID="48e3d778359a48a2ed95090ba4c3e02b55f0b4c6912fc357b361c2e82c66a99e"
Jul 16 23:18:56.314437 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.314332    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"]
Jul 16 23:18:56.314649 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.314582    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9" podUID="443c802b-22b2-41f6-b0aa-86e8a2b4cbbf" containerName="maas-api" containerID="cri-o://c2ee94c362daa482ced6c1c013654525b41a303d1f47d056ee4f6533a78c42f0" gracePeriod=30
Jul 16 23:18:56.575375 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.575355    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"
Jul 16 23:18:56.640688 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.640646    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-openshift-service-ca\") pod \"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf\" (UID: \"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf\") "
Jul 16 23:18:56.640895 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.640736    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-maas-api-tls\") pod \"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf\" (UID: \"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf\") "
Jul 16 23:18:56.640895 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.640817    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx827\" (UniqueName: \"kubernetes.io/projected/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-kube-api-access-sx827\") pod \"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf\" (UID: \"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf\") "
Jul 16 23:18:56.641136 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.641108    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "443c802b-22b2-41f6-b0aa-86e8a2b4cbbf" (UID: "443c802b-22b2-41f6-b0aa-86e8a2b4cbbf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:18:56.642961 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.642937    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-kube-api-access-sx827" (OuterVolumeSpecName: "kube-api-access-sx827") pod "443c802b-22b2-41f6-b0aa-86e8a2b4cbbf" (UID: "443c802b-22b2-41f6-b0aa-86e8a2b4cbbf"). InnerVolumeSpecName "kube-api-access-sx827". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:18:56.643052 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.642989    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "443c802b-22b2-41f6-b0aa-86e8a2b4cbbf" (UID: "443c802b-22b2-41f6-b0aa-86e8a2b4cbbf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:18:56.742517 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.742485    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx827\" (UniqueName: \"kubernetes.io/projected/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-kube-api-access-sx827\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:18:56.742517 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.742517    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:18:56.742912 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.742527    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:18:56.833994 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.833900    2565 generic.go:356] "Generic (PLEG): container finished" podID="443c802b-22b2-41f6-b0aa-86e8a2b4cbbf" containerID="c2ee94c362daa482ced6c1c013654525b41a303d1f47d056ee4f6533a78c42f0" exitCode=0
Jul 16 23:18:56.833994 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.833945    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"
Jul 16 23:18:56.834207 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.833988    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9" event={"ID":"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf","Type":"ContainerDied","Data":"c2ee94c362daa482ced6c1c013654525b41a303d1f47d056ee4f6533a78c42f0"}
Jul 16 23:18:56.834207 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.834046    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9" event={"ID":"443c802b-22b2-41f6-b0aa-86e8a2b4cbbf","Type":"ContainerDied","Data":"089a5b1c1d2cf6ad836c35078af5f5f4cf64a916f2f81db27a70c96171f6624d"}
Jul 16 23:18:56.834207 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.834068    2565 scope.go:117] "RemoveContainer" containerID="c2ee94c362daa482ced6c1c013654525b41a303d1f47d056ee4f6533a78c42f0"
Jul 16 23:18:56.835969 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.835948    2565 generic.go:356] "Generic (PLEG): container finished" podID="746bbe78-e7e7-431d-afc7-214a81973926" containerID="8d761f9a2ef512897a06054eff7e73ffc4e76fe6b0cef37954d6b85982801269" exitCode=6
Jul 16 23:18:56.836064 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.836015    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n" event={"ID":"746bbe78-e7e7-431d-afc7-214a81973926","Type":"ContainerDied","Data":"8d761f9a2ef512897a06054eff7e73ffc4e76fe6b0cef37954d6b85982801269"}
Jul 16 23:18:56.836330 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.836309    2565 scope.go:117] "RemoveContainer" containerID="8d761f9a2ef512897a06054eff7e73ffc4e76fe6b0cef37954d6b85982801269"
Jul 16 23:18:56.836533 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:18:56.836517    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n_odh-ai-gateway-infra(746bbe78-e7e7-431d-afc7-214a81973926)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n" podUID="746bbe78-e7e7-431d-afc7-214a81973926"
Jul 16 23:18:56.844372 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.844355    2565 scope.go:117] "RemoveContainer" containerID="c2ee94c362daa482ced6c1c013654525b41a303d1f47d056ee4f6533a78c42f0"
Jul 16 23:18:56.844662 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:18:56.844645    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ee94c362daa482ced6c1c013654525b41a303d1f47d056ee4f6533a78c42f0\": container with ID starting with c2ee94c362daa482ced6c1c013654525b41a303d1f47d056ee4f6533a78c42f0 not found: ID does not exist" containerID="c2ee94c362daa482ced6c1c013654525b41a303d1f47d056ee4f6533a78c42f0"
Jul 16 23:18:56.844715 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.844670    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ee94c362daa482ced6c1c013654525b41a303d1f47d056ee4f6533a78c42f0"} err="failed to get container status \"c2ee94c362daa482ced6c1c013654525b41a303d1f47d056ee4f6533a78c42f0\": rpc error: code = NotFound desc = could not find container \"c2ee94c362daa482ced6c1c013654525b41a303d1f47d056ee4f6533a78c42f0\": container with ID starting with c2ee94c362daa482ced6c1c013654525b41a303d1f47d056ee4f6533a78c42f0 not found: ID does not exist"
Jul 16 23:18:56.844715 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.844688    2565 scope.go:117] "RemoveContainer" containerID="48e3d778359a48a2ed95090ba4c3e02b55f0b4c6912fc357b361c2e82c66a99e"
Jul 16 23:18:56.863999 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.863963    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"]
Jul 16 23:18:56.867533 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:56.867494    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9"]
Jul 16 23:18:57.726156 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:57.726121    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="443c802b-22b2-41f6-b0aa-86e8a2b4cbbf" path="/var/lib/kubelet/pods/443c802b-22b2-41f6-b0aa-86e8a2b4cbbf/volumes"
Jul 16 23:18:57.862460 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:57.862421    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n"]
Jul 16 23:18:58.004867 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:58.004837    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n"
Jul 16 23:18:58.054554 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:58.054510    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/746bbe78-e7e7-431d-afc7-214a81973926-maas-api-service-ca\") pod \"746bbe78-e7e7-431d-afc7-214a81973926\" (UID: \"746bbe78-e7e7-431d-afc7-214a81973926\") "
Jul 16 23:18:58.054864 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:58.054840    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746bbe78-e7e7-431d-afc7-214a81973926-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "746bbe78-e7e7-431d-afc7-214a81973926" (UID: "746bbe78-e7e7-431d-afc7-214a81973926"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:18:58.155502 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:58.155451    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/746bbe78-e7e7-431d-afc7-214a81973926-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:18:58.850537 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:58.850509    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n"
Jul 16 23:18:58.850729 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:58.850502    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n" event={"ID":"746bbe78-e7e7-431d-afc7-214a81973926","Type":"ContainerDied","Data":"0660b06194bbdf6f57d6c433e9bf8265c9e31f227c883c7cc59d19f6334e3671"}
Jul 16 23:18:58.850729 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:58.850647    2565 scope.go:117] "RemoveContainer" containerID="8d761f9a2ef512897a06054eff7e73ffc4e76fe6b0cef37954d6b85982801269"
Jul 16 23:18:58.875317 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:58.875279    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n"]
Jul 16 23:18:58.880160 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:58.880133    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-jn54n"]
Jul 16 23:18:59.572046 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:59.572002    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9752734e-84bc966fb4-zg6h9" podUID="443c802b-22b2-41f6-b0aa-86e8a2b4cbbf" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.59:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 16 23:18:59.726336 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:18:59.726303    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="746bbe78-e7e7-431d-afc7-214a81973926" path="/var/lib/kubelet/pods/746bbe78-e7e7-431d-afc7-214a81973926/volumes"
Jul 16 23:19:20.789456 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.789413    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6"]
Jul 16 23:19:20.789976 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.789923    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="443c802b-22b2-41f6-b0aa-86e8a2b4cbbf" containerName="maas-api"
Jul 16 23:19:20.789976 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.789939    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="443c802b-22b2-41f6-b0aa-86e8a2b4cbbf" containerName="maas-api"
Jul 16 23:19:20.789976 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.789953    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746bbe78-e7e7-431d-afc7-214a81973926" containerName="revoke-keys"
Jul 16 23:19:20.789976 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.789958    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="746bbe78-e7e7-431d-afc7-214a81973926" containerName="revoke-keys"
Jul 16 23:19:20.789976 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.789968    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25aa5168-250d-4b3e-9de8-72282b82a95a" containerName="revoke-keys"
Jul 16 23:19:20.789976 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.789975    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="25aa5168-250d-4b3e-9de8-72282b82a95a" containerName="revoke-keys"
Jul 16 23:19:20.790227 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.789988    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746bbe78-e7e7-431d-afc7-214a81973926" containerName="revoke-keys"
Jul 16 23:19:20.790227 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.789993    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="746bbe78-e7e7-431d-afc7-214a81973926" containerName="revoke-keys"
Jul 16 23:19:20.790227 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.789999    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746bbe78-e7e7-431d-afc7-214a81973926" containerName="revoke-keys"
Jul 16 23:19:20.790227 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.790004    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="746bbe78-e7e7-431d-afc7-214a81973926" containerName="revoke-keys"
Jul 16 23:19:20.790227 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.790073    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="746bbe78-e7e7-431d-afc7-214a81973926" containerName="revoke-keys"
Jul 16 23:19:20.790227 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.790086    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="746bbe78-e7e7-431d-afc7-214a81973926" containerName="revoke-keys"
Jul 16 23:19:20.790227 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.790096    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="25aa5168-250d-4b3e-9de8-72282b82a95a" containerName="revoke-keys"
Jul 16 23:19:20.790227 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.790106    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="443c802b-22b2-41f6-b0aa-86e8a2b4cbbf" containerName="maas-api"
Jul 16 23:19:20.793144 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.793120    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6"
Jul 16 23:19:20.795353 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.795329    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:19:20.799369 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.799342    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6"]
Jul 16 23:19:20.972409 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:20.972372    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d84cb09-eb15-4356-ac80-05b719fb4994-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6\" (UID: \"8d84cb09-eb15-4356-ac80-05b719fb4994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6"
Jul 16 23:19:21.073016 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:21.072917    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d84cb09-eb15-4356-ac80-05b719fb4994-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6\" (UID: \"8d84cb09-eb15-4356-ac80-05b719fb4994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6"
Jul 16 23:19:21.073632 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:21.073610    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d84cb09-eb15-4356-ac80-05b719fb4994-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6\" (UID: \"8d84cb09-eb15-4356-ac80-05b719fb4994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6"
Jul 16 23:19:21.107050 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:21.107008    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6"
Jul 16 23:19:21.248126 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:21.248085    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6"]
Jul 16 23:19:21.252560 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:19:21.252491    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d84cb09_eb15_4356_ac80_05b719fb4994.slice/crio-26f745362749479e73183f114f56045aa62c2ac2da77a96eaa26fdd35dfadfb3 WatchSource:0}: Error finding container 26f745362749479e73183f114f56045aa62c2ac2da77a96eaa26fdd35dfadfb3: Status 404 returned error can't find the container with id 26f745362749479e73183f114f56045aa62c2ac2da77a96eaa26fdd35dfadfb3
Jul 16 23:19:21.966339 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:21.966294    2565 generic.go:356] "Generic (PLEG): container finished" podID="8d84cb09-eb15-4356-ac80-05b719fb4994" containerID="c33ebe612c534d3f9402d3adbdc46203f2b58e044ec073a41489a25fe886e639" exitCode=6
Jul 16 23:19:21.966833 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:21.966379    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6" event={"ID":"8d84cb09-eb15-4356-ac80-05b719fb4994","Type":"ContainerDied","Data":"c33ebe612c534d3f9402d3adbdc46203f2b58e044ec073a41489a25fe886e639"}
Jul 16 23:19:21.966833 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:21.966436    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6" event={"ID":"8d84cb09-eb15-4356-ac80-05b719fb4994","Type":"ContainerStarted","Data":"26f745362749479e73183f114f56045aa62c2ac2da77a96eaa26fdd35dfadfb3"}
Jul 16 23:19:21.966833 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:21.966619    2565 scope.go:117] "RemoveContainer" containerID="c33ebe612c534d3f9402d3adbdc46203f2b58e044ec073a41489a25fe886e639"
Jul 16 23:19:22.974106 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:22.974007    2565 generic.go:356] "Generic (PLEG): container finished" podID="8d84cb09-eb15-4356-ac80-05b719fb4994" containerID="e208b14128123ae7ddc50e064f5ade9b05be1e602c8cb1227d7f5168cac557bb" exitCode=6
Jul 16 23:19:22.974106 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:22.974089    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6" event={"ID":"8d84cb09-eb15-4356-ac80-05b719fb4994","Type":"ContainerDied","Data":"e208b14128123ae7ddc50e064f5ade9b05be1e602c8cb1227d7f5168cac557bb"}
Jul 16 23:19:22.974662 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:22.974143    2565 scope.go:117] "RemoveContainer" containerID="c33ebe612c534d3f9402d3adbdc46203f2b58e044ec073a41489a25fe886e639"
Jul 16 23:19:22.974662 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:22.974384    2565 scope.go:117] "RemoveContainer" containerID="e208b14128123ae7ddc50e064f5ade9b05be1e602c8cb1227d7f5168cac557bb"
Jul 16 23:19:22.974662 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:19:22.974633    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6_odh-ai-gateway-infra(8d84cb09-eb15-4356-ac80-05b719fb4994)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6" podUID="8d84cb09-eb15-4356-ac80-05b719fb4994"
Jul 16 23:19:23.980405 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:23.980368    2565 scope.go:117] "RemoveContainer" containerID="e208b14128123ae7ddc50e064f5ade9b05be1e602c8cb1227d7f5168cac557bb"
Jul 16 23:19:23.980827 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:19:23.980618    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6_odh-ai-gateway-infra(8d84cb09-eb15-4356-ac80-05b719fb4994)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6" podUID="8d84cb09-eb15-4356-ac80-05b719fb4994"
Jul 16 23:19:37.996877 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:37.996779    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b"]
Jul 16 23:19:37.997435 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:37.997415    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="746bbe78-e7e7-431d-afc7-214a81973926" containerName="revoke-keys"
Jul 16 23:19:38.001957 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:38.001934    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b"
Jul 16 23:19:38.007285 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:38.007238    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b"]
Jul 16 23:19:38.025311 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:38.025278    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b338560-1c94-4a7c-a721-a3fcb871fa4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b\" (UID: \"7b338560-1c94-4a7c-a721-a3fcb871fa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b"
Jul 16 23:19:38.126670 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:38.126623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b338560-1c94-4a7c-a721-a3fcb871fa4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b\" (UID: \"7b338560-1c94-4a7c-a721-a3fcb871fa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b"
Jul 16 23:19:38.127296 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:38.127273    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b338560-1c94-4a7c-a721-a3fcb871fa4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b\" (UID: \"7b338560-1c94-4a7c-a721-a3fcb871fa4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b"
Jul 16 23:19:38.313740 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:38.313649    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b"
Jul 16 23:19:38.460971 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:38.460935    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b"]
Jul 16 23:19:38.466583 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:19:38.466556    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b338560_1c94_4a7c_a721_a3fcb871fa4c.slice/crio-5e910cde7eb04d6418dc8b5833d81460bb193eece4447f8e8511b7071bb79e8e WatchSource:0}: Error finding container 5e910cde7eb04d6418dc8b5833d81460bb193eece4447f8e8511b7071bb79e8e: Status 404 returned error can't find the container with id 5e910cde7eb04d6418dc8b5833d81460bb193eece4447f8e8511b7071bb79e8e
Jul 16 23:19:39.053241 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:39.053202    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b" event={"ID":"7b338560-1c94-4a7c-a721-a3fcb871fa4c","Type":"ContainerStarted","Data":"b31d7f6c0193b4381e90ae70bb5a8744cf42aee5455eba1a4748ea95a1e30752"}
Jul 16 23:19:39.053582 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:39.053289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b" event={"ID":"7b338560-1c94-4a7c-a721-a3fcb871fa4c","Type":"ContainerStarted","Data":"5e910cde7eb04d6418dc8b5833d81460bb193eece4447f8e8511b7071bb79e8e"}
Jul 16 23:19:39.053582 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:39.053424    2565 scope.go:117] "RemoveContainer" containerID="b31d7f6c0193b4381e90ae70bb5a8744cf42aee5455eba1a4748ea95a1e30752"
Jul 16 23:19:39.723427 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:39.723341    2565 scope.go:117] "RemoveContainer" containerID="e208b14128123ae7ddc50e064f5ade9b05be1e602c8cb1227d7f5168cac557bb"
Jul 16 23:19:40.060458 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:40.060429    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6" event={"ID":"8d84cb09-eb15-4356-ac80-05b719fb4994","Type":"ContainerStarted","Data":"85724fdb4d0cd749600e7528aa2b8eec83321d784d372ebdf1a591c6ebf49cc6"}
Jul 16 23:19:40.062164 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:40.062135    2565 generic.go:356] "Generic (PLEG): container finished" podID="7b338560-1c94-4a7c-a721-a3fcb871fa4c" containerID="b31d7f6c0193b4381e90ae70bb5a8744cf42aee5455eba1a4748ea95a1e30752" exitCode=6
Jul 16 23:19:40.062164 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:40.062159    2565 generic.go:356] "Generic (PLEG): container finished" podID="7b338560-1c94-4a7c-a721-a3fcb871fa4c" containerID="2bce42eb154dee7b357b19bca1b1b55cacc43a0b82e6e3988eb6e5842c2a8c54" exitCode=6
Jul 16 23:19:40.062399 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:40.062196    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b" event={"ID":"7b338560-1c94-4a7c-a721-a3fcb871fa4c","Type":"ContainerDied","Data":"b31d7f6c0193b4381e90ae70bb5a8744cf42aee5455eba1a4748ea95a1e30752"}
Jul 16 23:19:40.062399 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:40.062232    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b" event={"ID":"7b338560-1c94-4a7c-a721-a3fcb871fa4c","Type":"ContainerDied","Data":"2bce42eb154dee7b357b19bca1b1b55cacc43a0b82e6e3988eb6e5842c2a8c54"}
Jul 16 23:19:40.062399 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:40.062280    2565 scope.go:117] "RemoveContainer" containerID="b31d7f6c0193b4381e90ae70bb5a8744cf42aee5455eba1a4748ea95a1e30752"
Jul 16 23:19:40.062563 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:40.062419    2565 scope.go:117] "RemoveContainer" containerID="2bce42eb154dee7b357b19bca1b1b55cacc43a0b82e6e3988eb6e5842c2a8c54"
Jul 16 23:19:40.062650 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:19:40.062628    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b_odh-ai-gateway-infra(7b338560-1c94-4a7c-a721-a3fcb871fa4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b" podUID="7b338560-1c94-4a7c-a721-a3fcb871fa4c"
Jul 16 23:19:40.072641 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:40.072619    2565 scope.go:117] "RemoveContainer" containerID="b31d7f6c0193b4381e90ae70bb5a8744cf42aee5455eba1a4748ea95a1e30752"
Jul 16 23:19:40.072897 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:19:40.072877    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b31d7f6c0193b4381e90ae70bb5a8744cf42aee5455eba1a4748ea95a1e30752\": container with ID starting with b31d7f6c0193b4381e90ae70bb5a8744cf42aee5455eba1a4748ea95a1e30752 not found: ID does not exist" containerID="b31d7f6c0193b4381e90ae70bb5a8744cf42aee5455eba1a4748ea95a1e30752"
Jul 16 23:19:40.072959 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:40.072909    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31d7f6c0193b4381e90ae70bb5a8744cf42aee5455eba1a4748ea95a1e30752"} err="failed to get container status \"b31d7f6c0193b4381e90ae70bb5a8744cf42aee5455eba1a4748ea95a1e30752\": rpc error: code = NotFound desc = could not find container \"b31d7f6c0193b4381e90ae70bb5a8744cf42aee5455eba1a4748ea95a1e30752\": container with ID starting with b31d7f6c0193b4381e90ae70bb5a8744cf42aee5455eba1a4748ea95a1e30752 not found: ID does not exist"
Jul 16 23:19:40.075951 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:40.075910    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6" podStartSLOduration=20.075899377 podStartE2EDuration="20.075899377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:19:40.074958056 +0000 UTC m=+2790.823442126" watchObservedRunningTime="2026-07-16 23:19:40.075899377 +0000 UTC m=+2790.824383447"
Jul 16 23:19:41.069795 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:41.069760    2565 generic.go:356] "Generic (PLEG): container finished" podID="8d84cb09-eb15-4356-ac80-05b719fb4994" containerID="85724fdb4d0cd749600e7528aa2b8eec83321d784d372ebdf1a591c6ebf49cc6" exitCode=6
Jul 16 23:19:41.070242 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:41.069830    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6" event={"ID":"8d84cb09-eb15-4356-ac80-05b719fb4994","Type":"ContainerDied","Data":"85724fdb4d0cd749600e7528aa2b8eec83321d784d372ebdf1a591c6ebf49cc6"}
Jul 16 23:19:41.070242 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:41.069904    2565 scope.go:117] "RemoveContainer" containerID="e208b14128123ae7ddc50e064f5ade9b05be1e602c8cb1227d7f5168cac557bb"
Jul 16 23:19:41.070242 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:41.070202    2565 scope.go:117] "RemoveContainer" containerID="85724fdb4d0cd749600e7528aa2b8eec83321d784d372ebdf1a591c6ebf49cc6"
Jul 16 23:19:41.070531 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:19:41.070496    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6_odh-ai-gateway-infra(8d84cb09-eb15-4356-ac80-05b719fb4994)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6" podUID="8d84cb09-eb15-4356-ac80-05b719fb4994"
Jul 16 23:19:41.071677 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:41.071662    2565 scope.go:117] "RemoveContainer" containerID="2bce42eb154dee7b357b19bca1b1b55cacc43a0b82e6e3988eb6e5842c2a8c54"
Jul 16 23:19:41.071849 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:19:41.071832    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b_odh-ai-gateway-infra(7b338560-1c94-4a7c-a721-a3fcb871fa4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b" podUID="7b338560-1c94-4a7c-a721-a3fcb871fa4c"
Jul 16 23:19:41.085036 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:41.085006    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6"]
Jul 16 23:19:42.229264 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:42.229225    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6"
Jul 16 23:19:42.266966 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:42.266929    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d84cb09-eb15-4356-ac80-05b719fb4994-maas-api-service-ca\") pod \"8d84cb09-eb15-4356-ac80-05b719fb4994\" (UID: \"8d84cb09-eb15-4356-ac80-05b719fb4994\") "
Jul 16 23:19:42.267409 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:42.267382    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d84cb09-eb15-4356-ac80-05b719fb4994-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8d84cb09-eb15-4356-ac80-05b719fb4994" (UID: "8d84cb09-eb15-4356-ac80-05b719fb4994"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:19:42.368198 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:42.368108    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d84cb09-eb15-4356-ac80-05b719fb4994-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:19:43.048614 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.048579    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd"]
Jul 16 23:19:43.048960 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.048948    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d84cb09-eb15-4356-ac80-05b719fb4994" containerName="revoke-keys"
Jul 16 23:19:43.049005 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.048962    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d84cb09-eb15-4356-ac80-05b719fb4994" containerName="revoke-keys"
Jul 16 23:19:43.049005 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.048975    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d84cb09-eb15-4356-ac80-05b719fb4994" containerName="revoke-keys"
Jul 16 23:19:43.049005 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.048981    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d84cb09-eb15-4356-ac80-05b719fb4994" containerName="revoke-keys"
Jul 16 23:19:43.049005 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.048991    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d84cb09-eb15-4356-ac80-05b719fb4994" containerName="revoke-keys"
Jul 16 23:19:43.049005 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.048998    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d84cb09-eb15-4356-ac80-05b719fb4994" containerName="revoke-keys"
Jul 16 23:19:43.049178 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.049090    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d84cb09-eb15-4356-ac80-05b719fb4994" containerName="revoke-keys"
Jul 16 23:19:43.049178 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.049100    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d84cb09-eb15-4356-ac80-05b719fb4994" containerName="revoke-keys"
Jul 16 23:19:43.052087 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.052071    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd"
Jul 16 23:19:43.059097 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.059061    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd"]
Jul 16 23:19:43.085061 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.085031    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6"
Jul 16 23:19:43.085213 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.085034    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6" event={"ID":"8d84cb09-eb15-4356-ac80-05b719fb4994","Type":"ContainerDied","Data":"26f745362749479e73183f114f56045aa62c2ac2da77a96eaa26fdd35dfadfb3"}
Jul 16 23:19:43.085213 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.085168    2565 scope.go:117] "RemoveContainer" containerID="85724fdb4d0cd749600e7528aa2b8eec83321d784d372ebdf1a591c6ebf49cc6"
Jul 16 23:19:43.113379 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.113347    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6"]
Jul 16 23:19:43.118641 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.118614    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8vqn6"]
Jul 16 23:19:43.175108 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.175064    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae904ea-fa70-4363-8b2c-2e7346fc4277-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-lzcxd\" (UID: \"1ae904ea-fa70-4363-8b2c-2e7346fc4277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd"
Jul 16 23:19:43.276197 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.276156    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae904ea-fa70-4363-8b2c-2e7346fc4277-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-lzcxd\" (UID: \"1ae904ea-fa70-4363-8b2c-2e7346fc4277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd"
Jul 16 23:19:43.276877 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.276859    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae904ea-fa70-4363-8b2c-2e7346fc4277-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-lzcxd\" (UID: \"1ae904ea-fa70-4363-8b2c-2e7346fc4277\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd"
Jul 16 23:19:43.363379 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.363343    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd"
Jul 16 23:19:43.499538 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.499505    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd"]
Jul 16 23:19:43.502739 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:19:43.502713    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae904ea_fa70_4363_8b2c_2e7346fc4277.slice/crio-1d5679637a5e576efdf5bd769d2ad5a618a04b62bb527c8c5cfa3f569ed3b7df WatchSource:0}: Error finding container 1d5679637a5e576efdf5bd769d2ad5a618a04b62bb527c8c5cfa3f569ed3b7df: Status 404 returned error can't find the container with id 1d5679637a5e576efdf5bd769d2ad5a618a04b62bb527c8c5cfa3f569ed3b7df
Jul 16 23:19:43.727001 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:43.726917    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d84cb09-eb15-4356-ac80-05b719fb4994" path="/var/lib/kubelet/pods/8d84cb09-eb15-4356-ac80-05b719fb4994/volumes"
Jul 16 23:19:44.091656 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:44.091619    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd" event={"ID":"1ae904ea-fa70-4363-8b2c-2e7346fc4277","Type":"ContainerStarted","Data":"cfb80ad134039dda75428b22da481236047f8e58b785bd471ada37f5fe3eebd8"}
Jul 16 23:19:44.091839 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:44.091673    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd" event={"ID":"1ae904ea-fa70-4363-8b2c-2e7346fc4277","Type":"ContainerStarted","Data":"1d5679637a5e576efdf5bd769d2ad5a618a04b62bb527c8c5cfa3f569ed3b7df"}
Jul 16 23:19:44.106845 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:44.106796    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd" podStartSLOduration=1.106780417 podStartE2EDuration="1.106780417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:19:44.106342276 +0000 UTC m=+2794.854826350" watchObservedRunningTime="2026-07-16 23:19:44.106780417 +0000 UTC m=+2794.855264487"
Jul 16 23:19:45.097903 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:45.097862    2565 generic.go:356] "Generic (PLEG): container finished" podID="1ae904ea-fa70-4363-8b2c-2e7346fc4277" containerID="cfb80ad134039dda75428b22da481236047f8e58b785bd471ada37f5fe3eebd8" exitCode=6
Jul 16 23:19:45.098335 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:45.097944    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd" event={"ID":"1ae904ea-fa70-4363-8b2c-2e7346fc4277","Type":"ContainerDied","Data":"cfb80ad134039dda75428b22da481236047f8e58b785bd471ada37f5fe3eebd8"}
Jul 16 23:19:45.098335 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:45.098126    2565 scope.go:117] "RemoveContainer" containerID="cfb80ad134039dda75428b22da481236047f8e58b785bd471ada37f5fe3eebd8"
Jul 16 23:19:46.108235 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:46.108198    2565 generic.go:356] "Generic (PLEG): container finished" podID="1ae904ea-fa70-4363-8b2c-2e7346fc4277" containerID="9e7aef937b3129e260af981e17eea52fc4388565814af0784945aaa75483042f" exitCode=6
Jul 16 23:19:46.108746 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:46.108284    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd" event={"ID":"1ae904ea-fa70-4363-8b2c-2e7346fc4277","Type":"ContainerDied","Data":"9e7aef937b3129e260af981e17eea52fc4388565814af0784945aaa75483042f"}
Jul 16 23:19:46.108746 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:46.108350    2565 scope.go:117] "RemoveContainer" containerID="cfb80ad134039dda75428b22da481236047f8e58b785bd471ada37f5fe3eebd8"
Jul 16 23:19:46.108746 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:46.108522    2565 scope.go:117] "RemoveContainer" containerID="9e7aef937b3129e260af981e17eea52fc4388565814af0784945aaa75483042f"
Jul 16 23:19:46.108912 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:19:46.108815    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-lzcxd_odh-ai-gateway-infra(1ae904ea-fa70-4363-8b2c-2e7346fc4277)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd" podUID="1ae904ea-fa70-4363-8b2c-2e7346fc4277"
Jul 16 23:19:47.115007 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:47.114983    2565 scope.go:117] "RemoveContainer" containerID="9e7aef937b3129e260af981e17eea52fc4388565814af0784945aaa75483042f"
Jul 16 23:19:47.115473 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:19:47.115167    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-lzcxd_odh-ai-gateway-infra(1ae904ea-fa70-4363-8b2c-2e7346fc4277)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd" podUID="1ae904ea-fa70-4363-8b2c-2e7346fc4277"
Jul 16 23:19:51.858035 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:51.857998    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk"]
Jul 16 23:19:51.858587 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:51.858551    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d84cb09-eb15-4356-ac80-05b719fb4994" containerName="revoke-keys"
Jul 16 23:19:51.861548 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:51.861528    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk"
Jul 16 23:19:51.868682 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:51.868644    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk"]
Jul 16 23:19:51.959755 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:51.959710    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/593f9ce0-0dc2-4be0-bfbe-de670813995e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk\" (UID: \"593f9ce0-0dc2-4be0-bfbe-de670813995e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk"
Jul 16 23:19:52.061106 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:52.061063    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/593f9ce0-0dc2-4be0-bfbe-de670813995e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk\" (UID: \"593f9ce0-0dc2-4be0-bfbe-de670813995e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk"
Jul 16 23:19:52.061746 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:52.061723    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/593f9ce0-0dc2-4be0-bfbe-de670813995e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk\" (UID: \"593f9ce0-0dc2-4be0-bfbe-de670813995e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk"
Jul 16 23:19:52.175420 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:52.175309    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk"
Jul 16 23:19:52.312777 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:52.312749    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk"]
Jul 16 23:19:52.315605 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:19:52.315572    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593f9ce0_0dc2_4be0_bfbe_de670813995e.slice/crio-e3748fee5a98fe8502ee30b567c0a585367d8bcd78ed8b6d5e8874dc87cd7ca1 WatchSource:0}: Error finding container e3748fee5a98fe8502ee30b567c0a585367d8bcd78ed8b6d5e8874dc87cd7ca1: Status 404 returned error can't find the container with id e3748fee5a98fe8502ee30b567c0a585367d8bcd78ed8b6d5e8874dc87cd7ca1
Jul 16 23:19:53.082030 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:53.081989    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"]
Jul 16 23:19:53.082455 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:53.082286    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg" podUID="e8e2713c-4c29-49ea-81bc-97edd584d3fd" containerName="istio-proxy" containerID="cri-o://907df6805fb1c6b9df1446c43f095ff581cf68d36fd2e9c0e1170b1a067032ce" gracePeriod=30
Jul 16 23:19:53.146747 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:53.146714    2565 generic.go:356] "Generic (PLEG): container finished" podID="593f9ce0-0dc2-4be0-bfbe-de670813995e" containerID="b0599a1bf1aceb1d107a234b89f00008192d0c62d68346b88edc752a7d025181" exitCode=6
Jul 16 23:19:53.146964 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:53.146803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk" event={"ID":"593f9ce0-0dc2-4be0-bfbe-de670813995e","Type":"ContainerDied","Data":"b0599a1bf1aceb1d107a234b89f00008192d0c62d68346b88edc752a7d025181"}
Jul 16 23:19:53.146964 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:53.146857    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk" event={"ID":"593f9ce0-0dc2-4be0-bfbe-de670813995e","Type":"ContainerStarted","Data":"e3748fee5a98fe8502ee30b567c0a585367d8bcd78ed8b6d5e8874dc87cd7ca1"}
Jul 16 23:19:53.147069 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:53.146988    2565 scope.go:117] "RemoveContainer" containerID="b0599a1bf1aceb1d107a234b89f00008192d0c62d68346b88edc752a7d025181"
Jul 16 23:19:54.153236 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:54.153144    2565 generic.go:356] "Generic (PLEG): container finished" podID="593f9ce0-0dc2-4be0-bfbe-de670813995e" containerID="655cff7c914d9090851f9a8dd0454c9473edb9c96d269fab4b740d8fa114ad12" exitCode=6
Jul 16 23:19:54.153678 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:54.153234    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk" event={"ID":"593f9ce0-0dc2-4be0-bfbe-de670813995e","Type":"ContainerDied","Data":"655cff7c914d9090851f9a8dd0454c9473edb9c96d269fab4b740d8fa114ad12"}
Jul 16 23:19:54.153678 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:54.153308    2565 scope.go:117] "RemoveContainer" containerID="b0599a1bf1aceb1d107a234b89f00008192d0c62d68346b88edc752a7d025181"
Jul 16 23:19:54.153678 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:54.153500    2565 scope.go:117] "RemoveContainer" containerID="655cff7c914d9090851f9a8dd0454c9473edb9c96d269fab4b740d8fa114ad12"
Jul 16 23:19:54.153803 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:19:54.153732    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk_odh-ai-gateway-infra(593f9ce0-0dc2-4be0-bfbe-de670813995e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk" podUID="593f9ce0-0dc2-4be0-bfbe-de670813995e"
Jul 16 23:19:54.721174 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:54.721144    2565 scope.go:117] "RemoveContainer" containerID="2bce42eb154dee7b357b19bca1b1b55cacc43a0b82e6e3988eb6e5842c2a8c54"
Jul 16 23:19:55.160884 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:55.160849    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b" event={"ID":"7b338560-1c94-4a7c-a721-a3fcb871fa4c","Type":"ContainerStarted","Data":"4ac14fadae8503e618734c78995c6063e8d513fe98ab9418b165e47f2040d49e"}
Jul 16 23:19:55.162552 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:55.162537    2565 scope.go:117] "RemoveContainer" containerID="655cff7c914d9090851f9a8dd0454c9473edb9c96d269fab4b740d8fa114ad12"
Jul 16 23:19:55.162705 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:19:55.162688    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk_odh-ai-gateway-infra(593f9ce0-0dc2-4be0-bfbe-de670813995e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk" podUID="593f9ce0-0dc2-4be0-bfbe-de670813995e"
Jul 16 23:19:55.182951 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:55.182890    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b" podStartSLOduration=18.182873095 podStartE2EDuration="18.182873095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:19:55.181942524 +0000 UTC m=+2805.930426607" watchObservedRunningTime="2026-07-16 23:19:55.182873095 +0000 UTC m=+2805.931357166"
Jul 16 23:19:56.169565 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:56.169526    2565 generic.go:356] "Generic (PLEG): container finished" podID="7b338560-1c94-4a7c-a721-a3fcb871fa4c" containerID="4ac14fadae8503e618734c78995c6063e8d513fe98ab9418b165e47f2040d49e" exitCode=6
Jul 16 23:19:56.170052 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:56.169581    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b" event={"ID":"7b338560-1c94-4a7c-a721-a3fcb871fa4c","Type":"ContainerDied","Data":"4ac14fadae8503e618734c78995c6063e8d513fe98ab9418b165e47f2040d49e"}
Jul 16 23:19:56.170052 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:56.169657    2565 scope.go:117] "RemoveContainer" containerID="2bce42eb154dee7b357b19bca1b1b55cacc43a0b82e6e3988eb6e5842c2a8c54"
Jul 16 23:19:56.170052 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:56.169950    2565 scope.go:117] "RemoveContainer" containerID="4ac14fadae8503e618734c78995c6063e8d513fe98ab9418b165e47f2040d49e"
Jul 16 23:19:56.170234 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:19:56.170211    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b_odh-ai-gateway-infra(7b338560-1c94-4a7c-a721-a3fcb871fa4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b" podUID="7b338560-1c94-4a7c-a721-a3fcb871fa4c"
Jul 16 23:19:56.191935 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:56.191909    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b"]
Jul 16 23:19:57.319356 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:57.319327    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b"
Jul 16 23:19:57.415961 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:57.415907    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b338560-1c94-4a7c-a721-a3fcb871fa4c-maas-api-service-ca\") pod \"7b338560-1c94-4a7c-a721-a3fcb871fa4c\" (UID: \"7b338560-1c94-4a7c-a721-a3fcb871fa4c\") "
Jul 16 23:19:57.416327 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:57.416302    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b338560-1c94-4a7c-a721-a3fcb871fa4c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b338560-1c94-4a7c-a721-a3fcb871fa4c" (UID: "7b338560-1c94-4a7c-a721-a3fcb871fa4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:19:57.517372 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:57.517281    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b338560-1c94-4a7c-a721-a3fcb871fa4c-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:19:58.182596 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.182557    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b" event={"ID":"7b338560-1c94-4a7c-a721-a3fcb871fa4c","Type":"ContainerDied","Data":"5e910cde7eb04d6418dc8b5833d81460bb193eece4447f8e8511b7071bb79e8e"}
Jul 16 23:19:58.182596 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.182582    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b"
Jul 16 23:19:58.182848 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.182625    2565 scope.go:117] "RemoveContainer" containerID="4ac14fadae8503e618734c78995c6063e8d513fe98ab9418b165e47f2040d49e"
Jul 16 23:19:58.203960 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.203927    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b"]
Jul 16 23:19:58.210308 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.210266    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-p8n9b"]
Jul 16 23:19:58.349869 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.349799    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:19:58.526840 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.526804    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-workload-socket\") pod \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") "
Jul 16 23:19:58.526840 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.526853    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-envoy\") pod \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") "
Jul 16 23:19:58.527187 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.526869    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-workload-certs\") pod \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") "
Jul 16 23:19:58.527187 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.526885    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-data\") pod \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") "
Jul 16 23:19:58.527187 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.526928    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-podinfo\") pod \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") "
Jul 16 23:19:58.527187 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.526976    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-credential-socket\") pod \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") "
Jul 16 23:19:58.527187 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.527044    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57lbn\" (UniqueName: \"kubernetes.io/projected/e8e2713c-4c29-49ea-81bc-97edd584d3fd-kube-api-access-57lbn\") pod \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") "
Jul 16 23:19:58.527187 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.527091    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istiod-ca-cert\") pod \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") "
Jul 16 23:19:58.527187 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.527107    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-token\") pod \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\" (UID: \"e8e2713c-4c29-49ea-81bc-97edd584d3fd\") "
Jul 16 23:19:58.527187 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.527163    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e8e2713c-4c29-49ea-81bc-97edd584d3fd" (UID: "e8e2713c-4c29-49ea-81bc-97edd584d3fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:19:58.527579 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.527184    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e8e2713c-4c29-49ea-81bc-97edd584d3fd" (UID: "e8e2713c-4c29-49ea-81bc-97edd584d3fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:19:58.527579 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.527473    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-workload-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:19:58.527579 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.527494    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-workload-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:19:58.527579 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.527507    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e8e2713c-4c29-49ea-81bc-97edd584d3fd" (UID: "e8e2713c-4c29-49ea-81bc-97edd584d3fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:19:58.527798 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.527752    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e8e2713c-4c29-49ea-81bc-97edd584d3fd" (UID: "e8e2713c-4c29-49ea-81bc-97edd584d3fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:19:58.527970 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.527946    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-data" (OuterVolumeSpecName: "istio-data") pod "e8e2713c-4c29-49ea-81bc-97edd584d3fd" (UID: "e8e2713c-4c29-49ea-81bc-97edd584d3fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:19:58.529343 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.529307    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e2713c-4c29-49ea-81bc-97edd584d3fd-kube-api-access-57lbn" (OuterVolumeSpecName: "kube-api-access-57lbn") pod "e8e2713c-4c29-49ea-81bc-97edd584d3fd" (UID: "e8e2713c-4c29-49ea-81bc-97edd584d3fd"). InnerVolumeSpecName "kube-api-access-57lbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:19:58.529451 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.529371    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-token" (OuterVolumeSpecName: "istio-token") pod "e8e2713c-4c29-49ea-81bc-97edd584d3fd" (UID: "e8e2713c-4c29-49ea-81bc-97edd584d3fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:19:58.529451 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.529385    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e8e2713c-4c29-49ea-81bc-97edd584d3fd" (UID: "e8e2713c-4c29-49ea-81bc-97edd584d3fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:19:58.529587 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.529563    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e8e2713c-4c29-49ea-81bc-97edd584d3fd" (UID: "e8e2713c-4c29-49ea-81bc-97edd584d3fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 23:19:58.629042 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.628958    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-envoy\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:19:58.629042 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.628988    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-data\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:19:58.629042 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.628997    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-podinfo\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:19:58.629042 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.629006    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8e2713c-4c29-49ea-81bc-97edd584d3fd-credential-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:19:58.629042 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.629016    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57lbn\" (UniqueName: \"kubernetes.io/projected/e8e2713c-4c29-49ea-81bc-97edd584d3fd-kube-api-access-57lbn\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:19:58.629042 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.629025    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istiod-ca-cert\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:19:58.629042 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.629033    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8e2713c-4c29-49ea-81bc-97edd584d3fd-istio-token\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:19:58.720866 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:58.720837    2565 scope.go:117] "RemoveContainer" containerID="9e7aef937b3129e260af981e17eea52fc4388565814af0784945aaa75483042f"
Jul 16 23:19:59.189714 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:59.189680    2565 generic.go:356] "Generic (PLEG): container finished" podID="e8e2713c-4c29-49ea-81bc-97edd584d3fd" containerID="907df6805fb1c6b9df1446c43f095ff581cf68d36fd2e9c0e1170b1a067032ce" exitCode=0
Jul 16 23:19:59.189918 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:59.189728    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"
Jul 16 23:19:59.189918 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:59.189765    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg" event={"ID":"e8e2713c-4c29-49ea-81bc-97edd584d3fd","Type":"ContainerDied","Data":"907df6805fb1c6b9df1446c43f095ff581cf68d36fd2e9c0e1170b1a067032ce"}
Jul 16 23:19:59.189918 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:59.189820    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg" event={"ID":"e8e2713c-4c29-49ea-81bc-97edd584d3fd","Type":"ContainerDied","Data":"8248b88865b4e655c456062dbf94e31ce24208feed98d60ccf230b4f25f55861"}
Jul 16 23:19:59.189918 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:59.189849    2565 scope.go:117] "RemoveContainer" containerID="907df6805fb1c6b9df1446c43f095ff581cf68d36fd2e9c0e1170b1a067032ce"
Jul 16 23:19:59.191832 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:59.191809    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd" event={"ID":"1ae904ea-fa70-4363-8b2c-2e7346fc4277","Type":"ContainerStarted","Data":"1c93c0b444afdfa5744efadae93ea260fcd1f8bdd1403548b1dfa49bbcda9922"}
Jul 16 23:19:59.198889 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:59.198865    2565 scope.go:117] "RemoveContainer" containerID="8c8d44cd8b33610c26be61cfa6d3e63d0dc2928857b77a42e1ffbd8282f8af1a"
Jul 16 23:19:59.206777 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:59.206763    2565 scope.go:117] "RemoveContainer" containerID="907df6805fb1c6b9df1446c43f095ff581cf68d36fd2e9c0e1170b1a067032ce"
Jul 16 23:19:59.207004 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:19:59.206986    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"907df6805fb1c6b9df1446c43f095ff581cf68d36fd2e9c0e1170b1a067032ce\": container with ID starting with 907df6805fb1c6b9df1446c43f095ff581cf68d36fd2e9c0e1170b1a067032ce not found: ID does not exist" containerID="907df6805fb1c6b9df1446c43f095ff581cf68d36fd2e9c0e1170b1a067032ce"
Jul 16 23:19:59.207046 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:59.207012    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"907df6805fb1c6b9df1446c43f095ff581cf68d36fd2e9c0e1170b1a067032ce"} err="failed to get container status \"907df6805fb1c6b9df1446c43f095ff581cf68d36fd2e9c0e1170b1a067032ce\": rpc error: code = NotFound desc = could not find container \"907df6805fb1c6b9df1446c43f095ff581cf68d36fd2e9c0e1170b1a067032ce\": container with ID starting with 907df6805fb1c6b9df1446c43f095ff581cf68d36fd2e9c0e1170b1a067032ce not found: ID does not exist"
Jul 16 23:19:59.207046 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:59.207028    2565 scope.go:117] "RemoveContainer" containerID="8c8d44cd8b33610c26be61cfa6d3e63d0dc2928857b77a42e1ffbd8282f8af1a"
Jul 16 23:19:59.207225 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:19:59.207210    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c8d44cd8b33610c26be61cfa6d3e63d0dc2928857b77a42e1ffbd8282f8af1a\": container with ID starting with 8c8d44cd8b33610c26be61cfa6d3e63d0dc2928857b77a42e1ffbd8282f8af1a not found: ID does not exist" containerID="8c8d44cd8b33610c26be61cfa6d3e63d0dc2928857b77a42e1ffbd8282f8af1a"
Jul 16 23:19:59.207279 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:59.207228    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8d44cd8b33610c26be61cfa6d3e63d0dc2928857b77a42e1ffbd8282f8af1a"} err="failed to get container status \"8c8d44cd8b33610c26be61cfa6d3e63d0dc2928857b77a42e1ffbd8282f8af1a\": rpc error: code = NotFound desc = could not find container \"8c8d44cd8b33610c26be61cfa6d3e63d0dc2928857b77a42e1ffbd8282f8af1a\": container with ID starting with 8c8d44cd8b33610c26be61cfa6d3e63d0dc2928857b77a42e1ffbd8282f8af1a not found: ID does not exist"
Jul 16 23:19:59.222623 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:59.222593    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"]
Jul 16 23:19:59.230045 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:59.230019    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9752734e-openshift-default-55f7cf78c5-gxjfg"]
Jul 16 23:19:59.731977 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:59.731946    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b338560-1c94-4a7c-a721-a3fcb871fa4c" path="/var/lib/kubelet/pods/7b338560-1c94-4a7c-a721-a3fcb871fa4c/volumes"
Jul 16 23:19:59.732410 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:19:59.732396    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8e2713c-4c29-49ea-81bc-97edd584d3fd" path="/var/lib/kubelet/pods/e8e2713c-4c29-49ea-81bc-97edd584d3fd/volumes"
Jul 16 23:20:00.199306 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:00.199272    2565 generic.go:356] "Generic (PLEG): container finished" podID="1ae904ea-fa70-4363-8b2c-2e7346fc4277" containerID="1c93c0b444afdfa5744efadae93ea260fcd1f8bdd1403548b1dfa49bbcda9922" exitCode=6
Jul 16 23:20:00.199306 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:00.199285    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd" event={"ID":"1ae904ea-fa70-4363-8b2c-2e7346fc4277","Type":"ContainerDied","Data":"1c93c0b444afdfa5744efadae93ea260fcd1f8bdd1403548b1dfa49bbcda9922"}
Jul 16 23:20:00.199578 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:00.199349    2565 scope.go:117] "RemoveContainer" containerID="9e7aef937b3129e260af981e17eea52fc4388565814af0784945aaa75483042f"
Jul 16 23:20:00.199635 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:00.199615    2565 scope.go:117] "RemoveContainer" containerID="1c93c0b444afdfa5744efadae93ea260fcd1f8bdd1403548b1dfa49bbcda9922"
Jul 16 23:20:00.199853 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:20:00.199829    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-lzcxd_odh-ai-gateway-infra(1ae904ea-fa70-4363-8b2c-2e7346fc4277)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd" podUID="1ae904ea-fa70-4363-8b2c-2e7346fc4277"
Jul 16 23:20:00.220259 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:00.220225    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd"]
Jul 16 23:20:01.351822 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:01.351793    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd"
Jul 16 23:20:01.455104 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:01.455057    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae904ea-fa70-4363-8b2c-2e7346fc4277-maas-api-service-ca\") pod \"1ae904ea-fa70-4363-8b2c-2e7346fc4277\" (UID: \"1ae904ea-fa70-4363-8b2c-2e7346fc4277\") "
Jul 16 23:20:01.455485 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:01.455462    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae904ea-fa70-4363-8b2c-2e7346fc4277-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1ae904ea-fa70-4363-8b2c-2e7346fc4277" (UID: "1ae904ea-fa70-4363-8b2c-2e7346fc4277"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:20:01.556589 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:01.556502    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae904ea-fa70-4363-8b2c-2e7346fc4277-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:20:02.212358 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:02.212322    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd" event={"ID":"1ae904ea-fa70-4363-8b2c-2e7346fc4277","Type":"ContainerDied","Data":"1d5679637a5e576efdf5bd769d2ad5a618a04b62bb527c8c5cfa3f569ed3b7df"}
Jul 16 23:20:02.212545 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:02.212381    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd"
Jul 16 23:20:02.212545 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:02.212385    2565 scope.go:117] "RemoveContainer" containerID="1c93c0b444afdfa5744efadae93ea260fcd1f8bdd1403548b1dfa49bbcda9922"
Jul 16 23:20:02.227359 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:02.227332    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd"]
Jul 16 23:20:02.235818 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:02.235788    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lzcxd"]
Jul 16 23:20:03.725731 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:03.725693    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ae904ea-fa70-4363-8b2c-2e7346fc4277" path="/var/lib/kubelet/pods/1ae904ea-fa70-4363-8b2c-2e7346fc4277/volumes"
Jul 16 23:20:09.723260 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:09.723226    2565 scope.go:117] "RemoveContainer" containerID="655cff7c914d9090851f9a8dd0454c9473edb9c96d269fab4b740d8fa114ad12"
Jul 16 23:20:10.259978 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:10.259937    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk" event={"ID":"593f9ce0-0dc2-4be0-bfbe-de670813995e","Type":"ContainerStarted","Data":"7ab437960faf7a45894ea099a867753c5a91f8bee95094b2e0bea76e511bb81e"}
Jul 16 23:20:10.276606 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:10.276556    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk" podStartSLOduration=19.276540438 podStartE2EDuration="19.276540438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:20:10.274934307 +0000 UTC m=+2821.023418378" watchObservedRunningTime="2026-07-16 23:20:10.276540438 +0000 UTC m=+2821.025024508"
Jul 16 23:20:11.266849 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:11.266816    2565 generic.go:356] "Generic (PLEG): container finished" podID="593f9ce0-0dc2-4be0-bfbe-de670813995e" containerID="7ab437960faf7a45894ea099a867753c5a91f8bee95094b2e0bea76e511bb81e" exitCode=6
Jul 16 23:20:11.267309 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:11.266861    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk" event={"ID":"593f9ce0-0dc2-4be0-bfbe-de670813995e","Type":"ContainerDied","Data":"7ab437960faf7a45894ea099a867753c5a91f8bee95094b2e0bea76e511bb81e"}
Jul 16 23:20:11.267309 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:11.266911    2565 scope.go:117] "RemoveContainer" containerID="655cff7c914d9090851f9a8dd0454c9473edb9c96d269fab4b740d8fa114ad12"
Jul 16 23:20:11.267309 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:11.267182    2565 scope.go:117] "RemoveContainer" containerID="7ab437960faf7a45894ea099a867753c5a91f8bee95094b2e0bea76e511bb81e"
Jul 16 23:20:11.267470 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:20:11.267417    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk_odh-ai-gateway-infra(593f9ce0-0dc2-4be0-bfbe-de670813995e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk" podUID="593f9ce0-0dc2-4be0-bfbe-de670813995e"
Jul 16 23:20:11.284111 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:11.284086    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk"]
Jul 16 23:20:12.407864 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:12.407836    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk"
Jul 16 23:20:12.449034 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:12.449004    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/593f9ce0-0dc2-4be0-bfbe-de670813995e-maas-api-service-ca\") pod \"593f9ce0-0dc2-4be0-bfbe-de670813995e\" (UID: \"593f9ce0-0dc2-4be0-bfbe-de670813995e\") "
Jul 16 23:20:12.449355 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:12.449334    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593f9ce0-0dc2-4be0-bfbe-de670813995e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "593f9ce0-0dc2-4be0-bfbe-de670813995e" (UID: "593f9ce0-0dc2-4be0-bfbe-de670813995e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:20:12.550037 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:12.549956    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/593f9ce0-0dc2-4be0-bfbe-de670813995e-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:20:13.280157 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:13.280126    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk"
Jul 16 23:20:13.280157 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:13.280150    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk" event={"ID":"593f9ce0-0dc2-4be0-bfbe-de670813995e","Type":"ContainerDied","Data":"e3748fee5a98fe8502ee30b567c0a585367d8bcd78ed8b6d5e8874dc87cd7ca1"}
Jul 16 23:20:13.280477 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:13.280235    2565 scope.go:117] "RemoveContainer" containerID="7ab437960faf7a45894ea099a867753c5a91f8bee95094b2e0bea76e511bb81e"
Jul 16 23:20:13.306483 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:13.306457    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk"]
Jul 16 23:20:13.312883 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:13.312860    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-f4gnk"]
Jul 16 23:20:13.727771 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:13.727733    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="593f9ce0-0dc2-4be0-bfbe-de670813995e" path="/var/lib/kubelet/pods/593f9ce0-0dc2-4be0-bfbe-de670813995e/volumes"
Jul 16 23:20:20.828344 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.828308    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv"]
Jul 16 23:20:20.828856 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.828834    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b338560-1c94-4a7c-a721-a3fcb871fa4c" containerName="revoke-keys"
Jul 16 23:20:20.828961 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.828860    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b338560-1c94-4a7c-a721-a3fcb871fa4c" containerName="revoke-keys"
Jul 16 23:20:20.828961 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.828874    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b338560-1c94-4a7c-a721-a3fcb871fa4c" containerName="revoke-keys"
Jul 16 23:20:20.828961 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.828883    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b338560-1c94-4a7c-a721-a3fcb871fa4c" containerName="revoke-keys"
Jul 16 23:20:20.828961 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.828891    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ae904ea-fa70-4363-8b2c-2e7346fc4277" containerName="revoke-keys"
Jul 16 23:20:20.828961 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.828899    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ae904ea-fa70-4363-8b2c-2e7346fc4277" containerName="revoke-keys"
Jul 16 23:20:20.828961 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.828910    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="593f9ce0-0dc2-4be0-bfbe-de670813995e" containerName="revoke-keys"
Jul 16 23:20:20.828961 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.828918    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="593f9ce0-0dc2-4be0-bfbe-de670813995e" containerName="revoke-keys"
Jul 16 23:20:20.828961 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.828929    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="593f9ce0-0dc2-4be0-bfbe-de670813995e" containerName="revoke-keys"
Jul 16 23:20:20.828961 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.828937    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="593f9ce0-0dc2-4be0-bfbe-de670813995e" containerName="revoke-keys"
Jul 16 23:20:20.828961 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.828948    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="593f9ce0-0dc2-4be0-bfbe-de670813995e" containerName="revoke-keys"
Jul 16 23:20:20.828961 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.828956    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="593f9ce0-0dc2-4be0-bfbe-de670813995e" containerName="revoke-keys"
Jul 16 23:20:20.829514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.828967    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e2713c-4c29-49ea-81bc-97edd584d3fd" containerName="istio-proxy"
Jul 16 23:20:20.829514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.828975    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e2713c-4c29-49ea-81bc-97edd584d3fd" containerName="istio-proxy"
Jul 16 23:20:20.829514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.829016    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ae904ea-fa70-4363-8b2c-2e7346fc4277" containerName="revoke-keys"
Jul 16 23:20:20.829514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.829027    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ae904ea-fa70-4363-8b2c-2e7346fc4277" containerName="revoke-keys"
Jul 16 23:20:20.829514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.829041    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8e2713c-4c29-49ea-81bc-97edd584d3fd" containerName="istio-proxy"
Jul 16 23:20:20.829514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.829050    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8e2713c-4c29-49ea-81bc-97edd584d3fd" containerName="istio-proxy"
Jul 16 23:20:20.829514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.829074    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ae904ea-fa70-4363-8b2c-2e7346fc4277" containerName="revoke-keys"
Jul 16 23:20:20.829514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.829084    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ae904ea-fa70-4363-8b2c-2e7346fc4277" containerName="revoke-keys"
Jul 16 23:20:20.829514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.829172    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b338560-1c94-4a7c-a721-a3fcb871fa4c" containerName="revoke-keys"
Jul 16 23:20:20.829514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.829183    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b338560-1c94-4a7c-a721-a3fcb871fa4c" containerName="revoke-keys"
Jul 16 23:20:20.829514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.829193    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e2713c-4c29-49ea-81bc-97edd584d3fd" containerName="istio-proxy"
Jul 16 23:20:20.829514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.829203    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ae904ea-fa70-4363-8b2c-2e7346fc4277" containerName="revoke-keys"
Jul 16 23:20:20.829514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.829213    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8e2713c-4c29-49ea-81bc-97edd584d3fd" containerName="istio-proxy"
Jul 16 23:20:20.829514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.829221    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ae904ea-fa70-4363-8b2c-2e7346fc4277" containerName="revoke-keys"
Jul 16 23:20:20.829514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.829229    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="593f9ce0-0dc2-4be0-bfbe-de670813995e" containerName="revoke-keys"
Jul 16 23:20:20.829514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.829238    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="593f9ce0-0dc2-4be0-bfbe-de670813995e" containerName="revoke-keys"
Jul 16 23:20:20.829514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.829264    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="593f9ce0-0dc2-4be0-bfbe-de670813995e" containerName="revoke-keys"
Jul 16 23:20:20.832285 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.832241    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv"
Jul 16 23:20:20.834409 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.834390    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:20:20.838024 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.837996    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv"]
Jul 16 23:20:20.922472 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:20.922435    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85312b60-97ef-47b6-a26c-ca6e41e0fa4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv\" (UID: \"85312b60-97ef-47b6-a26c-ca6e41e0fa4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv"
Jul 16 23:20:21.023641 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:21.023584    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85312b60-97ef-47b6-a26c-ca6e41e0fa4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv\" (UID: \"85312b60-97ef-47b6-a26c-ca6e41e0fa4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv"
Jul 16 23:20:21.024261 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:21.024225    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85312b60-97ef-47b6-a26c-ca6e41e0fa4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv\" (UID: \"85312b60-97ef-47b6-a26c-ca6e41e0fa4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv"
Jul 16 23:20:21.143763 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:21.143673    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv"
Jul 16 23:20:21.280388 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:21.280355    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv"]
Jul 16 23:20:21.284445 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:20:21.284409    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85312b60_97ef_47b6_a26c_ca6e41e0fa4f.slice/crio-e7fc38a8c32bb2a98d254e6dec7b8464fbda53961680f02d7e44027d3e4bb8f5 WatchSource:0}: Error finding container e7fc38a8c32bb2a98d254e6dec7b8464fbda53961680f02d7e44027d3e4bb8f5: Status 404 returned error can't find the container with id e7fc38a8c32bb2a98d254e6dec7b8464fbda53961680f02d7e44027d3e4bb8f5
Jul 16 23:20:21.323972 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:21.323941    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv" event={"ID":"85312b60-97ef-47b6-a26c-ca6e41e0fa4f","Type":"ContainerStarted","Data":"e7fc38a8c32bb2a98d254e6dec7b8464fbda53961680f02d7e44027d3e4bb8f5"}
Jul 16 23:20:22.330224 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:22.330191    2565 generic.go:356] "Generic (PLEG): container finished" podID="85312b60-97ef-47b6-a26c-ca6e41e0fa4f" containerID="49fa7daedd9b31c13a6b8570276b3075ba2b20c97430d7e2f5a3df6310601037" exitCode=6
Jul 16 23:20:22.330729 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:22.330260    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv" event={"ID":"85312b60-97ef-47b6-a26c-ca6e41e0fa4f","Type":"ContainerDied","Data":"49fa7daedd9b31c13a6b8570276b3075ba2b20c97430d7e2f5a3df6310601037"}
Jul 16 23:20:22.330729 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:22.330485    2565 scope.go:117] "RemoveContainer" containerID="49fa7daedd9b31c13a6b8570276b3075ba2b20c97430d7e2f5a3df6310601037"
Jul 16 23:20:23.336807 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:23.336773    2565 generic.go:356] "Generic (PLEG): container finished" podID="85312b60-97ef-47b6-a26c-ca6e41e0fa4f" containerID="dded0e0dad2feedd255af9335868f2dbdc3c317e4bddff16af81c80932abe5ab" exitCode=6
Jul 16 23:20:23.337371 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:23.336851    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv" event={"ID":"85312b60-97ef-47b6-a26c-ca6e41e0fa4f","Type":"ContainerDied","Data":"dded0e0dad2feedd255af9335868f2dbdc3c317e4bddff16af81c80932abe5ab"}
Jul 16 23:20:23.337371 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:23.336903    2565 scope.go:117] "RemoveContainer" containerID="49fa7daedd9b31c13a6b8570276b3075ba2b20c97430d7e2f5a3df6310601037"
Jul 16 23:20:23.337371 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:23.337191    2565 scope.go:117] "RemoveContainer" containerID="dded0e0dad2feedd255af9335868f2dbdc3c317e4bddff16af81c80932abe5ab"
Jul 16 23:20:23.337555 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:20:23.337486    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv_odh-ai-gateway-infra(85312b60-97ef-47b6-a26c-ca6e41e0fa4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv" podUID="85312b60-97ef-47b6-a26c-ca6e41e0fa4f"
Jul 16 23:20:24.343798 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:24.343767    2565 scope.go:117] "RemoveContainer" containerID="dded0e0dad2feedd255af9335868f2dbdc3c317e4bddff16af81c80932abe5ab"
Jul 16 23:20:24.344218 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:20:24.343956    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv_odh-ai-gateway-infra(85312b60-97ef-47b6-a26c-ca6e41e0fa4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv" podUID="85312b60-97ef-47b6-a26c-ca6e41e0fa4f"
Jul 16 23:20:33.093717 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:33.093677    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5"]
Jul 16 23:20:33.094135 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:33.094072    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b338560-1c94-4a7c-a721-a3fcb871fa4c" containerName="revoke-keys"
Jul 16 23:20:33.094135 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:33.094084    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b338560-1c94-4a7c-a721-a3fcb871fa4c" containerName="revoke-keys"
Jul 16 23:20:33.094209 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:33.094167    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ae904ea-fa70-4363-8b2c-2e7346fc4277" containerName="revoke-keys"
Jul 16 23:20:33.094209 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:33.094178    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b338560-1c94-4a7c-a721-a3fcb871fa4c" containerName="revoke-keys"
Jul 16 23:20:33.098487 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:33.098469    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5"
Jul 16 23:20:33.104179 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:33.104155    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5"]
Jul 16 23:20:33.235300 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:33.235233    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-6m7d5\" (UID: \"57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5"
Jul 16 23:20:33.336602 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:33.336559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-6m7d5\" (UID: \"57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5"
Jul 16 23:20:33.337180 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:33.337157    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-6m7d5\" (UID: \"57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5"
Jul 16 23:20:33.410354 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:33.410271    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5"
Jul 16 23:20:33.552303 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:33.552270    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5"]
Jul 16 23:20:33.554897 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:20:33.554868    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c8f69f_7f0f_46fb_85d6_56e0fe0fa4ed.slice/crio-039ae2b24e6a626c98543b32366e35ee23299919e07b98175ec03e50f9fbd183 WatchSource:0}: Error finding container 039ae2b24e6a626c98543b32366e35ee23299919e07b98175ec03e50f9fbd183: Status 404 returned error can't find the container with id 039ae2b24e6a626c98543b32366e35ee23299919e07b98175ec03e50f9fbd183
Jul 16 23:20:34.391901 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:34.391864    2565 generic.go:356] "Generic (PLEG): container finished" podID="57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed" containerID="50bc4e295bf73d56a550b86757ad63ad2a0f32b13b1814e40d49ff8faeb88834" exitCode=6
Jul 16 23:20:34.392410 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:34.391952    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5" event={"ID":"57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed","Type":"ContainerDied","Data":"50bc4e295bf73d56a550b86757ad63ad2a0f32b13b1814e40d49ff8faeb88834"}
Jul 16 23:20:34.392410 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:34.392007    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5" event={"ID":"57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed","Type":"ContainerStarted","Data":"039ae2b24e6a626c98543b32366e35ee23299919e07b98175ec03e50f9fbd183"}
Jul 16 23:20:34.392410 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:34.392241    2565 scope.go:117] "RemoveContainer" containerID="50bc4e295bf73d56a550b86757ad63ad2a0f32b13b1814e40d49ff8faeb88834"
Jul 16 23:20:35.398720 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:35.398682    2565 generic.go:356] "Generic (PLEG): container finished" podID="57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed" containerID="abc65f6adc7832130743b364c886777f6bcb3fea03138857cd0d97a61946dd1b" exitCode=6
Jul 16 23:20:35.399154 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:35.398729    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5" event={"ID":"57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed","Type":"ContainerDied","Data":"abc65f6adc7832130743b364c886777f6bcb3fea03138857cd0d97a61946dd1b"}
Jul 16 23:20:35.399154 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:35.398794    2565 scope.go:117] "RemoveContainer" containerID="50bc4e295bf73d56a550b86757ad63ad2a0f32b13b1814e40d49ff8faeb88834"
Jul 16 23:20:35.399154 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:35.398952    2565 scope.go:117] "RemoveContainer" containerID="abc65f6adc7832130743b364c886777f6bcb3fea03138857cd0d97a61946dd1b"
Jul 16 23:20:35.399287 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:20:35.399192    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-6m7d5_odh-ai-gateway-infra(57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5" podUID="57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed"
Jul 16 23:20:36.405745 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:36.405708    2565 scope.go:117] "RemoveContainer" containerID="abc65f6adc7832130743b364c886777f6bcb3fea03138857cd0d97a61946dd1b"
Jul 16 23:20:36.406152 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:20:36.405946    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-6m7d5_odh-ai-gateway-infra(57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5" podUID="57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed"
Jul 16 23:20:38.037305 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:38.037267    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch"]
Jul 16 23:20:38.041928 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:38.041912    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch"
Jul 16 23:20:38.047963 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:38.047924    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch"]
Jul 16 23:20:38.181886 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:38.181841    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee49714-bd27-47b3-bb91-950bd8f1e408-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch\" (UID: \"dee49714-bd27-47b3-bb91-950bd8f1e408\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch"
Jul 16 23:20:38.282452 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:38.282409    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee49714-bd27-47b3-bb91-950bd8f1e408-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch\" (UID: \"dee49714-bd27-47b3-bb91-950bd8f1e408\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch"
Jul 16 23:20:38.283150 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:38.283123    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee49714-bd27-47b3-bb91-950bd8f1e408-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch\" (UID: \"dee49714-bd27-47b3-bb91-950bd8f1e408\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch"
Jul 16 23:20:38.354387 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:38.354350    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch"
Jul 16 23:20:38.485185 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:38.485141    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch"]
Jul 16 23:20:38.488002 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:20:38.487972    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee49714_bd27_47b3_bb91_950bd8f1e408.slice/crio-820e9eded21ce4f5481fb21dfce527a64a56ab9c12267b9dfef7f027f3525420 WatchSource:0}: Error finding container 820e9eded21ce4f5481fb21dfce527a64a56ab9c12267b9dfef7f027f3525420: Status 404 returned error can't find the container with id 820e9eded21ce4f5481fb21dfce527a64a56ab9c12267b9dfef7f027f3525420
Jul 16 23:20:39.423464 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:39.423432    2565 generic.go:356] "Generic (PLEG): container finished" podID="dee49714-bd27-47b3-bb91-950bd8f1e408" containerID="670ad63ac979f3e2a385c5eaf176a8cfc8cd61288f7ea7f46cd5cca38d8187cf" exitCode=6
Jul 16 23:20:39.423886 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:39.423522    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch" event={"ID":"dee49714-bd27-47b3-bb91-950bd8f1e408","Type":"ContainerDied","Data":"670ad63ac979f3e2a385c5eaf176a8cfc8cd61288f7ea7f46cd5cca38d8187cf"}
Jul 16 23:20:39.423886 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:39.423587    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch" event={"ID":"dee49714-bd27-47b3-bb91-950bd8f1e408","Type":"ContainerStarted","Data":"820e9eded21ce4f5481fb21dfce527a64a56ab9c12267b9dfef7f027f3525420"}
Jul 16 23:20:39.423886 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:39.423735    2565 scope.go:117] "RemoveContainer" containerID="670ad63ac979f3e2a385c5eaf176a8cfc8cd61288f7ea7f46cd5cca38d8187cf"
Jul 16 23:20:39.726476 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:39.726402    2565 scope.go:117] "RemoveContainer" containerID="dded0e0dad2feedd255af9335868f2dbdc3c317e4bddff16af81c80932abe5ab"
Jul 16 23:20:40.430315 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:40.430282    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv" event={"ID":"85312b60-97ef-47b6-a26c-ca6e41e0fa4f","Type":"ContainerStarted","Data":"949e9406f4c27ef114e1d699943d4dfc9393cf0010b86da72f3527aa177ea64d"}
Jul 16 23:20:40.431981 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:40.431955    2565 generic.go:356] "Generic (PLEG): container finished" podID="dee49714-bd27-47b3-bb91-950bd8f1e408" containerID="d60d477d0663fe9a088bc64243db3399089068117fb8cf71b1b2e1adc1fa5ca1" exitCode=6
Jul 16 23:20:40.432116 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:40.431988    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch" event={"ID":"dee49714-bd27-47b3-bb91-950bd8f1e408","Type":"ContainerDied","Data":"d60d477d0663fe9a088bc64243db3399089068117fb8cf71b1b2e1adc1fa5ca1"}
Jul 16 23:20:40.432116 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:40.432038    2565 scope.go:117] "RemoveContainer" containerID="670ad63ac979f3e2a385c5eaf176a8cfc8cd61288f7ea7f46cd5cca38d8187cf"
Jul 16 23:20:40.432223 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:40.432192    2565 scope.go:117] "RemoveContainer" containerID="d60d477d0663fe9a088bc64243db3399089068117fb8cf71b1b2e1adc1fa5ca1"
Jul 16 23:20:40.432458 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:20:40.432436    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch_odh-ai-gateway-infra(dee49714-bd27-47b3-bb91-950bd8f1e408)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch" podUID="dee49714-bd27-47b3-bb91-950bd8f1e408"
Jul 16 23:20:40.448172 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:40.448123    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv" podStartSLOduration=20.448104843 podStartE2EDuration="20.448104843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:20:40.444669196 +0000 UTC m=+2851.193153268" watchObservedRunningTime="2026-07-16 23:20:40.448104843 +0000 UTC m=+2851.196588917"
Jul 16 23:20:41.438298 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:41.438269    2565 scope.go:117] "RemoveContainer" containerID="d60d477d0663fe9a088bc64243db3399089068117fb8cf71b1b2e1adc1fa5ca1"
Jul 16 23:20:41.438772 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:20:41.438534    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch_odh-ai-gateway-infra(dee49714-bd27-47b3-bb91-950bd8f1e408)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch" podUID="dee49714-bd27-47b3-bb91-950bd8f1e408"
Jul 16 23:20:41.439795 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:41.439772    2565 generic.go:356] "Generic (PLEG): container finished" podID="85312b60-97ef-47b6-a26c-ca6e41e0fa4f" containerID="949e9406f4c27ef114e1d699943d4dfc9393cf0010b86da72f3527aa177ea64d" exitCode=6
Jul 16 23:20:41.439886 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:41.439843    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv" event={"ID":"85312b60-97ef-47b6-a26c-ca6e41e0fa4f","Type":"ContainerDied","Data":"949e9406f4c27ef114e1d699943d4dfc9393cf0010b86da72f3527aa177ea64d"}
Jul 16 23:20:41.439927 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:41.439898    2565 scope.go:117] "RemoveContainer" containerID="dded0e0dad2feedd255af9335868f2dbdc3c317e4bddff16af81c80932abe5ab"
Jul 16 23:20:41.440137 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:41.440122    2565 scope.go:117] "RemoveContainer" containerID="949e9406f4c27ef114e1d699943d4dfc9393cf0010b86da72f3527aa177ea64d"
Jul 16 23:20:41.440380 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:20:41.440358    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv_odh-ai-gateway-infra(85312b60-97ef-47b6-a26c-ca6e41e0fa4f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv" podUID="85312b60-97ef-47b6-a26c-ca6e41e0fa4f"
Jul 16 23:20:41.458636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:41.458611    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv"]
Jul 16 23:20:42.581469 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:42.581440    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv"
Jul 16 23:20:42.726008 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:42.725916    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85312b60-97ef-47b6-a26c-ca6e41e0fa4f-maas-api-service-ca\") pod \"85312b60-97ef-47b6-a26c-ca6e41e0fa4f\" (UID: \"85312b60-97ef-47b6-a26c-ca6e41e0fa4f\") "
Jul 16 23:20:42.726267 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:42.726226    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85312b60-97ef-47b6-a26c-ca6e41e0fa4f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "85312b60-97ef-47b6-a26c-ca6e41e0fa4f" (UID: "85312b60-97ef-47b6-a26c-ca6e41e0fa4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:20:42.827712 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:42.827671    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85312b60-97ef-47b6-a26c-ca6e41e0fa4f-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:20:43.453369 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:43.453342    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv"
Jul 16 23:20:43.453564 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:43.453340    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv" event={"ID":"85312b60-97ef-47b6-a26c-ca6e41e0fa4f","Type":"ContainerDied","Data":"e7fc38a8c32bb2a98d254e6dec7b8464fbda53961680f02d7e44027d3e4bb8f5"}
Jul 16 23:20:43.453564 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:43.453475    2565 scope.go:117] "RemoveContainer" containerID="949e9406f4c27ef114e1d699943d4dfc9393cf0010b86da72f3527aa177ea64d"
Jul 16 23:20:43.479522 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:43.479487    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv"]
Jul 16 23:20:43.487447 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:43.487404    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hhqsv"]
Jul 16 23:20:43.726143 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:43.726063    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85312b60-97ef-47b6-a26c-ca6e41e0fa4f" path="/var/lib/kubelet/pods/85312b60-97ef-47b6-a26c-ca6e41e0fa4f/volumes"
Jul 16 23:20:46.721216 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:46.721183    2565 scope.go:117] "RemoveContainer" containerID="abc65f6adc7832130743b364c886777f6bcb3fea03138857cd0d97a61946dd1b"
Jul 16 23:20:47.073769 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.073683    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"]
Jul 16 23:20:47.074326 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.074307    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85312b60-97ef-47b6-a26c-ca6e41e0fa4f" containerName="revoke-keys"
Jul 16 23:20:47.074377 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.074329    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="85312b60-97ef-47b6-a26c-ca6e41e0fa4f" containerName="revoke-keys"
Jul 16 23:20:47.074377 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.074345    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85312b60-97ef-47b6-a26c-ca6e41e0fa4f" containerName="revoke-keys"
Jul 16 23:20:47.074377 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.074354    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="85312b60-97ef-47b6-a26c-ca6e41e0fa4f" containerName="revoke-keys"
Jul 16 23:20:47.074488 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.074463    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="85312b60-97ef-47b6-a26c-ca6e41e0fa4f" containerName="revoke-keys"
Jul 16 23:20:47.074488 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.074478    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="85312b60-97ef-47b6-a26c-ca6e41e0fa4f" containerName="revoke-keys"
Jul 16 23:20:47.077741 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.077721    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.080151 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.080130    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8a2515-openshift-default-dockercfg-knqlx\""
Jul 16 23:20:47.087921 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.087897    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"]
Jul 16 23:20:47.169686 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.169648    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-token\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.169994 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.169964    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-credential-socket\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.170526 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.170010    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl5v8\" (UniqueName: \"kubernetes.io/projected/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-kube-api-access-hl5v8\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.170526 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.170050    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istiod-ca-cert\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.170526 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.170089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-workload-socket\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.170526 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.170130    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-podinfo\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.170526 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.170180    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-data\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.170526 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.170292    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-workload-certs\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.170526 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.170318    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-envoy\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.271680 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.271640    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-token\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.271873 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.271688    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-credential-socket\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.271873 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.271821    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl5v8\" (UniqueName: \"kubernetes.io/projected/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-kube-api-access-hl5v8\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.271945 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.271871    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istiod-ca-cert\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.271945 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.271901    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-workload-socket\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.271945 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.271932    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-podinfo\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.272088 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.271980    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-data\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.272088 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.272057    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-workload-certs\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.272088 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.272084    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-envoy\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.272266 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.272126    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-credential-socket\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.272375 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.272356    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-workload-socket\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.272445 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.272409    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-data\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.272501 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.272475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-workload-certs\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.272688 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.272666    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istiod-ca-cert\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.274333 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.274315    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-envoy\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.274738 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.274579    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-podinfo\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.281651 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.281583    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl5v8\" (UniqueName: \"kubernetes.io/projected/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-kube-api-access-hl5v8\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.283875 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.283854    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-token\") pod \"e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") " pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.388995 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.388957    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:47.479402 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.479365    2565 generic.go:356] "Generic (PLEG): container finished" podID="57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed" containerID="49a395e16757fa3141090928a5907334b88558223da0f418bf1b432aec53eba5" exitCode=6
Jul 16 23:20:47.479575 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.479444    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5" event={"ID":"57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed","Type":"ContainerDied","Data":"49a395e16757fa3141090928a5907334b88558223da0f418bf1b432aec53eba5"}
Jul 16 23:20:47.479575 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.479554    2565 scope.go:117] "RemoveContainer" containerID="abc65f6adc7832130743b364c886777f6bcb3fea03138857cd0d97a61946dd1b"
Jul 16 23:20:47.479866 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.479849    2565 scope.go:117] "RemoveContainer" containerID="49a395e16757fa3141090928a5907334b88558223da0f418bf1b432aec53eba5"
Jul 16 23:20:47.480102 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:20:47.480080    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-6m7d5_odh-ai-gateway-infra(57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5" podUID="57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed"
Jul 16 23:20:47.545916 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.545802    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"]
Jul 16 23:20:47.548560 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:20:47.548520    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7678090d_6d22_4f7c_bc5e_b4f06b4f35d8.slice/crio-4687ee45ab95bf3e1db18c3d5c855b63438e86a7844426fa849b37cf5441b0bf WatchSource:0}: Error finding container 4687ee45ab95bf3e1db18c3d5c855b63438e86a7844426fa849b37cf5441b0bf: Status 404 returned error can't find the container with id 4687ee45ab95bf3e1db18c3d5c855b63438e86a7844426fa849b37cf5441b0bf
Jul 16 23:20:47.550889 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.550848    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:20:47.550972 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.550931    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:20:47.551023 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:47.550994    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:20:48.487414 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:48.487378    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7" event={"ID":"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8","Type":"ContainerStarted","Data":"c0d6bb527be08fc70ffd4ba3df9121af4ae9ca352c689d4ea33b7162abcb2d0b"}
Jul 16 23:20:48.487839 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:48.487434    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7" event={"ID":"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8","Type":"ContainerStarted","Data":"4687ee45ab95bf3e1db18c3d5c855b63438e86a7844426fa849b37cf5441b0bf"}
Jul 16 23:20:48.509165 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:48.509104    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7" podStartSLOduration=1.509089264 podStartE2EDuration="1.509089264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:20:48.507339057 +0000 UTC m=+2859.255823129" watchObservedRunningTime="2026-07-16 23:20:48.509089264 +0000 UTC m=+2859.257573335"
Jul 16 23:20:48.512629 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:48.512599    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5"]
Jul 16 23:20:48.656678 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:48.656654    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5"
Jul 16 23:20:48.787056 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:48.786960    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed-maas-api-service-ca\") pod \"57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed\" (UID: \"57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed\") "
Jul 16 23:20:48.787372 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:48.787350    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed" (UID: "57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:20:48.888826 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:48.888779    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:20:49.390018 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:49.389974    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:49.395289 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:49.395260    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:49.493156 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:49.493128    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5"
Jul 16 23:20:49.493156 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:49.493137    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5" event={"ID":"57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed","Type":"ContainerDied","Data":"039ae2b24e6a626c98543b32366e35ee23299919e07b98175ec03e50f9fbd183"}
Jul 16 23:20:49.493766 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:49.493197    2565 scope.go:117] "RemoveContainer" containerID="49a395e16757fa3141090928a5907334b88558223da0f418bf1b432aec53eba5"
Jul 16 23:20:49.493766 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:49.493586    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:49.494904 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:49.494889    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:20:49.533450 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:49.533416    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5"]
Jul 16 23:20:49.535402 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:49.535371    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-6m7d5"]
Jul 16 23:20:49.726788 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:49.726704    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed" path="/var/lib/kubelet/pods/57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed/volumes"
Jul 16 23:20:50.545591 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.545552    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"]
Jul 16 23:20:50.546101 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.546082    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed" containerName="revoke-keys"
Jul 16 23:20:50.546197 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.546105    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed" containerName="revoke-keys"
Jul 16 23:20:50.546197 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.546130    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85312b60-97ef-47b6-a26c-ca6e41e0fa4f" containerName="revoke-keys"
Jul 16 23:20:50.546197 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.546140    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="85312b60-97ef-47b6-a26c-ca6e41e0fa4f" containerName="revoke-keys"
Jul 16 23:20:50.546197 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.546163    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed" containerName="revoke-keys"
Jul 16 23:20:50.546197 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.546171    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed" containerName="revoke-keys"
Jul 16 23:20:50.546466 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.546283    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed" containerName="revoke-keys"
Jul 16 23:20:50.546466 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.546296    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed" containerName="revoke-keys"
Jul 16 23:20:50.546466 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.546304    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed" containerName="revoke-keys"
Jul 16 23:20:50.546466 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.546314    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="85312b60-97ef-47b6-a26c-ca6e41e0fa4f" containerName="revoke-keys"
Jul 16 23:20:50.550630 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.550601    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"
Jul 16 23:20:50.552877 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.552853    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8a2515-serving-cert\""
Jul 16 23:20:50.559816 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.559790    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"]
Jul 16 23:20:50.706160 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.706124    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-maas-api-tls\") pod \"maas-api-e2e-inf-a-8a2515-659686898b-s9x8k\" (UID: \"ca9a6e33-0b3f-4704-a13f-7ead17a7f967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"
Jul 16 23:20:50.706369 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.706179    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh8gp\" (UniqueName: \"kubernetes.io/projected/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-kube-api-access-nh8gp\") pod \"maas-api-e2e-inf-a-8a2515-659686898b-s9x8k\" (UID: \"ca9a6e33-0b3f-4704-a13f-7ead17a7f967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"
Jul 16 23:20:50.706369 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.706235    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8a2515-659686898b-s9x8k\" (UID: \"ca9a6e33-0b3f-4704-a13f-7ead17a7f967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"
Jul 16 23:20:50.806901 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.806803    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-maas-api-tls\") pod \"maas-api-e2e-inf-a-8a2515-659686898b-s9x8k\" (UID: \"ca9a6e33-0b3f-4704-a13f-7ead17a7f967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"
Jul 16 23:20:50.806901 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.806851    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh8gp\" (UniqueName: \"kubernetes.io/projected/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-kube-api-access-nh8gp\") pod \"maas-api-e2e-inf-a-8a2515-659686898b-s9x8k\" (UID: \"ca9a6e33-0b3f-4704-a13f-7ead17a7f967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"
Jul 16 23:20:50.806901 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.806883    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8a2515-659686898b-s9x8k\" (UID: \"ca9a6e33-0b3f-4704-a13f-7ead17a7f967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"
Jul 16 23:20:50.807160 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:20:50.807091    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-serving-cert: secret "maas-api-e2e-inf-a-8a2515-serving-cert" not found
Jul 16 23:20:50.807208 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:20:50.807167    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-maas-api-tls podName:ca9a6e33-0b3f-4704-a13f-7ead17a7f967 nodeName:}" failed. No retries permitted until 2026-07-16 23:20:51.307147639 +0000 UTC m=+2862.055631700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-maas-api-tls") pod "maas-api-e2e-inf-a-8a2515-659686898b-s9x8k" (UID: "ca9a6e33-0b3f-4704-a13f-7ead17a7f967") : secret "maas-api-e2e-inf-a-8a2515-serving-cert" not found
Jul 16 23:20:50.807571 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.807549    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8a2515-659686898b-s9x8k\" (UID: \"ca9a6e33-0b3f-4704-a13f-7ead17a7f967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"
Jul 16 23:20:50.818999 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:50.818969    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh8gp\" (UniqueName: \"kubernetes.io/projected/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-kube-api-access-nh8gp\") pod \"maas-api-e2e-inf-a-8a2515-659686898b-s9x8k\" (UID: \"ca9a6e33-0b3f-4704-a13f-7ead17a7f967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"
Jul 16 23:20:51.312549 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:51.312497    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-maas-api-tls\") pod \"maas-api-e2e-inf-a-8a2515-659686898b-s9x8k\" (UID: \"ca9a6e33-0b3f-4704-a13f-7ead17a7f967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"
Jul 16 23:20:51.314966 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:51.314933    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-maas-api-tls\") pod \"maas-api-e2e-inf-a-8a2515-659686898b-s9x8k\" (UID: \"ca9a6e33-0b3f-4704-a13f-7ead17a7f967\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"
Jul 16 23:20:51.462277 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:51.462210    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"
Jul 16 23:20:51.610150 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:51.610119    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"]
Jul 16 23:20:51.612732 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:20:51.612699    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9a6e33_0b3f_4704_a13f_7ead17a7f967.slice/crio-3026db3cf40f53819d56ac7513e6e82911615bb8847b914713e0f71d392d7149 WatchSource:0}: Error finding container 3026db3cf40f53819d56ac7513e6e82911615bb8847b914713e0f71d392d7149: Status 404 returned error can't find the container with id 3026db3cf40f53819d56ac7513e6e82911615bb8847b914713e0f71d392d7149
Jul 16 23:20:52.521086 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:52.520987    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k" event={"ID":"ca9a6e33-0b3f-4704-a13f-7ead17a7f967","Type":"ContainerStarted","Data":"978c08c2676007b2a520751ff9d40cc8af2f568ded1c1bbd938802050f56bb1d"}
Jul 16 23:20:52.521086 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:52.521041    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k" event={"ID":"ca9a6e33-0b3f-4704-a13f-7ead17a7f967","Type":"ContainerStarted","Data":"3026db3cf40f53819d56ac7513e6e82911615bb8847b914713e0f71d392d7149"}
Jul 16 23:20:52.521338 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:52.521099    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"
Jul 16 23:20:52.539613 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:52.539561    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k" podStartSLOduration=1.924003457 podStartE2EDuration="2.539544624s" totalImagesPullingTime="615.541167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:20:52.536982628 +0000 UTC m=+2863.285466712" watchObservedRunningTime="2026-07-16 23:20:52.539544624 +0000 UTC m=+2863.288028695"
Jul 16 23:20:53.721177 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:53.721144    2565 scope.go:117] "RemoveContainer" containerID="d60d477d0663fe9a088bc64243db3399089068117fb8cf71b1b2e1adc1fa5ca1"
Jul 16 23:20:54.532308 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:54.532267    2565 generic.go:356] "Generic (PLEG): container finished" podID="dee49714-bd27-47b3-bb91-950bd8f1e408" containerID="af3f61bb53c9f7bc8ed31d2072aba7f4069638bd7ec22a983dda63b925bdbb0c" exitCode=6
Jul 16 23:20:54.532502 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:54.532281    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch" event={"ID":"dee49714-bd27-47b3-bb91-950bd8f1e408","Type":"ContainerDied","Data":"af3f61bb53c9f7bc8ed31d2072aba7f4069638bd7ec22a983dda63b925bdbb0c"}
Jul 16 23:20:54.532502 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:54.532395    2565 scope.go:117] "RemoveContainer" containerID="d60d477d0663fe9a088bc64243db3399089068117fb8cf71b1b2e1adc1fa5ca1"
Jul 16 23:20:54.532676 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:54.532656    2565 scope.go:117] "RemoveContainer" containerID="af3f61bb53c9f7bc8ed31d2072aba7f4069638bd7ec22a983dda63b925bdbb0c"
Jul 16 23:20:54.532930 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:20:54.532904    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch_odh-ai-gateway-infra(dee49714-bd27-47b3-bb91-950bd8f1e408)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch" podUID="dee49714-bd27-47b3-bb91-950bd8f1e408"
Jul 16 23:20:55.455475 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.455434    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"]
Jul 16 23:20:55.456021 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.456006    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed" containerName="revoke-keys"
Jul 16 23:20:55.456075 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.456025    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="57c8f69f-7f0f-46fb-85d6-56e0fe0fa4ed" containerName="revoke-keys"
Jul 16 23:20:55.460624 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.460599    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.462774 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.462751    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4d775e-openshift-default-dockercfg-ngvd2\""
Jul 16 23:20:55.469547 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.469518    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"]
Jul 16 23:20:55.559582 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.559545    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch"]
Jul 16 23:20:55.651940 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.651902    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fd15e8-f607-449d-a7e3-10a8fa205782-istiod-ca-cert\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.652130 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.651952    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-data\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.652130 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.651968    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-token\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.652130 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.652021    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-credential-socket\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.652130 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.652114    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-workload-socket\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.652324 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.652159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-podinfo\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.652324 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.652221    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7ds4\" (UniqueName: \"kubernetes.io/projected/c3fd15e8-f607-449d-a7e3-10a8fa205782-kube-api-access-m7ds4\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.652324 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.652269    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-workload-certs\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.652324 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.652317    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-envoy\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.707293 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.707205    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch"
Jul 16 23:20:55.753462 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.753418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-envoy\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.753678 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.753500    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fd15e8-f607-449d-a7e3-10a8fa205782-istiod-ca-cert\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.753678 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.753530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-data\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.753678 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.753545    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-token\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.753678 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.753564    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-credential-socket\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.753678 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.753611    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-workload-socket\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.753678 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.753644    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-podinfo\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.754032 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.753700    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7ds4\" (UniqueName: \"kubernetes.io/projected/c3fd15e8-f607-449d-a7e3-10a8fa205782-kube-api-access-m7ds4\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.754032 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.753726    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-workload-certs\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.754032 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.753963    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-data\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.754204 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.754034    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-workload-certs\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.754286 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.754235    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-workload-socket\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.754421 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.754378    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fd15e8-f607-449d-a7e3-10a8fa205782-istiod-ca-cert\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.754615 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.754591    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-credential-socket\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.756572 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.756519    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-envoy\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.756894 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.756872    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-podinfo\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.765269 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.765212    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-token\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.765415 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.765389    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7ds4\" (UniqueName: \"kubernetes.io/projected/c3fd15e8-f607-449d-a7e3-10a8fa205782-kube-api-access-m7ds4\") pod \"e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") " pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.773208 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.773180    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:55.854542 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.854499    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee49714-bd27-47b3-bb91-950bd8f1e408-maas-api-service-ca\") pod \"dee49714-bd27-47b3-bb91-950bd8f1e408\" (UID: \"dee49714-bd27-47b3-bb91-950bd8f1e408\") "
Jul 16 23:20:55.855761 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.854961    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dee49714-bd27-47b3-bb91-950bd8f1e408-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dee49714-bd27-47b3-bb91-950bd8f1e408" (UID: "dee49714-bd27-47b3-bb91-950bd8f1e408"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:20:55.855761 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.855454    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dee49714-bd27-47b3-bb91-950bd8f1e408-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:20:55.921704 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.921677    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"]
Jul 16 23:20:55.924125 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:20:55.924095    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3fd15e8_f607_449d_a7e3_10a8fa205782.slice/crio-82050c5a8cb769d7826fc5980507901bebae61704b7b7dd6ec709a2f0e245eed WatchSource:0}: Error finding container 82050c5a8cb769d7826fc5980507901bebae61704b7b7dd6ec709a2f0e245eed: Status 404 returned error can't find the container with id 82050c5a8cb769d7826fc5980507901bebae61704b7b7dd6ec709a2f0e245eed
Jul 16 23:20:55.926366 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.926334    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:20:55.926422 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.926402    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:20:55.926455 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:55.926430    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:20:56.553590 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:56.553550    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9" event={"ID":"c3fd15e8-f607-449d-a7e3-10a8fa205782","Type":"ContainerStarted","Data":"940f5c1a5bf42b7a0ce1cf48a2629c375d35c48f9322ab1b7a099da489a7cbba"}
Jul 16 23:20:56.554075 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:56.553653    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9" event={"ID":"c3fd15e8-f607-449d-a7e3-10a8fa205782","Type":"ContainerStarted","Data":"82050c5a8cb769d7826fc5980507901bebae61704b7b7dd6ec709a2f0e245eed"}
Jul 16 23:20:56.554974 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:56.554944    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch" event={"ID":"dee49714-bd27-47b3-bb91-950bd8f1e408","Type":"ContainerDied","Data":"820e9eded21ce4f5481fb21dfce527a64a56ab9c12267b9dfef7f027f3525420"}
Jul 16 23:20:56.554974 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:56.554967    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch"
Jul 16 23:20:56.555153 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:56.555009    2565 scope.go:117] "RemoveContainer" containerID="af3f61bb53c9f7bc8ed31d2072aba7f4069638bd7ec22a983dda63b925bdbb0c"
Jul 16 23:20:56.576715 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:56.576663    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9" podStartSLOduration=1.576644959 podStartE2EDuration="1.576644959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:20:56.573968806 +0000 UTC m=+2867.322452878" watchObservedRunningTime="2026-07-16 23:20:56.576644959 +0000 UTC m=+2867.325129031"
Jul 16 23:20:56.586426 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:56.586385    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch"]
Jul 16 23:20:56.588913 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:56.588884    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-zh8ch"]
Jul 16 23:20:56.773602 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:56.773558    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:56.778691 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:56.778663    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:57.561203 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:57.561163    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:57.562273 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:57.562233    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:20:57.726477 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:57.726441    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dee49714-bd27-47b3-bb91-950bd8f1e408" path="/var/lib/kubelet/pods/dee49714-bd27-47b3-bb91-950bd8f1e408/volumes"
Jul 16 23:20:58.531651 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.531621    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"
Jul 16 23:20:58.637108 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.637075    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"]
Jul 16 23:20:58.637612 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.637597    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee49714-bd27-47b3-bb91-950bd8f1e408" containerName="revoke-keys"
Jul 16 23:20:58.637612 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.637613    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee49714-bd27-47b3-bb91-950bd8f1e408" containerName="revoke-keys"
Jul 16 23:20:58.637699 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.637623    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee49714-bd27-47b3-bb91-950bd8f1e408" containerName="revoke-keys"
Jul 16 23:20:58.637699 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.637629    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee49714-bd27-47b3-bb91-950bd8f1e408" containerName="revoke-keys"
Jul 16 23:20:58.637699 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.637644    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee49714-bd27-47b3-bb91-950bd8f1e408" containerName="revoke-keys"
Jul 16 23:20:58.637699 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.637651    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee49714-bd27-47b3-bb91-950bd8f1e408" containerName="revoke-keys"
Jul 16 23:20:58.637821 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.637712    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dee49714-bd27-47b3-bb91-950bd8f1e408" containerName="revoke-keys"
Jul 16 23:20:58.637821 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.637723    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dee49714-bd27-47b3-bb91-950bd8f1e408" containerName="revoke-keys"
Jul 16 23:20:58.642348 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.642321    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"
Jul 16 23:20:58.644694 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.644665    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4d775e-serving-cert\""
Jul 16 23:20:58.649968 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.649898    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"]
Jul 16 23:20:58.678546 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.678508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a66218-1bfc-40a1-9421-1a6c31893502-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn\" (UID: \"36a66218-1bfc-40a1-9421-1a6c31893502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"
Jul 16 23:20:58.678743 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.678560    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36a66218-1bfc-40a1-9421-1a6c31893502-maas-api-tls\") pod \"maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn\" (UID: \"36a66218-1bfc-40a1-9421-1a6c31893502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"
Jul 16 23:20:58.678743 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.678719    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s94sg\" (UniqueName: \"kubernetes.io/projected/36a66218-1bfc-40a1-9421-1a6c31893502-kube-api-access-s94sg\") pod \"maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn\" (UID: \"36a66218-1bfc-40a1-9421-1a6c31893502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"
Jul 16 23:20:58.779823 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.779776    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s94sg\" (UniqueName: \"kubernetes.io/projected/36a66218-1bfc-40a1-9421-1a6c31893502-kube-api-access-s94sg\") pod \"maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn\" (UID: \"36a66218-1bfc-40a1-9421-1a6c31893502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"
Jul 16 23:20:58.779823 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.779830    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a66218-1bfc-40a1-9421-1a6c31893502-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn\" (UID: \"36a66218-1bfc-40a1-9421-1a6c31893502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"
Jul 16 23:20:58.780083 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.779875    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36a66218-1bfc-40a1-9421-1a6c31893502-maas-api-tls\") pod \"maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn\" (UID: \"36a66218-1bfc-40a1-9421-1a6c31893502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"
Jul 16 23:20:58.780083 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:20:58.780047    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-serving-cert: secret "maas-api-e2e-inf-b-4d775e-serving-cert" not found
Jul 16 23:20:58.780163 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:20:58.780121    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36a66218-1bfc-40a1-9421-1a6c31893502-maas-api-tls podName:36a66218-1bfc-40a1-9421-1a6c31893502 nodeName:}" failed. No retries permitted until 2026-07-16 23:20:59.280100376 +0000 UTC m=+2870.028584425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36a66218-1bfc-40a1-9421-1a6c31893502-maas-api-tls") pod "maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn" (UID: "36a66218-1bfc-40a1-9421-1a6c31893502") : secret "maas-api-e2e-inf-b-4d775e-serving-cert" not found
Jul 16 23:20:58.780647 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.780626    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a66218-1bfc-40a1-9421-1a6c31893502-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn\" (UID: \"36a66218-1bfc-40a1-9421-1a6c31893502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"
Jul 16 23:20:58.788545 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:58.788492    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s94sg\" (UniqueName: \"kubernetes.io/projected/36a66218-1bfc-40a1-9421-1a6c31893502-kube-api-access-s94sg\") pod \"maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn\" (UID: \"36a66218-1bfc-40a1-9421-1a6c31893502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"
Jul 16 23:20:59.285334 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:59.285295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36a66218-1bfc-40a1-9421-1a6c31893502-maas-api-tls\") pod \"maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn\" (UID: \"36a66218-1bfc-40a1-9421-1a6c31893502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"
Jul 16 23:20:59.287847 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:59.287818    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36a66218-1bfc-40a1-9421-1a6c31893502-maas-api-tls\") pod \"maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn\" (UID: \"36a66218-1bfc-40a1-9421-1a6c31893502\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"
Jul 16 23:20:59.555384 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:59.555293    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"
Jul 16 23:20:59.716177 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:20:59.716140    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"]
Jul 16 23:20:59.719828 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:20:59.719794    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a66218_1bfc_40a1_9421_1a6c31893502.slice/crio-8f431a3b640a76da3549e0a98514e79e4a6ac040e6e463701edaf270d9acbc15 WatchSource:0}: Error finding container 8f431a3b640a76da3549e0a98514e79e4a6ac040e6e463701edaf270d9acbc15: Status 404 returned error can't find the container with id 8f431a3b640a76da3549e0a98514e79e4a6ac040e6e463701edaf270d9acbc15
Jul 16 23:21:00.577392 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:00.577359    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn" event={"ID":"36a66218-1bfc-40a1-9421-1a6c31893502","Type":"ContainerStarted","Data":"0ac663b439760251e7040950c2f39d9ace6963f35764536776bd5453488ea485"}
Jul 16 23:21:00.577595 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:00.577412    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn" event={"ID":"36a66218-1bfc-40a1-9421-1a6c31893502","Type":"ContainerStarted","Data":"8f431a3b640a76da3549e0a98514e79e4a6ac040e6e463701edaf270d9acbc15"}
Jul 16 23:21:00.577595 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:00.577463    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"
Jul 16 23:21:00.594169 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:00.594116    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn" podStartSLOduration=2.112812985 podStartE2EDuration="2.594100483s" totalImagesPullingTime="481.287498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:21:00.592212511 +0000 UTC m=+2871.340696582" watchObservedRunningTime="2026-07-16 23:21:00.594100483 +0000 UTC m=+2871.342584567"
Jul 16 23:21:05.494686 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:05.494634    2565 patch_prober.go:28] interesting pod/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 16 23:21:05.495119 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:05.494719    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7" podUID="7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 16 23:21:06.588509 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:06.588476    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"
Jul 16 23:21:07.615487 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:07.615449    2565 generic.go:356] "Generic (PLEG): container finished" podID="7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" containerID="c0d6bb527be08fc70ffd4ba3df9121af4ae9ca352c689d4ea33b7162abcb2d0b" exitCode=0
Jul 16 23:21:07.615933 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:07.615526    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7" event={"ID":"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8","Type":"ContainerDied","Data":"c0d6bb527be08fc70ffd4ba3df9121af4ae9ca352c689d4ea33b7162abcb2d0b"}
Jul 16 23:21:07.615984 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:07.615938    2565 scope.go:117] "RemoveContainer" containerID="c0d6bb527be08fc70ffd4ba3df9121af4ae9ca352c689d4ea33b7162abcb2d0b"
Jul 16 23:21:07.616992 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:07.616962    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:21:07.617093 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:07.617058    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:21:07.617156 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:07.617116    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:21:08.622859 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:08.622826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7" event={"ID":"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8","Type":"ContainerStarted","Data":"7c4e8df7a168c0e12985271dd5583fb9af7d80446b7bf377819b03f5090cdf0c"}
Jul 16 23:21:09.389262 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:09.389217    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:21:09.389573 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:09.389549    2565 patch_prober.go:28] interesting pod/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Jul 16 23:21:09.389715 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:09.389692    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7" podUID="7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Jul 16 23:21:10.389670 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:10.389571    2565 patch_prober.go:28] interesting pod/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Jul 16 23:21:10.389670 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:10.389632    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7" podUID="7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Jul 16 23:21:11.389745 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:11.389707    2565 patch_prober.go:28] interesting pod/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Jul 16 23:21:11.390142 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:11.389775    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7" podUID="7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Jul 16 23:21:12.390103 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:12.390058    2565 patch_prober.go:28] interesting pod/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Jul 16 23:21:12.390530 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:12.390124    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7" podUID="7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Jul 16 23:21:12.928455 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:12.927930    2565 patch_prober.go:28] interesting pod/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": read tcp 10.132.0.2:40534->10.132.0.71:15021: read: connection reset by peer" start-of-body=
Jul 16 23:21:12.928455 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:12.927998    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9" podUID="c3fd15e8-f607-449d-a7e3-10a8fa205782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": read tcp 10.132.0.2:40534->10.132.0.71:15021: read: connection reset by peer"
Jul 16 23:21:13.392613 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:13.392581    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:21:13.393102 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:13.392967    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:21:13.393658 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:13.393636    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:21:13.653160 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:13.653073    2565 generic.go:356] "Generic (PLEG): container finished" podID="c3fd15e8-f607-449d-a7e3-10a8fa205782" containerID="940f5c1a5bf42b7a0ce1cf48a2629c375d35c48f9322ab1b7a099da489a7cbba" exitCode=0
Jul 16 23:21:13.653350 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:13.653149    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9" event={"ID":"c3fd15e8-f607-449d-a7e3-10a8fa205782","Type":"ContainerDied","Data":"940f5c1a5bf42b7a0ce1cf48a2629c375d35c48f9322ab1b7a099da489a7cbba"}
Jul 16 23:21:13.653703 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:13.653684    2565 scope.go:117] "RemoveContainer" containerID="940f5c1a5bf42b7a0ce1cf48a2629c375d35c48f9322ab1b7a099da489a7cbba"
Jul 16 23:21:13.654740 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:13.654713    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:21:13.654798 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:13.654785    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:21:13.654835 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:13.654826    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:21:14.660863 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:14.660829    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9" event={"ID":"c3fd15e8-f607-449d-a7e3-10a8fa205782","Type":"ContainerStarted","Data":"f2cd5cf4c3717e730b1d76034d40dd250cbe8c5a757ad92bd58ea3c1e0b301c2"}
Jul 16 23:21:14.773842 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:14.773804    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:21:14.774059 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:14.774007    2565 patch_prober.go:28] interesting pod/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Jul 16 23:21:14.774139 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:14.774064    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9" podUID="c3fd15e8-f607-449d-a7e3-10a8fa205782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Jul 16 23:21:15.774050 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:15.774010    2565 patch_prober.go:28] interesting pod/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Jul 16 23:21:15.774497 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:15.774068    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9" podUID="c3fd15e8-f607-449d-a7e3-10a8fa205782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Jul 16 23:21:16.774588 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:16.774535    2565 patch_prober.go:28] interesting pod/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Jul 16 23:21:16.775101 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:16.774621    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9" podUID="c3fd15e8-f607-449d-a7e3-10a8fa205782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Jul 16 23:21:17.773820 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:17.773782    2565 patch_prober.go:28] interesting pod/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Jul 16 23:21:17.774016 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:17.773838    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9" podUID="c3fd15e8-f607-449d-a7e3-10a8fa205782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Jul 16 23:21:18.774600 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:18.774556    2565 patch_prober.go:28] interesting pod/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Jul 16 23:21:18.775107 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:18.774627    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9" podUID="c3fd15e8-f607-449d-a7e3-10a8fa205782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Jul 16 23:21:19.777347 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:19.777317    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:21:19.777731 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:19.777607    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:21:19.778242 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:19.778226    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:21:51.976852 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:51.976811    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk"]
Jul 16 23:21:51.977518 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:51.977503    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dee49714-bd27-47b3-bb91-950bd8f1e408" containerName="revoke-keys"
Jul 16 23:21:51.979994 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:51.979973    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk"
Jul 16 23:21:51.982268 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:51.982220    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:21:51.987167 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:51.987136    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk"]
Jul 16 23:21:52.010751 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:52.010718    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f972cdb-2b3b-4829-896a-b344afc28709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk\" (UID: \"4f972cdb-2b3b-4829-896a-b344afc28709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk"
Jul 16 23:21:52.086942 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:52.086899    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8a2515-7d2hx"]
Jul 16 23:21:52.089669 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:52.089645    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8a2515-7d2hx"
Jul 16 23:21:52.098477 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:52.098450    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8a2515-7d2hx"]
Jul 16 23:21:52.112164 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:52.112122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f972cdb-2b3b-4829-896a-b344afc28709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk\" (UID: \"4f972cdb-2b3b-4829-896a-b344afc28709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk"
Jul 16 23:21:52.112366 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:52.112273    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60692b60-a957-4bb8-9b92-737f7afd01f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8a2515-7d2hx\" (UID: \"60692b60-a957-4bb8-9b92-737f7afd01f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8a2515-7d2hx"
Jul 16 23:21:52.112761 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:52.112741    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f972cdb-2b3b-4829-896a-b344afc28709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk\" (UID: \"4f972cdb-2b3b-4829-896a-b344afc28709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk"
Jul 16 23:21:52.213537 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:52.213493    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60692b60-a957-4bb8-9b92-737f7afd01f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8a2515-7d2hx\" (UID: \"60692b60-a957-4bb8-9b92-737f7afd01f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8a2515-7d2hx"
Jul 16 23:21:52.214105 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:52.214084    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60692b60-a957-4bb8-9b92-737f7afd01f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8a2515-7d2hx\" (UID: \"60692b60-a957-4bb8-9b92-737f7afd01f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8a2515-7d2hx"
Jul 16 23:21:52.293434 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:52.293341    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk"
Jul 16 23:21:52.401465 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:52.401433    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8a2515-7d2hx"
Jul 16 23:21:52.434962 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:52.434933    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk"]
Jul 16 23:21:52.437481 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:21:52.437447    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f972cdb_2b3b_4829_896a_b344afc28709.slice/crio-03524ce30891d4c7967b52f18aac4c3d1bac6c66469dab3750e885bd7065cbde WatchSource:0}: Error finding container 03524ce30891d4c7967b52f18aac4c3d1bac6c66469dab3750e885bd7065cbde: Status 404 returned error can't find the container with id 03524ce30891d4c7967b52f18aac4c3d1bac6c66469dab3750e885bd7065cbde
Jul 16 23:21:52.551053 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:52.551026    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8a2515-7d2hx"]
Jul 16 23:21:52.553400 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:21:52.553378    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60692b60_a957_4bb8_9b92_737f7afd01f6.slice/crio-58ce76e9ba4209166113df25d16a078293af5cfe5f175ffa3d94b3147873ca65 WatchSource:0}: Error finding container 58ce76e9ba4209166113df25d16a078293af5cfe5f175ffa3d94b3147873ca65: Status 404 returned error can't find the container with id 58ce76e9ba4209166113df25d16a078293af5cfe5f175ffa3d94b3147873ca65
Jul 16 23:21:52.850365 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:52.850324    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk" event={"ID":"4f972cdb-2b3b-4829-896a-b344afc28709","Type":"ContainerStarted","Data":"1a3c2fe4828511390b96b48498e871c65d775d9643d9c4223b61cf9fed2deba6"}
Jul 16 23:21:52.850524 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:52.850384    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk" event={"ID":"4f972cdb-2b3b-4829-896a-b344afc28709","Type":"ContainerStarted","Data":"03524ce30891d4c7967b52f18aac4c3d1bac6c66469dab3750e885bd7065cbde"}
Jul 16 23:21:52.852219 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:52.852191    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8a2515-7d2hx" event={"ID":"60692b60-a957-4bb8-9b92-737f7afd01f6","Type":"ContainerStarted","Data":"58ce76e9ba4209166113df25d16a078293af5cfe5f175ffa3d94b3147873ca65"}
Jul 16 23:21:52.866577 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:52.866517    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk" podStartSLOduration=1.866499814 podStartE2EDuration="1.866499814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:21:52.864827856 +0000 UTC m=+2923.613311927" watchObservedRunningTime="2026-07-16 23:21:52.866499814 +0000 UTC m=+2923.614983884"
Jul 16 23:21:53.858754 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:53.858717    2565 generic.go:356] "Generic (PLEG): container finished" podID="60692b60-a957-4bb8-9b92-737f7afd01f6" containerID="91948315e7740de44f12c790e182a924c497445946701dac6328b61b1ae339f7" exitCode=0
Jul 16 23:21:53.859206 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:53.858798    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8a2515-7d2hx" event={"ID":"60692b60-a957-4bb8-9b92-737f7afd01f6","Type":"ContainerDied","Data":"91948315e7740de44f12c790e182a924c497445946701dac6328b61b1ae339f7"}
Jul 16 23:21:53.860395 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:53.860373    2565 generic.go:356] "Generic (PLEG): container finished" podID="4f972cdb-2b3b-4829-896a-b344afc28709" containerID="1a3c2fe4828511390b96b48498e871c65d775d9643d9c4223b61cf9fed2deba6" exitCode=6
Jul 16 23:21:53.860512 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:53.860419    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk" event={"ID":"4f972cdb-2b3b-4829-896a-b344afc28709","Type":"ContainerDied","Data":"1a3c2fe4828511390b96b48498e871c65d775d9643d9c4223b61cf9fed2deba6"}
Jul 16 23:21:53.860623 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:53.860612    2565 scope.go:117] "RemoveContainer" containerID="1a3c2fe4828511390b96b48498e871c65d775d9643d9c4223b61cf9fed2deba6"
Jul 16 23:21:54.867604 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:54.867569    2565 generic.go:356] "Generic (PLEG): container finished" podID="4f972cdb-2b3b-4829-896a-b344afc28709" containerID="c981ef078f33f841ccd50fbbc2bb6e2f3c045cb3fd2f0d07cd20a30a11bd3b99" exitCode=6
Jul 16 23:21:54.868074 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:54.867643    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk" event={"ID":"4f972cdb-2b3b-4829-896a-b344afc28709","Type":"ContainerDied","Data":"c981ef078f33f841ccd50fbbc2bb6e2f3c045cb3fd2f0d07cd20a30a11bd3b99"}
Jul 16 23:21:54.868074 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:54.867687    2565 scope.go:117] "RemoveContainer" containerID="1a3c2fe4828511390b96b48498e871c65d775d9643d9c4223b61cf9fed2deba6"
Jul 16 23:21:54.868074 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:54.867858    2565 scope.go:117] "RemoveContainer" containerID="c981ef078f33f841ccd50fbbc2bb6e2f3c045cb3fd2f0d07cd20a30a11bd3b99"
Jul 16 23:21:54.868525 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:21:54.868158    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk_odh-ai-gateway-infra(4f972cdb-2b3b-4829-896a-b344afc28709)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk" podUID="4f972cdb-2b3b-4829-896a-b344afc28709"
Jul 16 23:21:55.010973 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:55.010942    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8a2515-7d2hx"
Jul 16 23:21:55.045191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:55.045147    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60692b60-a957-4bb8-9b92-737f7afd01f6-maas-api-service-ca\") pod \"60692b60-a957-4bb8-9b92-737f7afd01f6\" (UID: \"60692b60-a957-4bb8-9b92-737f7afd01f6\") "
Jul 16 23:21:55.045536 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:55.045514    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60692b60-a957-4bb8-9b92-737f7afd01f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60692b60-a957-4bb8-9b92-737f7afd01f6" (UID: "60692b60-a957-4bb8-9b92-737f7afd01f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:21:55.045605 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:55.045591    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60692b60-a957-4bb8-9b92-737f7afd01f6-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:21:55.881725 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:55.881692    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8a2515-7d2hx"
Jul 16 23:21:55.882171 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:55.881732    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8a2515-7d2hx" event={"ID":"60692b60-a957-4bb8-9b92-737f7afd01f6","Type":"ContainerDied","Data":"58ce76e9ba4209166113df25d16a078293af5cfe5f175ffa3d94b3147873ca65"}
Jul 16 23:21:55.882171 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:55.881816    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58ce76e9ba4209166113df25d16a078293af5cfe5f175ffa3d94b3147873ca65"
Jul 16 23:21:55.883640 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:21:55.883572    2565 scope.go:117] "RemoveContainer" containerID="c981ef078f33f841ccd50fbbc2bb6e2f3c045cb3fd2f0d07cd20a30a11bd3b99"
Jul 16 23:21:55.883831 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:21:55.883804    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk_odh-ai-gateway-infra(4f972cdb-2b3b-4829-896a-b344afc28709)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk" podUID="4f972cdb-2b3b-4829-896a-b344afc28709"
Jul 16 23:22:02.465868 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:02.465815    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"]
Jul 16 23:22:02.466753 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:02.466713    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k" podUID="ca9a6e33-0b3f-4704-a13f-7ead17a7f967" containerName="maas-api" containerID="cri-o://978c08c2676007b2a520751ff9d40cc8af2f568ded1c1bbd938802050f56bb1d" gracePeriod=30
Jul 16 23:22:02.926365 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:02.926333    2565 generic.go:356] "Generic (PLEG): container finished" podID="ca9a6e33-0b3f-4704-a13f-7ead17a7f967" containerID="978c08c2676007b2a520751ff9d40cc8af2f568ded1c1bbd938802050f56bb1d" exitCode=0
Jul 16 23:22:02.926527 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:02.926380    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k" event={"ID":"ca9a6e33-0b3f-4704-a13f-7ead17a7f967","Type":"ContainerDied","Data":"978c08c2676007b2a520751ff9d40cc8af2f568ded1c1bbd938802050f56bb1d"}
Jul 16 23:22:02.926527 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:02.926426    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k" event={"ID":"ca9a6e33-0b3f-4704-a13f-7ead17a7f967","Type":"ContainerDied","Data":"3026db3cf40f53819d56ac7513e6e82911615bb8847b914713e0f71d392d7149"}
Jul 16 23:22:02.926527 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:02.926448    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3026db3cf40f53819d56ac7513e6e82911615bb8847b914713e0f71d392d7149"
Jul 16 23:22:02.927441 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:02.927422    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"
Jul 16 23:22:03.023842 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:03.023752    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-openshift-service-ca\") pod \"ca9a6e33-0b3f-4704-a13f-7ead17a7f967\" (UID: \"ca9a6e33-0b3f-4704-a13f-7ead17a7f967\") "
Jul 16 23:22:03.023842 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:03.023831    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh8gp\" (UniqueName: \"kubernetes.io/projected/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-kube-api-access-nh8gp\") pod \"ca9a6e33-0b3f-4704-a13f-7ead17a7f967\" (UID: \"ca9a6e33-0b3f-4704-a13f-7ead17a7f967\") "
Jul 16 23:22:03.024051 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:03.023871    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-maas-api-tls\") pod \"ca9a6e33-0b3f-4704-a13f-7ead17a7f967\" (UID: \"ca9a6e33-0b3f-4704-a13f-7ead17a7f967\") "
Jul 16 23:22:03.024144 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:03.024114    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca9a6e33-0b3f-4704-a13f-7ead17a7f967" (UID: "ca9a6e33-0b3f-4704-a13f-7ead17a7f967"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:22:03.025943 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:03.025917    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-kube-api-access-nh8gp" (OuterVolumeSpecName: "kube-api-access-nh8gp") pod "ca9a6e33-0b3f-4704-a13f-7ead17a7f967" (UID: "ca9a6e33-0b3f-4704-a13f-7ead17a7f967"). InnerVolumeSpecName "kube-api-access-nh8gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:22:03.026056 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:03.025922    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca9a6e33-0b3f-4704-a13f-7ead17a7f967" (UID: "ca9a6e33-0b3f-4704-a13f-7ead17a7f967"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:22:03.124643 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:03.124602    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:03.124643 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:03.124637    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:03.124643 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:03.124648    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh8gp\" (UniqueName: \"kubernetes.io/projected/ca9a6e33-0b3f-4704-a13f-7ead17a7f967-kube-api-access-nh8gp\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:03.931702 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:03.931669    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"
Jul 16 23:22:03.947091 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:03.947055    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"]
Jul 16 23:22:03.952783 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:03.952742    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8a2515-659686898b-s9x8k"]
Jul 16 23:22:05.726316 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:05.726283    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca9a6e33-0b3f-4704-a13f-7ead17a7f967" path="/var/lib/kubelet/pods/ca9a6e33-0b3f-4704-a13f-7ead17a7f967/volumes"
Jul 16 23:22:10.720851 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:10.720818    2565 scope.go:117] "RemoveContainer" containerID="c981ef078f33f841ccd50fbbc2bb6e2f3c045cb3fd2f0d07cd20a30a11bd3b99"
Jul 16 23:22:11.983315 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:11.983274    2565 generic.go:356] "Generic (PLEG): container finished" podID="4f972cdb-2b3b-4829-896a-b344afc28709" containerID="3eca4c4c9d8af142152bc864e1c1275ef4740f33e13898c91769452bfac1f9bf" exitCode=6
Jul 16 23:22:11.983315 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:11.983289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk" event={"ID":"4f972cdb-2b3b-4829-896a-b344afc28709","Type":"ContainerDied","Data":"3eca4c4c9d8af142152bc864e1c1275ef4740f33e13898c91769452bfac1f9bf"}
Jul 16 23:22:11.983840 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:11.983388    2565 scope.go:117] "RemoveContainer" containerID="c981ef078f33f841ccd50fbbc2bb6e2f3c045cb3fd2f0d07cd20a30a11bd3b99"
Jul 16 23:22:11.983840 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:11.983726    2565 scope.go:117] "RemoveContainer" containerID="3eca4c4c9d8af142152bc864e1c1275ef4740f33e13898c91769452bfac1f9bf"
Jul 16 23:22:11.983967 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:22:11.983941    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk_odh-ai-gateway-infra(4f972cdb-2b3b-4829-896a-b344afc28709)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk" podUID="4f972cdb-2b3b-4829-896a-b344afc28709"
Jul 16 23:22:13.012403 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:13.012360    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk"]
Jul 16 23:22:13.151730 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:13.151702    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk"
Jul 16 23:22:13.323651 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:13.323561    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f972cdb-2b3b-4829-896a-b344afc28709-maas-api-service-ca\") pod \"4f972cdb-2b3b-4829-896a-b344afc28709\" (UID: \"4f972cdb-2b3b-4829-896a-b344afc28709\") "
Jul 16 23:22:13.323921 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:13.323896    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f972cdb-2b3b-4829-896a-b344afc28709-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4f972cdb-2b3b-4829-896a-b344afc28709" (UID: "4f972cdb-2b3b-4829-896a-b344afc28709"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:22:13.424959 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:13.424922    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f972cdb-2b3b-4829-896a-b344afc28709-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:13.997653 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:13.997568    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk"
Jul 16 23:22:13.997653 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:13.997581    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk" event={"ID":"4f972cdb-2b3b-4829-896a-b344afc28709","Type":"ContainerDied","Data":"03524ce30891d4c7967b52f18aac4c3d1bac6c66469dab3750e885bd7065cbde"}
Jul 16 23:22:13.997653 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:13.997644    2565 scope.go:117] "RemoveContainer" containerID="3eca4c4c9d8af142152bc864e1c1275ef4740f33e13898c91769452bfac1f9bf"
Jul 16 23:22:14.013241 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:14.013212    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk"]
Jul 16 23:22:14.018940 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:14.018916    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-dm4mk"]
Jul 16 23:22:15.725834 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:15.725797    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f972cdb-2b3b-4829-896a-b344afc28709" path="/var/lib/kubelet/pods/4f972cdb-2b3b-4829-896a-b344afc28709/volumes"
Jul 16 23:22:17.885135 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:17.885101    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"]
Jul 16 23:22:17.885654 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:17.885356    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7" podUID="7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" containerName="istio-proxy" containerID="cri-o://7c4e8df7a168c0e12985271dd5583fb9af7d80446b7bf377819b03f5090cdf0c" gracePeriod=30
Jul 16 23:22:23.050092 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.050059    2565 generic.go:356] "Generic (PLEG): container finished" podID="7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" containerID="7c4e8df7a168c0e12985271dd5583fb9af7d80446b7bf377819b03f5090cdf0c" exitCode=0
Jul 16 23:22:23.050509 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.050100    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7" event={"ID":"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8","Type":"ContainerDied","Data":"7c4e8df7a168c0e12985271dd5583fb9af7d80446b7bf377819b03f5090cdf0c"}
Jul 16 23:22:23.050509 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.050166    2565 scope.go:117] "RemoveContainer" containerID="c0d6bb527be08fc70ffd4ba3df9121af4ae9ca352c689d4ea33b7162abcb2d0b"
Jul 16 23:22:23.152880 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.152801    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:22:23.216182 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.216150    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-workload-certs\") pod \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") "
Jul 16 23:22:23.216182 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.216193    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-podinfo\") pod \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") "
Jul 16 23:22:23.216487 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.216218    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-credential-socket\") pod \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") "
Jul 16 23:22:23.216487 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.216275    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-token\") pod \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") "
Jul 16 23:22:23.216487 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.216299    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-workload-socket\") pod \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") "
Jul 16 23:22:23.216487 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.216337    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istiod-ca-cert\") pod \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") "
Jul 16 23:22:23.216487 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.216412    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-data\") pod \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") "
Jul 16 23:22:23.216487 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.216441    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl5v8\" (UniqueName: \"kubernetes.io/projected/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-kube-api-access-hl5v8\") pod \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") "
Jul 16 23:22:23.216487 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.216467    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-envoy\") pod \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\" (UID: \"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8\") "
Jul 16 23:22:23.216845 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.216763    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" (UID: "7678090d-6d22-4f7c-bc5e-b4f06b4f35d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:22:23.216845 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.216832    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" (UID: "7678090d-6d22-4f7c-bc5e-b4f06b4f35d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:22:23.216947 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.216845    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" (UID: "7678090d-6d22-4f7c-bc5e-b4f06b4f35d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:22:23.217075 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.217047    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" (UID: "7678090d-6d22-4f7c-bc5e-b4f06b4f35d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:22:23.217554 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.217520    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-data" (OuterVolumeSpecName: "istio-data") pod "7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" (UID: "7678090d-6d22-4f7c-bc5e-b4f06b4f35d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:22:23.218700 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.218676    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-token" (OuterVolumeSpecName: "istio-token") pod "7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" (UID: "7678090d-6d22-4f7c-bc5e-b4f06b4f35d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:22:23.218799 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.218721    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" (UID: "7678090d-6d22-4f7c-bc5e-b4f06b4f35d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 23:22:23.218799 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.218741    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" (UID: "7678090d-6d22-4f7c-bc5e-b4f06b4f35d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:22:23.218799 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.218769    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-kube-api-access-hl5v8" (OuterVolumeSpecName: "kube-api-access-hl5v8") pod "7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" (UID: "7678090d-6d22-4f7c-bc5e-b4f06b4f35d8"). InnerVolumeSpecName "kube-api-access-hl5v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:22:23.317814 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.317776    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-data\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:23.317814 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.317808    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl5v8\" (UniqueName: \"kubernetes.io/projected/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-kube-api-access-hl5v8\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:23.317814 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.317818    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-envoy\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:23.317814 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.317827    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-workload-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:23.318094 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.317835    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-podinfo\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:23.318094 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.317843    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-credential-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:23.318094 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.317852    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istio-token\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:23.318094 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.317863    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-workload-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:23.318094 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:23.317874    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8-istiod-ca-cert\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:24.057044 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:24.056953    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"
Jul 16 23:22:24.057044 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:24.056999    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7" event={"ID":"7678090d-6d22-4f7c-bc5e-b4f06b4f35d8","Type":"ContainerDied","Data":"4687ee45ab95bf3e1db18c3d5c855b63438e86a7844426fa849b37cf5441b0bf"}
Jul 16 23:22:24.057563 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:24.057070    2565 scope.go:117] "RemoveContainer" containerID="7c4e8df7a168c0e12985271dd5583fb9af7d80446b7bf377819b03f5090cdf0c"
Jul 16 23:22:24.076379 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:24.076343    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"]
Jul 16 23:22:24.080535 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:24.080503    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8a2515-openshift-default-65d648b677-pvhw7"]
Jul 16 23:22:25.726336 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:25.726301    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" path="/var/lib/kubelet/pods/7678090d-6d22-4f7c-bc5e-b4f06b4f35d8/volumes"
Jul 16 23:22:28.745445 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:28.745409    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"]
Jul 16 23:22:28.745992 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:28.745956    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn" podUID="36a66218-1bfc-40a1-9421-1a6c31893502" containerName="maas-api" containerID="cri-o://0ac663b439760251e7040950c2f39d9ace6963f35764536776bd5453488ea485" gracePeriod=30
Jul 16 23:22:29.008327 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.008221    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"
Jul 16 23:22:29.070349 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.070310    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a66218-1bfc-40a1-9421-1a6c31893502-openshift-service-ca\") pod \"36a66218-1bfc-40a1-9421-1a6c31893502\" (UID: \"36a66218-1bfc-40a1-9421-1a6c31893502\") "
Jul 16 23:22:29.070349 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.070349    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s94sg\" (UniqueName: \"kubernetes.io/projected/36a66218-1bfc-40a1-9421-1a6c31893502-kube-api-access-s94sg\") pod \"36a66218-1bfc-40a1-9421-1a6c31893502\" (UID: \"36a66218-1bfc-40a1-9421-1a6c31893502\") "
Jul 16 23:22:29.070619 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.070378    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36a66218-1bfc-40a1-9421-1a6c31893502-maas-api-tls\") pod \"36a66218-1bfc-40a1-9421-1a6c31893502\" (UID: \"36a66218-1bfc-40a1-9421-1a6c31893502\") "
Jul 16 23:22:29.070774 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.070747    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a66218-1bfc-40a1-9421-1a6c31893502-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "36a66218-1bfc-40a1-9421-1a6c31893502" (UID: "36a66218-1bfc-40a1-9421-1a6c31893502"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:22:29.072627 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.072594    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a66218-1bfc-40a1-9421-1a6c31893502-kube-api-access-s94sg" (OuterVolumeSpecName: "kube-api-access-s94sg") pod "36a66218-1bfc-40a1-9421-1a6c31893502" (UID: "36a66218-1bfc-40a1-9421-1a6c31893502"). InnerVolumeSpecName "kube-api-access-s94sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:22:29.072720 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.072671    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36a66218-1bfc-40a1-9421-1a6c31893502-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "36a66218-1bfc-40a1-9421-1a6c31893502" (UID: "36a66218-1bfc-40a1-9421-1a6c31893502"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:22:29.085995 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.085961    2565 generic.go:356] "Generic (PLEG): container finished" podID="36a66218-1bfc-40a1-9421-1a6c31893502" containerID="0ac663b439760251e7040950c2f39d9ace6963f35764536776bd5453488ea485" exitCode=0
Jul 16 23:22:29.086171 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.086023    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"
Jul 16 23:22:29.086171 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.086036    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn" event={"ID":"36a66218-1bfc-40a1-9421-1a6c31893502","Type":"ContainerDied","Data":"0ac663b439760251e7040950c2f39d9ace6963f35764536776bd5453488ea485"}
Jul 16 23:22:29.086171 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.086113    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn" event={"ID":"36a66218-1bfc-40a1-9421-1a6c31893502","Type":"ContainerDied","Data":"8f431a3b640a76da3549e0a98514e79e4a6ac040e6e463701edaf270d9acbc15"}
Jul 16 23:22:29.086171 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.086135    2565 scope.go:117] "RemoveContainer" containerID="0ac663b439760251e7040950c2f39d9ace6963f35764536776bd5453488ea485"
Jul 16 23:22:29.095361 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.095338    2565 scope.go:117] "RemoveContainer" containerID="0ac663b439760251e7040950c2f39d9ace6963f35764536776bd5453488ea485"
Jul 16 23:22:29.095650 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:22:29.095631    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac663b439760251e7040950c2f39d9ace6963f35764536776bd5453488ea485\": container with ID starting with 0ac663b439760251e7040950c2f39d9ace6963f35764536776bd5453488ea485 not found: ID does not exist" containerID="0ac663b439760251e7040950c2f39d9ace6963f35764536776bd5453488ea485"
Jul 16 23:22:29.095704 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.095661    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac663b439760251e7040950c2f39d9ace6963f35764536776bd5453488ea485"} err="failed to get container status \"0ac663b439760251e7040950c2f39d9ace6963f35764536776bd5453488ea485\": rpc error: code = NotFound desc = could not find container \"0ac663b439760251e7040950c2f39d9ace6963f35764536776bd5453488ea485\": container with ID starting with 0ac663b439760251e7040950c2f39d9ace6963f35764536776bd5453488ea485 not found: ID does not exist"
Jul 16 23:22:29.107417 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.107380    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"]
Jul 16 23:22:29.111465 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.111437    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4d775e-77f7d54449-4kbwn"]
Jul 16 23:22:29.171963 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.171922    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a66218-1bfc-40a1-9421-1a6c31893502-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:29.171963 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.171957    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s94sg\" (UniqueName: \"kubernetes.io/projected/36a66218-1bfc-40a1-9421-1a6c31893502-kube-api-access-s94sg\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:29.171963 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.171970    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36a66218-1bfc-40a1-9421-1a6c31893502-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:29.726395 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:29.726364    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36a66218-1bfc-40a1-9421-1a6c31893502" path="/var/lib/kubelet/pods/36a66218-1bfc-40a1-9421-1a6c31893502/volumes"
Jul 16 23:22:44.021352 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.021271    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"]
Jul 16 23:22:44.021878 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.021497    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9" podUID="c3fd15e8-f607-449d-a7e3-10a8fa205782" containerName="istio-proxy" containerID="cri-o://f2cd5cf4c3717e730b1d76034d40dd250cbe8c5a757ad92bd58ea3c1e0b301c2" gracePeriod=30
Jul 16 23:22:44.867762 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.867724    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"]
Jul 16 23:22:44.868100 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868087    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f972cdb-2b3b-4829-896a-b344afc28709" containerName="revoke-keys"
Jul 16 23:22:44.868146 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868100    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f972cdb-2b3b-4829-896a-b344afc28709" containerName="revoke-keys"
Jul 16 23:22:44.868146 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868111    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f972cdb-2b3b-4829-896a-b344afc28709" containerName="revoke-keys"
Jul 16 23:22:44.868146 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868117    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f972cdb-2b3b-4829-896a-b344afc28709" containerName="revoke-keys"
Jul 16 23:22:44.868146 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868129    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" containerName="istio-proxy"
Jul 16 23:22:44.868146 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868134    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" containerName="istio-proxy"
Jul 16 23:22:44.868146 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868143    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36a66218-1bfc-40a1-9421-1a6c31893502" containerName="maas-api"
Jul 16 23:22:44.868374 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868148    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="36a66218-1bfc-40a1-9421-1a6c31893502" containerName="maas-api"
Jul 16 23:22:44.868374 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868162    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca9a6e33-0b3f-4704-a13f-7ead17a7f967" containerName="maas-api"
Jul 16 23:22:44.868374 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868167    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca9a6e33-0b3f-4704-a13f-7ead17a7f967" containerName="maas-api"
Jul 16 23:22:44.868374 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868174    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60692b60-a957-4bb8-9b92-737f7afd01f6" containerName="revoke-keys"
Jul 16 23:22:44.868374 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868179    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="60692b60-a957-4bb8-9b92-737f7afd01f6" containerName="revoke-keys"
Jul 16 23:22:44.868374 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868233    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" containerName="istio-proxy"
Jul 16 23:22:44.868374 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868240    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f972cdb-2b3b-4829-896a-b344afc28709" containerName="revoke-keys"
Jul 16 23:22:44.868374 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868264    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="36a66218-1bfc-40a1-9421-1a6c31893502" containerName="maas-api"
Jul 16 23:22:44.868374 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868275    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" containerName="istio-proxy"
Jul 16 23:22:44.868374 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868283    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="60692b60-a957-4bb8-9b92-737f7afd01f6" containerName="revoke-keys"
Jul 16 23:22:44.868374 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868290    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca9a6e33-0b3f-4704-a13f-7ead17a7f967" containerName="maas-api"
Jul 16 23:22:44.868374 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.868297    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f972cdb-2b3b-4829-896a-b344afc28709" containerName="revoke-keys"
Jul 16 23:22:44.872904 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.872884    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:44.875304 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.875279    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d302e4-openshift-default-dockercfg-kglr7\""
Jul 16 23:22:44.882483 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.882455    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"]
Jul 16 23:22:44.916311 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.916269    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-token\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:44.916532 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.916339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-envoy\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:44.916532 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.916363    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-workload-certs\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:44.916532 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.916388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istiod-ca-cert\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:44.916532 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.916414    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gldfj\" (UniqueName: \"kubernetes.io/projected/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-kube-api-access-gldfj\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:44.916532 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.916432    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-data\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:44.916797 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.916523    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-credential-socket\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:44.916797 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.916625    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-podinfo\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:44.916797 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:44.916677    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-workload-socket\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.017888 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.017844    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-workload-socket\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.018097 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.017901    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-token\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.018097 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.017943    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-envoy\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.018097 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.017973    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-workload-certs\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.018097 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.018007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istiod-ca-cert\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.018354 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.018206    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gldfj\" (UniqueName: \"kubernetes.io/projected/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-kube-api-access-gldfj\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.018354 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.018276    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-data\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.018354 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.018331    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-credential-socket\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.018519 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.018371    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-workload-socket\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.018519 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.018387    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-podinfo\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.018519 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.018477    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-workload-certs\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.018741 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.018701    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-credential-socket\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.018741 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.018732    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-data\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.018839 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.018730    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istiod-ca-cert\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.020368 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.020346    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-envoy\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.020877 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.020856    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-podinfo\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.026063 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.026036    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-token\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.026391 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.026184    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gldfj\" (UniqueName: \"kubernetes.io/projected/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-kube-api-access-gldfj\") pod \"e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") " pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.185502 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.185400    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:45.346388 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.346345    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"]
Jul 16 23:22:45.350333 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:22:45.350306    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f4db0c_5f03_4a96_9d24_c5aea6aa8b53.slice/crio-245db529d253a983dbcfafa6d2a645c52f9afde88404ddea2d43d8e23ef9363e WatchSource:0}: Error finding container 245db529d253a983dbcfafa6d2a645c52f9afde88404ddea2d43d8e23ef9363e: Status 404 returned error can't find the container with id 245db529d253a983dbcfafa6d2a645c52f9afde88404ddea2d43d8e23ef9363e
Jul 16 23:22:45.352512 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.352475    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:22:45.352616 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.352568    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:22:45.352680 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:45.352614    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:22:46.171946 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:46.171907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474" event={"ID":"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53","Type":"ContainerStarted","Data":"55bf8c86501c047b71e9a7ae21a9d3d914aeabddff92f961ec86ee423af2d27f"}
Jul 16 23:22:46.172432 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:46.171960    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474" event={"ID":"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53","Type":"ContainerStarted","Data":"245db529d253a983dbcfafa6d2a645c52f9afde88404ddea2d43d8e23ef9363e"}
Jul 16 23:22:46.185527 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:46.185496    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:46.191352 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:46.191319    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:46.192013 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:46.191971    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474" podStartSLOduration=2.191955736 podStartE2EDuration="2.191955736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:22:46.189382258 +0000 UTC m=+2976.937866324" watchObservedRunningTime="2026-07-16 23:22:46.191955736 +0000 UTC m=+2976.940439879"
Jul 16 23:22:47.177547 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:47.177505    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:47.178671 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:47.178649    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:22:48.267412 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.267378    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"]
Jul 16 23:22:48.267908 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.267890    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f972cdb-2b3b-4829-896a-b344afc28709" containerName="revoke-keys"
Jul 16 23:22:48.267908 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.267908    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f972cdb-2b3b-4829-896a-b344afc28709" containerName="revoke-keys"
Jul 16 23:22:48.268026 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.267925    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" containerName="istio-proxy"
Jul 16 23:22:48.268026 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.267930    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7678090d-6d22-4f7c-bc5e-b4f06b4f35d8" containerName="istio-proxy"
Jul 16 23:22:48.268026 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.268007    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f972cdb-2b3b-4829-896a-b344afc28709" containerName="revoke-keys"
Jul 16 23:22:48.271198 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.271176    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"
Jul 16 23:22:48.273376 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.273349    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d302e4-serving-cert\""
Jul 16 23:22:48.281961 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.281931    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"]
Jul 16 23:22:48.353359 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.353323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/631d0460-339f-4583-8d6b-259a50119b47-openshift-service-ca\") pod \"maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs\" (UID: \"631d0460-339f-4583-8d6b-259a50119b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"
Jul 16 23:22:48.353359 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.353367    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fws7\" (UniqueName: \"kubernetes.io/projected/631d0460-339f-4583-8d6b-259a50119b47-kube-api-access-4fws7\") pod \"maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs\" (UID: \"631d0460-339f-4583-8d6b-259a50119b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"
Jul 16 23:22:48.353583 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.353469    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/631d0460-339f-4583-8d6b-259a50119b47-maas-api-tls\") pod \"maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs\" (UID: \"631d0460-339f-4583-8d6b-259a50119b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"
Jul 16 23:22:48.454678 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.454639    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/631d0460-339f-4583-8d6b-259a50119b47-openshift-service-ca\") pod \"maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs\" (UID: \"631d0460-339f-4583-8d6b-259a50119b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"
Jul 16 23:22:48.454678 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.454686    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fws7\" (UniqueName: \"kubernetes.io/projected/631d0460-339f-4583-8d6b-259a50119b47-kube-api-access-4fws7\") pod \"maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs\" (UID: \"631d0460-339f-4583-8d6b-259a50119b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"
Jul 16 23:22:48.454924 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.454752    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/631d0460-339f-4583-8d6b-259a50119b47-maas-api-tls\") pod \"maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs\" (UID: \"631d0460-339f-4583-8d6b-259a50119b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"
Jul 16 23:22:48.454924 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:22:48.454871    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-serving-cert: secret "maas-api-e2e-api-a-d302e4-serving-cert" not found
Jul 16 23:22:48.455019 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:22:48.454928    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/631d0460-339f-4583-8d6b-259a50119b47-maas-api-tls podName:631d0460-339f-4583-8d6b-259a50119b47 nodeName:}" failed. No retries permitted until 2026-07-16 23:22:48.95491111 +0000 UTC m=+2979.703395159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/631d0460-339f-4583-8d6b-259a50119b47-maas-api-tls") pod "maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs" (UID: "631d0460-339f-4583-8d6b-259a50119b47") : secret "maas-api-e2e-api-a-d302e4-serving-cert" not found
Jul 16 23:22:48.455455 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.455435    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/631d0460-339f-4583-8d6b-259a50119b47-openshift-service-ca\") pod \"maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs\" (UID: \"631d0460-339f-4583-8d6b-259a50119b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"
Jul 16 23:22:48.477868 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.477831    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fws7\" (UniqueName: \"kubernetes.io/projected/631d0460-339f-4583-8d6b-259a50119b47-kube-api-access-4fws7\") pod \"maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs\" (UID: \"631d0460-339f-4583-8d6b-259a50119b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"
Jul 16 23:22:48.959766 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.959718    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/631d0460-339f-4583-8d6b-259a50119b47-maas-api-tls\") pod \"maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs\" (UID: \"631d0460-339f-4583-8d6b-259a50119b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"
Jul 16 23:22:48.962235 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:48.962207    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/631d0460-339f-4583-8d6b-259a50119b47-maas-api-tls\") pod \"maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs\" (UID: \"631d0460-339f-4583-8d6b-259a50119b47\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"
Jul 16 23:22:49.183909 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.183881    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"
Jul 16 23:22:49.189374 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.189344    2565 generic.go:356] "Generic (PLEG): container finished" podID="c3fd15e8-f607-449d-a7e3-10a8fa205782" containerID="f2cd5cf4c3717e730b1d76034d40dd250cbe8c5a757ad92bd58ea3c1e0b301c2" exitCode=0
Jul 16 23:22:49.189523 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.189420    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9" event={"ID":"c3fd15e8-f607-449d-a7e3-10a8fa205782","Type":"ContainerDied","Data":"f2cd5cf4c3717e730b1d76034d40dd250cbe8c5a757ad92bd58ea3c1e0b301c2"}
Jul 16 23:22:49.189523 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.189486    2565 scope.go:117] "RemoveContainer" containerID="940f5c1a5bf42b7a0ce1cf48a2629c375d35c48f9322ab1b7a099da489a7cbba"
Jul 16 23:22:49.282758 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.282710    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:22:49.343852 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.343755    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"]
Jul 16 23:22:49.346111 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:22:49.346085    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631d0460_339f_4583_8d6b_259a50119b47.slice/crio-f1a7900935004d9d31fbb5f85a7067d40595f656eafb204e831a9a725c6d99b4 WatchSource:0}: Error finding container f1a7900935004d9d31fbb5f85a7067d40595f656eafb204e831a9a725c6d99b4: Status 404 returned error can't find the container with id f1a7900935004d9d31fbb5f85a7067d40595f656eafb204e831a9a725c6d99b4
Jul 16 23:22:49.364546 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.364513    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7ds4\" (UniqueName: \"kubernetes.io/projected/c3fd15e8-f607-449d-a7e3-10a8fa205782-kube-api-access-m7ds4\") pod \"c3fd15e8-f607-449d-a7e3-10a8fa205782\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") "
Jul 16 23:22:49.364705 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.364573    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-workload-certs\") pod \"c3fd15e8-f607-449d-a7e3-10a8fa205782\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") "
Jul 16 23:22:49.364705 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.364615    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-podinfo\") pod \"c3fd15e8-f607-449d-a7e3-10a8fa205782\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") "
Jul 16 23:22:49.364795 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.364712    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fd15e8-f607-449d-a7e3-10a8fa205782-istiod-ca-cert\") pod \"c3fd15e8-f607-449d-a7e3-10a8fa205782\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") "
Jul 16 23:22:49.364795 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.364773    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-token\") pod \"c3fd15e8-f607-449d-a7e3-10a8fa205782\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") "
Jul 16 23:22:49.364887 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.364806    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-credential-socket\") pod \"c3fd15e8-f607-449d-a7e3-10a8fa205782\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") "
Jul 16 23:22:49.364887 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.364838    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-workload-socket\") pod \"c3fd15e8-f607-449d-a7e3-10a8fa205782\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") "
Jul 16 23:22:49.364992 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.364891    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-envoy\") pod \"c3fd15e8-f607-449d-a7e3-10a8fa205782\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") "
Jul 16 23:22:49.364992 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.364934    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-data\") pod \"c3fd15e8-f607-449d-a7e3-10a8fa205782\" (UID: \"c3fd15e8-f607-449d-a7e3-10a8fa205782\") "
Jul 16 23:22:49.364992 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.364960    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c3fd15e8-f607-449d-a7e3-10a8fa205782" (UID: "c3fd15e8-f607-449d-a7e3-10a8fa205782"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:22:49.365147 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.365057    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3fd15e8-f607-449d-a7e3-10a8fa205782-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c3fd15e8-f607-449d-a7e3-10a8fa205782" (UID: "c3fd15e8-f607-449d-a7e3-10a8fa205782"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:22:49.365206 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.365161    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c3fd15e8-f607-449d-a7e3-10a8fa205782" (UID: "c3fd15e8-f607-449d-a7e3-10a8fa205782"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:22:49.365206 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.365183    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c3fd15e8-f607-449d-a7e3-10a8fa205782" (UID: "c3fd15e8-f607-449d-a7e3-10a8fa205782"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:22:49.365483 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.365454    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-workload-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:49.365483 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.365484    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3fd15e8-f607-449d-a7e3-10a8fa205782-istiod-ca-cert\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:49.365677 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.365499    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-credential-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:49.365677 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.365515    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-workload-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:49.365832 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.365811    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-data" (OuterVolumeSpecName: "istio-data") pod "c3fd15e8-f607-449d-a7e3-10a8fa205782" (UID: "c3fd15e8-f607-449d-a7e3-10a8fa205782"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:22:49.366934 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.366908    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c3fd15e8-f607-449d-a7e3-10a8fa205782" (UID: "c3fd15e8-f607-449d-a7e3-10a8fa205782"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 23:22:49.366934 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.366925    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3fd15e8-f607-449d-a7e3-10a8fa205782-kube-api-access-m7ds4" (OuterVolumeSpecName: "kube-api-access-m7ds4") pod "c3fd15e8-f607-449d-a7e3-10a8fa205782" (UID: "c3fd15e8-f607-449d-a7e3-10a8fa205782"). InnerVolumeSpecName "kube-api-access-m7ds4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:22:49.367398 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.367368    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c3fd15e8-f607-449d-a7e3-10a8fa205782" (UID: "c3fd15e8-f607-449d-a7e3-10a8fa205782"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:22:49.367463 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.367382    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-token" (OuterVolumeSpecName: "istio-token") pod "c3fd15e8-f607-449d-a7e3-10a8fa205782" (UID: "c3fd15e8-f607-449d-a7e3-10a8fa205782"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:22:49.466540 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.466502    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-data\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:49.466540 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.466532    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7ds4\" (UniqueName: \"kubernetes.io/projected/c3fd15e8-f607-449d-a7e3-10a8fa205782-kube-api-access-m7ds4\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:49.466540 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.466542    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-podinfo\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:49.466540 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.466550    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-token\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:49.466821 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:49.466558    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3fd15e8-f607-449d-a7e3-10a8fa205782-istio-envoy\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:22:50.195547 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:50.195513    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"
Jul 16 23:22:50.195738 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:50.195546    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9" event={"ID":"c3fd15e8-f607-449d-a7e3-10a8fa205782","Type":"ContainerDied","Data":"82050c5a8cb769d7826fc5980507901bebae61704b7b7dd6ec709a2f0e245eed"}
Jul 16 23:22:50.195738 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:50.195627    2565 scope.go:117] "RemoveContainer" containerID="f2cd5cf4c3717e730b1d76034d40dd250cbe8c5a757ad92bd58ea3c1e0b301c2"
Jul 16 23:22:50.197283 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:50.197233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs" event={"ID":"631d0460-339f-4583-8d6b-259a50119b47","Type":"ContainerStarted","Data":"625066ea4c95bf68d88448d2133c7e3aba22f8b2783ae8daefec7581f87c1c71"}
Jul 16 23:22:50.197417 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:50.197306    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs" event={"ID":"631d0460-339f-4583-8d6b-259a50119b47","Type":"ContainerStarted","Data":"f1a7900935004d9d31fbb5f85a7067d40595f656eafb204e831a9a725c6d99b4"}
Jul 16 23:22:50.197417 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:50.197330    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"
Jul 16 23:22:50.211978 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:50.211951    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"]
Jul 16 23:22:50.215740 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:50.215717    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4d775e-openshift-default-c5b75576c-f7bm9"]
Jul 16 23:22:50.235298 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:50.235217    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs" podStartSLOduration=1.774266447 podStartE2EDuration="2.235201753s" totalImagesPullingTime="460.935306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:22:50.233771098 +0000 UTC m=+2980.982255190" watchObservedRunningTime="2026-07-16 23:22:50.235201753 +0000 UTC m=+2980.983685824"
Jul 16 23:22:51.726714 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:51.726680    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3fd15e8-f607-449d-a7e3-10a8fa205782" path="/var/lib/kubelet/pods/c3fd15e8-f607-449d-a7e3-10a8fa205782/volumes"
Jul 16 23:22:53.297373 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.297336    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"]
Jul 16 23:22:53.297922 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.297900    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3fd15e8-f607-449d-a7e3-10a8fa205782" containerName="istio-proxy"
Jul 16 23:22:53.297988 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.297924    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3fd15e8-f607-449d-a7e3-10a8fa205782" containerName="istio-proxy"
Jul 16 23:22:53.297988 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.297942    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3fd15e8-f607-449d-a7e3-10a8fa205782" containerName="istio-proxy"
Jul 16 23:22:53.297988 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.297948    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3fd15e8-f607-449d-a7e3-10a8fa205782" containerName="istio-proxy"
Jul 16 23:22:53.298144 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.298040    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3fd15e8-f607-449d-a7e3-10a8fa205782" containerName="istio-proxy"
Jul 16 23:22:53.298144 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.298054    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3fd15e8-f607-449d-a7e3-10a8fa205782" containerName="istio-proxy"
Jul 16 23:22:53.302721 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.302700    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.305115 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.305093    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-339cb8-openshift-default-dockercfg-pgt5v\""
Jul 16 23:22:53.311375 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.311345    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"]
Jul 16 23:22:53.406830 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.406779    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-token\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.407024 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.406834    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-workload-socket\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.407024 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.406865    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-workload-certs\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.407024 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.406972    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-data\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.407210 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.407048    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istiod-ca-cert\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.407210 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.407100    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-credential-socket\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.407210 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.407176    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-podinfo\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.407405 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.407231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-envoy\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.407405 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.407308    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvr9w\" (UniqueName: \"kubernetes.io/projected/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-kube-api-access-jvr9w\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.508199 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.508161    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-podinfo\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.508453 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.508211    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-envoy\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.508453 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.508357    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvr9w\" (UniqueName: \"kubernetes.io/projected/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-kube-api-access-jvr9w\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.508453 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.508423    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-token\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.508453 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.508446    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-workload-socket\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.509021 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.508466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-workload-certs\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.509021 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.508498    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-data\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.509021 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.508561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istiod-ca-cert\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.509021 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.508613    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-credential-socket\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.509021 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.508860    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-workload-socket\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.509021 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.508885    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-workload-certs\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.509021 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.508945    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-data\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.509021 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.509017    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-credential-socket\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.509513 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.509275    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istiod-ca-cert\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.510532 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.510510    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-envoy\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.510790 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.510774    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-podinfo\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.516229 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.516206    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-token\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.516451 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.516427    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvr9w\" (UniqueName: \"kubernetes.io/projected/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-kube-api-access-jvr9w\") pod \"e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") " pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.615428 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.615387    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:53.763106 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.763015    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"]
Jul 16 23:22:53.765511 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:22:53.765476    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6e953b_8160_4e8d_9c4a_81c6c4ac095e.slice/crio-412d0b3d76fc9b4f497215ec6e1ee0d7a6239d18b6cc3515e2652259fa6eaba1 WatchSource:0}: Error finding container 412d0b3d76fc9b4f497215ec6e1ee0d7a6239d18b6cc3515e2652259fa6eaba1: Status 404 returned error can't find the container with id 412d0b3d76fc9b4f497215ec6e1ee0d7a6239d18b6cc3515e2652259fa6eaba1
Jul 16 23:22:53.767918 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.767880    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:22:53.768025 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.767955    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:22:53.768025 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:53.767984    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:22:54.221967 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:54.221919    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w" event={"ID":"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e","Type":"ContainerStarted","Data":"5ed039e15620b654fb7172e62e3900b9a93a836720a3ccbc8957357cfbf383ea"}
Jul 16 23:22:54.222192 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:54.222001    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w" event={"ID":"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e","Type":"ContainerStarted","Data":"412d0b3d76fc9b4f497215ec6e1ee0d7a6239d18b6cc3515e2652259fa6eaba1"}
Jul 16 23:22:54.245777 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:54.245721    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w" podStartSLOduration=1.24570534 podStartE2EDuration="1.24570534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:22:54.243963095 +0000 UTC m=+2984.992447167" watchObservedRunningTime="2026-07-16 23:22:54.24570534 +0000 UTC m=+2984.994189410"
Jul 16 23:22:54.616336 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:54.616280    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:54.621590 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:54.621561    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:55.226908 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:55.226874    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:55.227824 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:55.227805    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:22:56.201187 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:56.201147    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"]
Jul 16 23:22:56.205114 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:56.205082    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"
Jul 16 23:22:56.207398 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:56.207373    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-339cb8-serving-cert\""
Jul 16 23:22:56.209006 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:56.208985    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"
Jul 16 23:22:56.216882 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:56.216853    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"]
Jul 16 23:22:56.348274 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:56.347814    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca92db7-1cd3-498c-9411-dd0d7ddd3452-maas-api-tls\") pod \"maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76\" (UID: \"5ca92db7-1cd3-498c-9411-dd0d7ddd3452\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"
Jul 16 23:22:56.348274 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:56.347948    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca92db7-1cd3-498c-9411-dd0d7ddd3452-openshift-service-ca\") pod \"maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76\" (UID: \"5ca92db7-1cd3-498c-9411-dd0d7ddd3452\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"
Jul 16 23:22:56.348274 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:56.347982    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wrnb\" (UniqueName: \"kubernetes.io/projected/5ca92db7-1cd3-498c-9411-dd0d7ddd3452-kube-api-access-2wrnb\") pod \"maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76\" (UID: \"5ca92db7-1cd3-498c-9411-dd0d7ddd3452\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"
Jul 16 23:22:56.455784 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:56.455669    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca92db7-1cd3-498c-9411-dd0d7ddd3452-maas-api-tls\") pod \"maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76\" (UID: \"5ca92db7-1cd3-498c-9411-dd0d7ddd3452\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"
Jul 16 23:22:56.455784 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:56.455773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca92db7-1cd3-498c-9411-dd0d7ddd3452-openshift-service-ca\") pod \"maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76\" (UID: \"5ca92db7-1cd3-498c-9411-dd0d7ddd3452\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"
Jul 16 23:22:56.456037 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:56.455808    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wrnb\" (UniqueName: \"kubernetes.io/projected/5ca92db7-1cd3-498c-9411-dd0d7ddd3452-kube-api-access-2wrnb\") pod \"maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76\" (UID: \"5ca92db7-1cd3-498c-9411-dd0d7ddd3452\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"
Jul 16 23:22:56.457654 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:56.457617    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca92db7-1cd3-498c-9411-dd0d7ddd3452-openshift-service-ca\") pod \"maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76\" (UID: \"5ca92db7-1cd3-498c-9411-dd0d7ddd3452\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"
Jul 16 23:22:56.460034 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:56.460001    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca92db7-1cd3-498c-9411-dd0d7ddd3452-maas-api-tls\") pod \"maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76\" (UID: \"5ca92db7-1cd3-498c-9411-dd0d7ddd3452\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"
Jul 16 23:22:56.480349 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:56.480304    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wrnb\" (UniqueName: \"kubernetes.io/projected/5ca92db7-1cd3-498c-9411-dd0d7ddd3452-kube-api-access-2wrnb\") pod \"maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76\" (UID: \"5ca92db7-1cd3-498c-9411-dd0d7ddd3452\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"
Jul 16 23:22:56.518063 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:56.518013    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"
Jul 16 23:22:56.714166 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:56.714072    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"]
Jul 16 23:22:56.717449 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:22:56.717410    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca92db7_1cd3_498c_9411_dd0d7ddd3452.slice/crio-3cb8de376b04a23d3752a865d01e5bd8312300c70cfb80c305f1291a75847f71 WatchSource:0}: Error finding container 3cb8de376b04a23d3752a865d01e5bd8312300c70cfb80c305f1291a75847f71: Status 404 returned error can't find the container with id 3cb8de376b04a23d3752a865d01e5bd8312300c70cfb80c305f1291a75847f71
Jul 16 23:22:57.272731 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:57.272694    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76" event={"ID":"5ca92db7-1cd3-498c-9411-dd0d7ddd3452","Type":"ContainerStarted","Data":"8dfbfec3868b378cca878c465129633d19e53f0c6e71264106fcc7601e14efa3"}
Jul 16 23:22:57.273171 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:57.272915    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76" event={"ID":"5ca92db7-1cd3-498c-9411-dd0d7ddd3452","Type":"ContainerStarted","Data":"3cb8de376b04a23d3752a865d01e5bd8312300c70cfb80c305f1291a75847f71"}
Jul 16 23:22:57.273171 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:57.273022    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"
Jul 16 23:22:57.293758 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:22:57.293693    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76" podStartSLOduration=0.924904793 podStartE2EDuration="1.29367528s" totalImagesPullingTime="368.770487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:22:57.293104066 +0000 UTC m=+2988.041588139" watchObservedRunningTime="2026-07-16 23:22:57.29367528 +0000 UTC m=+2988.042159351"
Jul 16 23:23:02.179384 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:02.179341    2565 patch_prober.go:28] interesting pod/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Jul 16 23:23:02.179850 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:02.179429    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474" podUID="08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Jul 16 23:23:02.301687 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:02.301655    2565 generic.go:356] "Generic (PLEG): container finished" podID="08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" containerID="55bf8c86501c047b71e9a7ae21a9d3d914aeabddff92f961ec86ee423af2d27f" exitCode=0
Jul 16 23:23:02.301878 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:02.301732    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474" event={"ID":"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53","Type":"ContainerDied","Data":"55bf8c86501c047b71e9a7ae21a9d3d914aeabddff92f961ec86ee423af2d27f"}
Jul 16 23:23:02.302163 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:02.302148    2565 scope.go:117] "RemoveContainer" containerID="55bf8c86501c047b71e9a7ae21a9d3d914aeabddff92f961ec86ee423af2d27f"
Jul 16 23:23:02.303193 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:02.303164    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:23:02.303275 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:02.303234    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:23:02.303317 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:02.303290    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:23:03.287086 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:03.287050    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"
Jul 16 23:23:03.309179 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:03.309143    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474" event={"ID":"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53","Type":"ContainerStarted","Data":"5438832f069a3dfffc4d03d0cefbea3653b957351aa19b8488b3101acf670cb6"}
Jul 16 23:23:04.186200 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:04.186158    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:23:04.186455 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:04.186307    2565 patch_prober.go:28] interesting pod/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Jul 16 23:23:04.186455 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:04.186356    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474" podUID="08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Jul 16 23:23:05.186656 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:05.186618    2565 patch_prober.go:28] interesting pod/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Jul 16 23:23:05.187052 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:05.186678    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474" podUID="08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Jul 16 23:23:06.186095 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:06.186054    2565 patch_prober.go:28] interesting pod/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Jul 16 23:23:06.186341 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:06.186111    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474" podUID="08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Jul 16 23:23:07.185853 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:07.185807    2565 patch_prober.go:28] interesting pod/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Jul 16 23:23:07.186463 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:07.185877    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474" podUID="08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Jul 16 23:23:08.190275 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:08.190224    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:23:08.190764 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:08.190518    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:23:08.191689 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:08.191655    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:23:08.338413 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:08.338378    2565 generic.go:356] "Generic (PLEG): container finished" podID="4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" containerID="5ed039e15620b654fb7172e62e3900b9a93a836720a3ccbc8957357cfbf383ea" exitCode=0
Jul 16 23:23:08.338585 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:08.338449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w" event={"ID":"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e","Type":"ContainerDied","Data":"5ed039e15620b654fb7172e62e3900b9a93a836720a3ccbc8957357cfbf383ea"}
Jul 16 23:23:08.338971 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:08.338955    2565 scope.go:117] "RemoveContainer" containerID="5ed039e15620b654fb7172e62e3900b9a93a836720a3ccbc8957357cfbf383ea"
Jul 16 23:23:08.339971 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:08.339935    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:23:08.340093 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:08.340010    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:23:08.340093 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:08.340047    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:23:09.352729 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:09.352688    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w" event={"ID":"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e","Type":"ContainerStarted","Data":"a5e1355ef425643f7b07bdc95fd23a4ebec848dbd99cea1a14e012ded56f0306"}
Jul 16 23:23:09.615786 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:09.615700    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:23:09.617005 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:09.616976    2565 patch_prober.go:28] interesting pod/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Jul 16 23:23:09.617122 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:09.617027    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w" podUID="4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Jul 16 23:23:10.227339 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:10.227288    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:23:10.616266 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:10.616220    2565 patch_prober.go:28] interesting pod/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Jul 16 23:23:10.616778 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:10.616342    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w" podUID="4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Jul 16 23:23:11.615957 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:11.615917    2565 patch_prober.go:28] interesting pod/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Jul 16 23:23:11.616157 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:11.615976    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w" podUID="4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Jul 16 23:23:12.616785 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:12.616741    2565 patch_prober.go:28] interesting pod/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Jul 16 23:23:12.617208 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:12.616815    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w" podUID="4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Jul 16 23:23:13.620457 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:13.620425    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:23:13.621409 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:13.621385    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:23:18.603315 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:18.603276    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"]
Jul 16 23:23:18.603911 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:18.603538    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs" podUID="631d0460-339f-4583-8d6b-259a50119b47" containerName="maas-api" containerID="cri-o://625066ea4c95bf68d88448d2133c7e3aba22f8b2783ae8daefec7581f87c1c71" gracePeriod=30
Jul 16 23:23:18.863643 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:18.863581    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"
Jul 16 23:23:18.991292 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:18.991226    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/631d0460-339f-4583-8d6b-259a50119b47-maas-api-tls\") pod \"631d0460-339f-4583-8d6b-259a50119b47\" (UID: \"631d0460-339f-4583-8d6b-259a50119b47\") "
Jul 16 23:23:18.991462 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:18.991316    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/631d0460-339f-4583-8d6b-259a50119b47-openshift-service-ca\") pod \"631d0460-339f-4583-8d6b-259a50119b47\" (UID: \"631d0460-339f-4583-8d6b-259a50119b47\") "
Jul 16 23:23:18.991462 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:18.991381    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fws7\" (UniqueName: \"kubernetes.io/projected/631d0460-339f-4583-8d6b-259a50119b47-kube-api-access-4fws7\") pod \"631d0460-339f-4583-8d6b-259a50119b47\" (UID: \"631d0460-339f-4583-8d6b-259a50119b47\") "
Jul 16 23:23:18.991703 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:18.991670    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631d0460-339f-4583-8d6b-259a50119b47-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "631d0460-339f-4583-8d6b-259a50119b47" (UID: "631d0460-339f-4583-8d6b-259a50119b47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:23:18.993445 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:18.993407    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631d0460-339f-4583-8d6b-259a50119b47-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "631d0460-339f-4583-8d6b-259a50119b47" (UID: "631d0460-339f-4583-8d6b-259a50119b47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:23:18.993445 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:18.993433    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631d0460-339f-4583-8d6b-259a50119b47-kube-api-access-4fws7" (OuterVolumeSpecName: "kube-api-access-4fws7") pod "631d0460-339f-4583-8d6b-259a50119b47" (UID: "631d0460-339f-4583-8d6b-259a50119b47"). InnerVolumeSpecName "kube-api-access-4fws7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:23:19.092365 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:19.092322    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fws7\" (UniqueName: \"kubernetes.io/projected/631d0460-339f-4583-8d6b-259a50119b47-kube-api-access-4fws7\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:23:19.092365 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:19.092361    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/631d0460-339f-4583-8d6b-259a50119b47-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:23:19.092578 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:19.092377    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/631d0460-339f-4583-8d6b-259a50119b47-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:23:19.411527 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:19.411493    2565 generic.go:356] "Generic (PLEG): container finished" podID="631d0460-339f-4583-8d6b-259a50119b47" containerID="625066ea4c95bf68d88448d2133c7e3aba22f8b2783ae8daefec7581f87c1c71" exitCode=0
Jul 16 23:23:19.411718 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:19.411553    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs" event={"ID":"631d0460-339f-4583-8d6b-259a50119b47","Type":"ContainerDied","Data":"625066ea4c95bf68d88448d2133c7e3aba22f8b2783ae8daefec7581f87c1c71"}
Jul 16 23:23:19.411718 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:19.411616    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs" event={"ID":"631d0460-339f-4583-8d6b-259a50119b47","Type":"ContainerDied","Data":"f1a7900935004d9d31fbb5f85a7067d40595f656eafb204e831a9a725c6d99b4"}
Jul 16 23:23:19.411718 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:19.411553    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"
Jul 16 23:23:19.411718 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:19.411638    2565 scope.go:117] "RemoveContainer" containerID="625066ea4c95bf68d88448d2133c7e3aba22f8b2783ae8daefec7581f87c1c71"
Jul 16 23:23:19.421299 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:19.421222    2565 scope.go:117] "RemoveContainer" containerID="625066ea4c95bf68d88448d2133c7e3aba22f8b2783ae8daefec7581f87c1c71"
Jul 16 23:23:19.421576 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:23:19.421557    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"625066ea4c95bf68d88448d2133c7e3aba22f8b2783ae8daefec7581f87c1c71\": container with ID starting with 625066ea4c95bf68d88448d2133c7e3aba22f8b2783ae8daefec7581f87c1c71 not found: ID does not exist" containerID="625066ea4c95bf68d88448d2133c7e3aba22f8b2783ae8daefec7581f87c1c71"
Jul 16 23:23:19.421626 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:19.421586    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"625066ea4c95bf68d88448d2133c7e3aba22f8b2783ae8daefec7581f87c1c71"} err="failed to get container status \"625066ea4c95bf68d88448d2133c7e3aba22f8b2783ae8daefec7581f87c1c71\": rpc error: code = NotFound desc = could not find container \"625066ea4c95bf68d88448d2133c7e3aba22f8b2783ae8daefec7581f87c1c71\": container with ID starting with 625066ea4c95bf68d88448d2133c7e3aba22f8b2783ae8daefec7581f87c1c71 not found: ID does not exist"
Jul 16 23:23:19.434124 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:19.434093    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"]
Jul 16 23:23:19.437828 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:19.437802    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d302e4-5bcd7965c7-hwcvs"]
Jul 16 23:23:19.726182 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:19.726086    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="631d0460-339f-4583-8d6b-259a50119b47" path="/var/lib/kubelet/pods/631d0460-339f-4583-8d6b-259a50119b47/volumes"
Jul 16 23:23:34.191781 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:34.191736    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"]
Jul 16 23:23:34.192209 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:34.192011    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474" podUID="08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" containerName="istio-proxy" containerID="cri-o://5438832f069a3dfffc4d03d0cefbea3653b957351aa19b8488b3101acf670cb6" gracePeriod=30
Jul 16 23:23:38.193710 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:38.193664    2565 patch_prober.go:28] interesting pod/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 23:23:38.194115 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:38.193737    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474" podUID="08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 23:23:39.449625 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.449595    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:23:39.492598 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.492506    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-credential-socket\") pod \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") "
Jul 16 23:23:39.492598 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.492568    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-data\") pod \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") "
Jul 16 23:23:39.492843 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.492600    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gldfj\" (UniqueName: \"kubernetes.io/projected/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-kube-api-access-gldfj\") pod \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") "
Jul 16 23:23:39.492843 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.492640    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-token\") pod \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") "
Jul 16 23:23:39.492843 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.492695    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-envoy\") pod \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") "
Jul 16 23:23:39.492843 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.492741    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istiod-ca-cert\") pod \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") "
Jul 16 23:23:39.492843 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.492763    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-workload-socket\") pod \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") "
Jul 16 23:23:39.492843 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.492801    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" (UID: "08f4db0c-5f03-4a96-9d24-c5aea6aa8b53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:23:39.493152 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.492856    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-podinfo\") pod \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") "
Jul 16 23:23:39.493152 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.492884    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-workload-certs\") pod \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\" (UID: \"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53\") "
Jul 16 23:23:39.493152 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.493017    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" (UID: "08f4db0c-5f03-4a96-9d24-c5aea6aa8b53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:23:39.493346 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.493219    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-workload-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:23:39.493346 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.493238    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-credential-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:23:39.493346 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.493284    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" (UID: "08f4db0c-5f03-4a96-9d24-c5aea6aa8b53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:23:39.493346 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.493324    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" (UID: "08f4db0c-5f03-4a96-9d24-c5aea6aa8b53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:23:39.493580 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.493557    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-data" (OuterVolumeSpecName: "istio-data") pod "08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" (UID: "08f4db0c-5f03-4a96-9d24-c5aea6aa8b53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:23:39.494948 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.494922    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-kube-api-access-gldfj" (OuterVolumeSpecName: "kube-api-access-gldfj") pod "08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" (UID: "08f4db0c-5f03-4a96-9d24-c5aea6aa8b53"). InnerVolumeSpecName "kube-api-access-gldfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:23:39.495587 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.495562    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" (UID: "08f4db0c-5f03-4a96-9d24-c5aea6aa8b53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:23:39.495670 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.495596    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" (UID: "08f4db0c-5f03-4a96-9d24-c5aea6aa8b53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 23:23:39.495670 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.495605    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-token" (OuterVolumeSpecName: "istio-token") pod "08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" (UID: "08f4db0c-5f03-4a96-9d24-c5aea6aa8b53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:23:39.514217 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.514186    2565 generic.go:356] "Generic (PLEG): container finished" podID="08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" containerID="5438832f069a3dfffc4d03d0cefbea3653b957351aa19b8488b3101acf670cb6" exitCode=0
Jul 16 23:23:39.514378 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.514229    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"
Jul 16 23:23:39.514378 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.514283    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474" event={"ID":"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53","Type":"ContainerDied","Data":"5438832f069a3dfffc4d03d0cefbea3653b957351aa19b8488b3101acf670cb6"}
Jul 16 23:23:39.514378 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.514346    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474" event={"ID":"08f4db0c-5f03-4a96-9d24-c5aea6aa8b53","Type":"ContainerDied","Data":"245db529d253a983dbcfafa6d2a645c52f9afde88404ddea2d43d8e23ef9363e"}
Jul 16 23:23:39.514378 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.514369    2565 scope.go:117] "RemoveContainer" containerID="5438832f069a3dfffc4d03d0cefbea3653b957351aa19b8488b3101acf670cb6"
Jul 16 23:23:39.523549 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.523527    2565 scope.go:117] "RemoveContainer" containerID="55bf8c86501c047b71e9a7ae21a9d3d914aeabddff92f961ec86ee423af2d27f"
Jul 16 23:23:39.533826 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.533792    2565 scope.go:117] "RemoveContainer" containerID="5438832f069a3dfffc4d03d0cefbea3653b957351aa19b8488b3101acf670cb6"
Jul 16 23:23:39.534742 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:23:39.534706    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5438832f069a3dfffc4d03d0cefbea3653b957351aa19b8488b3101acf670cb6\": container with ID starting with 5438832f069a3dfffc4d03d0cefbea3653b957351aa19b8488b3101acf670cb6 not found: ID does not exist" containerID="5438832f069a3dfffc4d03d0cefbea3653b957351aa19b8488b3101acf670cb6"
Jul 16 23:23:39.534904 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.534752    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5438832f069a3dfffc4d03d0cefbea3653b957351aa19b8488b3101acf670cb6"} err="failed to get container status \"5438832f069a3dfffc4d03d0cefbea3653b957351aa19b8488b3101acf670cb6\": rpc error: code = NotFound desc = could not find container \"5438832f069a3dfffc4d03d0cefbea3653b957351aa19b8488b3101acf670cb6\": container with ID starting with 5438832f069a3dfffc4d03d0cefbea3653b957351aa19b8488b3101acf670cb6 not found: ID does not exist"
Jul 16 23:23:39.534904 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.534789    2565 scope.go:117] "RemoveContainer" containerID="55bf8c86501c047b71e9a7ae21a9d3d914aeabddff92f961ec86ee423af2d27f"
Jul 16 23:23:39.535318 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:23:39.535290    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55bf8c86501c047b71e9a7ae21a9d3d914aeabddff92f961ec86ee423af2d27f\": container with ID starting with 55bf8c86501c047b71e9a7ae21a9d3d914aeabddff92f961ec86ee423af2d27f not found: ID does not exist" containerID="55bf8c86501c047b71e9a7ae21a9d3d914aeabddff92f961ec86ee423af2d27f"
Jul 16 23:23:39.535421 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.535329    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55bf8c86501c047b71e9a7ae21a9d3d914aeabddff92f961ec86ee423af2d27f"} err="failed to get container status \"55bf8c86501c047b71e9a7ae21a9d3d914aeabddff92f961ec86ee423af2d27f\": rpc error: code = NotFound desc = could not find container \"55bf8c86501c047b71e9a7ae21a9d3d914aeabddff92f961ec86ee423af2d27f\": container with ID starting with 55bf8c86501c047b71e9a7ae21a9d3d914aeabddff92f961ec86ee423af2d27f not found: ID does not exist"
Jul 16 23:23:39.537180 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.537160    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"]
Jul 16 23:23:39.540735 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.540712    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d302e4-openshift-default-6b8fd4979f-kh474"]
Jul 16 23:23:39.594430 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.594383    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istiod-ca-cert\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:23:39.594430 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.594421    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-podinfo\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:23:39.594430 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.594430    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-workload-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:23:39.594430 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.594438    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-data\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:23:39.594715 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.594447    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gldfj\" (UniqueName: \"kubernetes.io/projected/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-kube-api-access-gldfj\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:23:39.594715 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.594456    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-token\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:23:39.594715 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.594465    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53-istio-envoy\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:23:39.726815 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:39.726779    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" path="/var/lib/kubelet/pods/08f4db0c-5f03-4a96-9d24-c5aea6aa8b53/volumes"
Jul 16 23:23:45.016405 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.016362    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"]
Jul 16 23:23:45.016900 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.016636    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76" podUID="5ca92db7-1cd3-498c-9411-dd0d7ddd3452" containerName="maas-api" containerID="cri-o://8dfbfec3868b378cca878c465129633d19e53f0c6e71264106fcc7601e14efa3" gracePeriod=30
Jul 16 23:23:45.273660 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.273591    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"
Jul 16 23:23:45.349117 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.349079    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca92db7-1cd3-498c-9411-dd0d7ddd3452-openshift-service-ca\") pod \"5ca92db7-1cd3-498c-9411-dd0d7ddd3452\" (UID: \"5ca92db7-1cd3-498c-9411-dd0d7ddd3452\") "
Jul 16 23:23:45.349359 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.349189    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca92db7-1cd3-498c-9411-dd0d7ddd3452-maas-api-tls\") pod \"5ca92db7-1cd3-498c-9411-dd0d7ddd3452\" (UID: \"5ca92db7-1cd3-498c-9411-dd0d7ddd3452\") "
Jul 16 23:23:45.349359 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.349225    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wrnb\" (UniqueName: \"kubernetes.io/projected/5ca92db7-1cd3-498c-9411-dd0d7ddd3452-kube-api-access-2wrnb\") pod \"5ca92db7-1cd3-498c-9411-dd0d7ddd3452\" (UID: \"5ca92db7-1cd3-498c-9411-dd0d7ddd3452\") "
Jul 16 23:23:45.349494 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.349446    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca92db7-1cd3-498c-9411-dd0d7ddd3452-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5ca92db7-1cd3-498c-9411-dd0d7ddd3452" (UID: "5ca92db7-1cd3-498c-9411-dd0d7ddd3452"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:23:45.351354 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.351329    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca92db7-1cd3-498c-9411-dd0d7ddd3452-kube-api-access-2wrnb" (OuterVolumeSpecName: "kube-api-access-2wrnb") pod "5ca92db7-1cd3-498c-9411-dd0d7ddd3452" (UID: "5ca92db7-1cd3-498c-9411-dd0d7ddd3452"). InnerVolumeSpecName "kube-api-access-2wrnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:23:45.351354 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.351343    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ca92db7-1cd3-498c-9411-dd0d7ddd3452-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5ca92db7-1cd3-498c-9411-dd0d7ddd3452" (UID: "5ca92db7-1cd3-498c-9411-dd0d7ddd3452"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:23:45.450515 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.450482    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ca92db7-1cd3-498c-9411-dd0d7ddd3452-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:23:45.450515 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.450509    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wrnb\" (UniqueName: \"kubernetes.io/projected/5ca92db7-1cd3-498c-9411-dd0d7ddd3452-kube-api-access-2wrnb\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:23:45.450515 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.450519    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca92db7-1cd3-498c-9411-dd0d7ddd3452-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:23:45.546218 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.546132    2565 generic.go:356] "Generic (PLEG): container finished" podID="5ca92db7-1cd3-498c-9411-dd0d7ddd3452" containerID="8dfbfec3868b378cca878c465129633d19e53f0c6e71264106fcc7601e14efa3" exitCode=0
Jul 16 23:23:45.546218 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.546184    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"
Jul 16 23:23:45.546425 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.546220    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76" event={"ID":"5ca92db7-1cd3-498c-9411-dd0d7ddd3452","Type":"ContainerDied","Data":"8dfbfec3868b378cca878c465129633d19e53f0c6e71264106fcc7601e14efa3"}
Jul 16 23:23:45.546425 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.546292    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76" event={"ID":"5ca92db7-1cd3-498c-9411-dd0d7ddd3452","Type":"ContainerDied","Data":"3cb8de376b04a23d3752a865d01e5bd8312300c70cfb80c305f1291a75847f71"}
Jul 16 23:23:45.546425 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.546316    2565 scope.go:117] "RemoveContainer" containerID="8dfbfec3868b378cca878c465129633d19e53f0c6e71264106fcc7601e14efa3"
Jul 16 23:23:45.555195 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.555179    2565 scope.go:117] "RemoveContainer" containerID="8dfbfec3868b378cca878c465129633d19e53f0c6e71264106fcc7601e14efa3"
Jul 16 23:23:45.555463 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:23:45.555447    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dfbfec3868b378cca878c465129633d19e53f0c6e71264106fcc7601e14efa3\": container with ID starting with 8dfbfec3868b378cca878c465129633d19e53f0c6e71264106fcc7601e14efa3 not found: ID does not exist" containerID="8dfbfec3868b378cca878c465129633d19e53f0c6e71264106fcc7601e14efa3"
Jul 16 23:23:45.555518 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.555472    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dfbfec3868b378cca878c465129633d19e53f0c6e71264106fcc7601e14efa3"} err="failed to get container status \"8dfbfec3868b378cca878c465129633d19e53f0c6e71264106fcc7601e14efa3\": rpc error: code = NotFound desc = could not find container \"8dfbfec3868b378cca878c465129633d19e53f0c6e71264106fcc7601e14efa3\": container with ID starting with 8dfbfec3868b378cca878c465129633d19e53f0c6e71264106fcc7601e14efa3 not found: ID does not exist"
Jul 16 23:23:45.567876 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.567839    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"]
Jul 16 23:23:45.574300 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.574272    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-339cb8-58cc86f7bd-sbh76"]
Jul 16 23:23:45.726577 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:45.726541    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ca92db7-1cd3-498c-9411-dd0d7ddd3452" path="/var/lib/kubelet/pods/5ca92db7-1cd3-498c-9411-dd0d7ddd3452/volumes"
Jul 16 23:23:50.982082 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:50.982046    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx"]
Jul 16 23:23:50.982602 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:50.982417    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="631d0460-339f-4583-8d6b-259a50119b47" containerName="maas-api"
Jul 16 23:23:50.982602 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:50.982429    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="631d0460-339f-4583-8d6b-259a50119b47" containerName="maas-api"
Jul 16 23:23:50.982602 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:50.982439    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" containerName="istio-proxy"
Jul 16 23:23:50.982602 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:50.982445    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" containerName="istio-proxy"
Jul 16 23:23:50.982602 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:50.982468    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" containerName="istio-proxy"
Jul 16 23:23:50.982602 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:50.982475    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" containerName="istio-proxy"
Jul 16 23:23:50.982602 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:50.982483    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ca92db7-1cd3-498c-9411-dd0d7ddd3452" containerName="maas-api"
Jul 16 23:23:50.982602 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:50.982488    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ca92db7-1cd3-498c-9411-dd0d7ddd3452" containerName="maas-api"
Jul 16 23:23:50.982602 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:50.982557    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="631d0460-339f-4583-8d6b-259a50119b47" containerName="maas-api"
Jul 16 23:23:50.982602 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:50.982565    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" containerName="istio-proxy"
Jul 16 23:23:50.982602 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:50.982572    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ca92db7-1cd3-498c-9411-dd0d7ddd3452" containerName="maas-api"
Jul 16 23:23:50.985257 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:50.985223    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx"
Jul 16 23:23:50.987371 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:50.987347    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:23:50.992215 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:50.992185    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx"]
Jul 16 23:23:51.103922 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:51.103885    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b3be06-71b1-4549-95c5-0bf6b3299463-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx\" (UID: \"26b3be06-71b1-4549-95c5-0bf6b3299463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx"
Jul 16 23:23:51.204955 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:51.204924    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b3be06-71b1-4549-95c5-0bf6b3299463-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx\" (UID: \"26b3be06-71b1-4549-95c5-0bf6b3299463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx"
Jul 16 23:23:51.205591 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:51.205572    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b3be06-71b1-4549-95c5-0bf6b3299463-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx\" (UID: \"26b3be06-71b1-4549-95c5-0bf6b3299463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx"
Jul 16 23:23:51.297990 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:51.297904    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx"
Jul 16 23:23:51.437677 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:51.437647    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx"]
Jul 16 23:23:51.439944 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:23:51.439920    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b3be06_71b1_4549_95c5_0bf6b3299463.slice/crio-ec73c8e50cb24d86d9d20214c1bb31c82bf2a429b45ca437e62b8ea4da9ff189 WatchSource:0}: Error finding container ec73c8e50cb24d86d9d20214c1bb31c82bf2a429b45ca437e62b8ea4da9ff189: Status 404 returned error can't find the container with id ec73c8e50cb24d86d9d20214c1bb31c82bf2a429b45ca437e62b8ea4da9ff189
Jul 16 23:23:51.578181 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:51.578149    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx" event={"ID":"26b3be06-71b1-4549-95c5-0bf6b3299463","Type":"ContainerStarted","Data":"ec73c8e50cb24d86d9d20214c1bb31c82bf2a429b45ca437e62b8ea4da9ff189"}
Jul 16 23:23:52.585004 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:52.584963    2565 generic.go:356] "Generic (PLEG): container finished" podID="26b3be06-71b1-4549-95c5-0bf6b3299463" containerID="e944c6ad0433be02cc2a64772f4f71ffedaa06a5f61a856dd540360fba4f3544" exitCode=6
Jul 16 23:23:52.585434 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:52.585034    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx" event={"ID":"26b3be06-71b1-4549-95c5-0bf6b3299463","Type":"ContainerDied","Data":"e944c6ad0433be02cc2a64772f4f71ffedaa06a5f61a856dd540360fba4f3544"}
Jul 16 23:23:52.585434 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:52.585276    2565 scope.go:117] "RemoveContainer" containerID="e944c6ad0433be02cc2a64772f4f71ffedaa06a5f61a856dd540360fba4f3544"
Jul 16 23:23:53.594846 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:53.594811    2565 generic.go:356] "Generic (PLEG): container finished" podID="26b3be06-71b1-4549-95c5-0bf6b3299463" containerID="29b0930caaeba1e50913a8b4646ca52b662a878dbdf3b5990556d7463b686792" exitCode=6
Jul 16 23:23:53.595353 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:53.594895    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx" event={"ID":"26b3be06-71b1-4549-95c5-0bf6b3299463","Type":"ContainerDied","Data":"29b0930caaeba1e50913a8b4646ca52b662a878dbdf3b5990556d7463b686792"}
Jul 16 23:23:53.595353 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:53.594957    2565 scope.go:117] "RemoveContainer" containerID="e944c6ad0433be02cc2a64772f4f71ffedaa06a5f61a856dd540360fba4f3544"
Jul 16 23:23:53.595353 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:53.595218    2565 scope.go:117] "RemoveContainer" containerID="29b0930caaeba1e50913a8b4646ca52b662a878dbdf3b5990556d7463b686792"
Jul 16 23:23:53.595545 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:23:53.595473    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx_odh-ai-gateway-infra(26b3be06-71b1-4549-95c5-0bf6b3299463)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx" podUID="26b3be06-71b1-4549-95c5-0bf6b3299463"
Jul 16 23:23:54.601664 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:23:54.601636    2565 scope.go:117] "RemoveContainer" containerID="29b0930caaeba1e50913a8b4646ca52b662a878dbdf3b5990556d7463b686792"
Jul 16 23:23:54.602101 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:23:54.601922    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx_odh-ai-gateway-infra(26b3be06-71b1-4549-95c5-0bf6b3299463)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx" podUID="26b3be06-71b1-4549-95c5-0bf6b3299463"
Jul 16 23:24:00.272362 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:00.272316    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"]
Jul 16 23:24:00.272859 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:00.272550    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w" podUID="4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" containerName="istio-proxy" containerID="cri-o://a5e1355ef425643f7b07bdc95fd23a4ebec848dbd99cea1a14e012ded56f0306" gracePeriod=30
Jul 16 23:24:01.131234 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.131193    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"]
Jul 16 23:24:01.131767 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.131745    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="08f4db0c-5f03-4a96-9d24-c5aea6aa8b53" containerName="istio-proxy"
Jul 16 23:24:01.135857 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.135834    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.138016 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.137995    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-48992b-openshift-default-dockercfg-tph5k\""
Jul 16 23:24:01.146155 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.146125    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"]
Jul 16 23:24:01.204977 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.204943    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-podinfo\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.205167 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.204984    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-envoy\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.205167 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.205006    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-workload-certs\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.205167 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.205101    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-credential-socket\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.205167 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.205134    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-token\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.205167 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.205152    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-data\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.205373 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.205274    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6dr4\" (UniqueName: \"kubernetes.io/projected/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-kube-api-access-c6dr4\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.205373 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.205365    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-workload-socket\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.205441 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.205403    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istiod-ca-cert\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.306330 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.306285    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-credential-socket\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.306330 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.306331    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-token\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.306884 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.306350    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-data\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.306884 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.306386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6dr4\" (UniqueName: \"kubernetes.io/projected/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-kube-api-access-c6dr4\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.306884 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.306445    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-workload-socket\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.306884 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.306490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istiod-ca-cert\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.306884 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.306512    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-podinfo\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.306884 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.306542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-envoy\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.306884 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.306695    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-credential-socket\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.306884 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.306721    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-workload-certs\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.306884 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.306767    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-data\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.306884 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.306863    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-workload-socket\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.307277 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.307104    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-workload-certs\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.307277 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.307212    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istiod-ca-cert\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.308960 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.308942    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-envoy\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.309104 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.309086    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-podinfo\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.314170 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.314147    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-token\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.314309 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.314197    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6dr4\" (UniqueName: \"kubernetes.io/projected/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-kube-api-access-c6dr4\") pod \"e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") " pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.447362 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.447259    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:01.595466 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.595438    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"]
Jul 16 23:24:01.597945 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:24:01.597913    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7352f0_4d62_4dd6_aaae_d1fc9a8c740b.slice/crio-1a42b7e2643c5e1803aaab47b846a28b001813fb37668a7665eafaee48acaf57 WatchSource:0}: Error finding container 1a42b7e2643c5e1803aaab47b846a28b001813fb37668a7665eafaee48acaf57: Status 404 returned error can't find the container with id 1a42b7e2643c5e1803aaab47b846a28b001813fb37668a7665eafaee48acaf57
Jul 16 23:24:01.600293 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.600223    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:24:01.600414 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.600333    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:24:01.600414 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.600375    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:24:01.638064 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:01.638030    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g" event={"ID":"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b","Type":"ContainerStarted","Data":"1a42b7e2643c5e1803aaab47b846a28b001813fb37668a7665eafaee48acaf57"}
Jul 16 23:24:02.644144 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:02.644099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g" event={"ID":"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b","Type":"ContainerStarted","Data":"38d85c9eb45021204d97762633ae9f8cd6d51ed7ca17395fd40816a9fba3edfb"}
Jul 16 23:24:02.663943 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:02.663878    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g" podStartSLOduration=1.6638556389999999 podStartE2EDuration="1.663855639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:24:02.66289141 +0000 UTC m=+3053.411375504" watchObservedRunningTime="2026-07-16 23:24:02.663855639 +0000 UTC m=+3053.412339711"
Jul 16 23:24:03.448016 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:03.447982    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:03.452751 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:03.452725    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:03.650935 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:03.650897    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:03.651856 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:03.651839    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:04.752533 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:04.752501    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"]
Jul 16 23:24:04.756112 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:04.756089    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"
Jul 16 23:24:04.760040 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:04.760009    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-48992b-serving-cert\""
Jul 16 23:24:04.767889 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:04.767861    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"]
Jul 16 23:24:04.842304 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:04.842237    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv\" (UID: \"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"
Jul 16 23:24:04.842492 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:04.842366    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f-maas-api-tls\") pod \"maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv\" (UID: \"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"
Jul 16 23:24:04.842492 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:04.842458    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pk6x\" (UniqueName: \"kubernetes.io/projected/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f-kube-api-access-7pk6x\") pod \"maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv\" (UID: \"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"
Jul 16 23:24:04.943397 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:04.943359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pk6x\" (UniqueName: \"kubernetes.io/projected/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f-kube-api-access-7pk6x\") pod \"maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv\" (UID: \"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"
Jul 16 23:24:04.943624 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:04.943430    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv\" (UID: \"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"
Jul 16 23:24:04.943624 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:04.943472    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f-maas-api-tls\") pod \"maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv\" (UID: \"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"
Jul 16 23:24:04.944106 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:04.944081    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv\" (UID: \"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"
Jul 16 23:24:04.946031 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:04.946013    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f-maas-api-tls\") pod \"maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv\" (UID: \"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"
Jul 16 23:24:04.951969 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:04.951945    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pk6x\" (UniqueName: \"kubernetes.io/projected/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f-kube-api-access-7pk6x\") pod \"maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv\" (UID: \"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"
Jul 16 23:24:05.067000 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.066903    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"
Jul 16 23:24:05.208155 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.208127    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"]
Jul 16 23:24:05.210319 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:24:05.210290    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9049fc_bd3b_404c_8d9a_85dbb9e6ad1f.slice/crio-b6e304174a9bf4ed0a5393fc20f3726b88583bc64fd61068d52a668e3a4c30cd WatchSource:0}: Error finding container b6e304174a9bf4ed0a5393fc20f3726b88583bc64fd61068d52a668e3a4c30cd: Status 404 returned error can't find the container with id b6e304174a9bf4ed0a5393fc20f3726b88583bc64fd61068d52a668e3a4c30cd
Jul 16 23:24:05.211853 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.211829    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 23:24:05.519401 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.519371    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:24:05.650841 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.650808    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-token\") pod \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") "
Jul 16 23:24:05.650959 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.650883    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-envoy\") pod \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") "
Jul 16 23:24:05.650959 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.650919    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-workload-socket\") pod \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") "
Jul 16 23:24:05.651068 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.650962    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-podinfo\") pod \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") "
Jul 16 23:24:05.651068 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.651002    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-credential-socket\") pod \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") "
Jul 16 23:24:05.651068 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.651036    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-workload-certs\") pod \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") "
Jul 16 23:24:05.651224 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.651101    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-data\") pod \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") "
Jul 16 23:24:05.651224 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.651178    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istiod-ca-cert\") pod \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") "
Jul 16 23:24:05.651349 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.651240    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvr9w\" (UniqueName: \"kubernetes.io/projected/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-kube-api-access-jvr9w\") pod \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\" (UID: \"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e\") "
Jul 16 23:24:05.651349 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.651279    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" (UID: "4b6e953b-8160-4e8d-9c4a-81c6c4ac095e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:24:05.651349 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.651306    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" (UID: "4b6e953b-8160-4e8d-9c4a-81c6c4ac095e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:24:05.651505 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.651452    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" (UID: "4b6e953b-8160-4e8d-9c4a-81c6c4ac095e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:24:05.651647 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.651625    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-workload-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:24:05.651738 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.651642    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" (UID: "4b6e953b-8160-4e8d-9c4a-81c6c4ac095e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:24:05.651738 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.651652    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-credential-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:24:05.651738 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.651685    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-workload-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:24:05.652030 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.652004    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-data" (OuterVolumeSpecName: "istio-data") pod "4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" (UID: "4b6e953b-8160-4e8d-9c4a-81c6c4ac095e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:24:05.653385 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.653362    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" (UID: "4b6e953b-8160-4e8d-9c4a-81c6c4ac095e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 23:24:05.653607 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.653588    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" (UID: "4b6e953b-8160-4e8d-9c4a-81c6c4ac095e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:24:05.653676 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.653651    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-token" (OuterVolumeSpecName: "istio-token") pod "4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" (UID: "4b6e953b-8160-4e8d-9c4a-81c6c4ac095e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:24:05.653860 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.653840    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-kube-api-access-jvr9w" (OuterVolumeSpecName: "kube-api-access-jvr9w") pod "4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" (UID: "4b6e953b-8160-4e8d-9c4a-81c6c4ac095e"). InnerVolumeSpecName "kube-api-access-jvr9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:24:05.663772 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.663734    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv" event={"ID":"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f","Type":"ContainerStarted","Data":"b6e304174a9bf4ed0a5393fc20f3726b88583bc64fd61068d52a668e3a4c30cd"}
Jul 16 23:24:05.665641 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.665616    2565 generic.go:356] "Generic (PLEG): container finished" podID="4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" containerID="a5e1355ef425643f7b07bdc95fd23a4ebec848dbd99cea1a14e012ded56f0306" exitCode=0
Jul 16 23:24:05.665732 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.665658    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"
Jul 16 23:24:05.665732 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.665695    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w" event={"ID":"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e","Type":"ContainerDied","Data":"a5e1355ef425643f7b07bdc95fd23a4ebec848dbd99cea1a14e012ded56f0306"}
Jul 16 23:24:05.665813 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.665753    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w" event={"ID":"4b6e953b-8160-4e8d-9c4a-81c6c4ac095e","Type":"ContainerDied","Data":"412d0b3d76fc9b4f497215ec6e1ee0d7a6239d18b6cc3515e2652259fa6eaba1"}
Jul 16 23:24:05.665813 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.665785    2565 scope.go:117] "RemoveContainer" containerID="a5e1355ef425643f7b07bdc95fd23a4ebec848dbd99cea1a14e012ded56f0306"
Jul 16 23:24:05.675119 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.675100    2565 scope.go:117] "RemoveContainer" containerID="5ed039e15620b654fb7172e62e3900b9a93a836720a3ccbc8957357cfbf383ea"
Jul 16 23:24:05.683811 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.683786    2565 scope.go:117] "RemoveContainer" containerID="a5e1355ef425643f7b07bdc95fd23a4ebec848dbd99cea1a14e012ded56f0306"
Jul 16 23:24:05.684112 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:24:05.684094    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e1355ef425643f7b07bdc95fd23a4ebec848dbd99cea1a14e012ded56f0306\": container with ID starting with a5e1355ef425643f7b07bdc95fd23a4ebec848dbd99cea1a14e012ded56f0306 not found: ID does not exist" containerID="a5e1355ef425643f7b07bdc95fd23a4ebec848dbd99cea1a14e012ded56f0306"
Jul 16 23:24:05.684202 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.684121    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e1355ef425643f7b07bdc95fd23a4ebec848dbd99cea1a14e012ded56f0306"} err="failed to get container status \"a5e1355ef425643f7b07bdc95fd23a4ebec848dbd99cea1a14e012ded56f0306\": rpc error: code = NotFound desc = could not find container \"a5e1355ef425643f7b07bdc95fd23a4ebec848dbd99cea1a14e012ded56f0306\": container with ID starting with a5e1355ef425643f7b07bdc95fd23a4ebec848dbd99cea1a14e012ded56f0306 not found: ID does not exist"
Jul 16 23:24:05.684202 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.684139    2565 scope.go:117] "RemoveContainer" containerID="5ed039e15620b654fb7172e62e3900b9a93a836720a3ccbc8957357cfbf383ea"
Jul 16 23:24:05.684459 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:24:05.684441    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed039e15620b654fb7172e62e3900b9a93a836720a3ccbc8957357cfbf383ea\": container with ID starting with 5ed039e15620b654fb7172e62e3900b9a93a836720a3ccbc8957357cfbf383ea not found: ID does not exist" containerID="5ed039e15620b654fb7172e62e3900b9a93a836720a3ccbc8957357cfbf383ea"
Jul 16 23:24:05.684505 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.684466    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed039e15620b654fb7172e62e3900b9a93a836720a3ccbc8957357cfbf383ea"} err="failed to get container status \"5ed039e15620b654fb7172e62e3900b9a93a836720a3ccbc8957357cfbf383ea\": rpc error: code = NotFound desc = could not find container \"5ed039e15620b654fb7172e62e3900b9a93a836720a3ccbc8957357cfbf383ea\": container with ID starting with 5ed039e15620b654fb7172e62e3900b9a93a836720a3ccbc8957357cfbf383ea not found: ID does not exist"
Jul 16 23:24:05.687563 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.687540    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"]
Jul 16 23:24:05.693940 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.693918    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-339cb8-openshift-default-5b4fcdc68f-r4b5w"]
Jul 16 23:24:05.727141 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.727108    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" path="/var/lib/kubelet/pods/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e/volumes"
Jul 16 23:24:05.752635 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.752593    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-data\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:24:05.752635 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.752627    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istiod-ca-cert\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:24:05.752635 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.752640    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvr9w\" (UniqueName: \"kubernetes.io/projected/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-kube-api-access-jvr9w\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:24:05.753083 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.752654    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-token\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:24:05.753083 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.752667    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-envoy\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:24:05.753083 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:05.752677    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6e953b-8160-4e8d-9c4a-81c6c4ac095e-istio-podinfo\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:24:06.673035 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:06.672990    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv" event={"ID":"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f","Type":"ContainerStarted","Data":"fa5e157d2a337b735c1b6b79a2691a3f2b7aaa6dec660a6e8d105521f6ef26a6"}
Jul 16 23:24:06.673295 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:06.673184    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"
Jul 16 23:24:06.688196 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:06.688135    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv" podStartSLOduration=2.287382908 podStartE2EDuration="2.688119946s" totalImagesPullingTime="400.737038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:24:06.687850564 +0000 UTC m=+3057.436334635" watchObservedRunningTime="2026-07-16 23:24:06.688119946 +0000 UTC m=+3057.436604017"
Jul 16 23:24:06.721084 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:06.721049    2565 scope.go:117] "RemoveContainer" containerID="29b0930caaeba1e50913a8b4646ca52b662a878dbdf3b5990556d7463b686792"
Jul 16 23:24:07.681744 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:07.681713    2565 generic.go:356] "Generic (PLEG): container finished" podID="26b3be06-71b1-4549-95c5-0bf6b3299463" containerID="fb93133af29b31c3b21a8ad675bdcd5db2968f16d66dbe49fa26e818826aab7a" exitCode=6
Jul 16 23:24:07.682171 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:07.681791    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx" event={"ID":"26b3be06-71b1-4549-95c5-0bf6b3299463","Type":"ContainerDied","Data":"fb93133af29b31c3b21a8ad675bdcd5db2968f16d66dbe49fa26e818826aab7a"}
Jul 16 23:24:07.682171 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:07.681865    2565 scope.go:117] "RemoveContainer" containerID="29b0930caaeba1e50913a8b4646ca52b662a878dbdf3b5990556d7463b686792"
Jul 16 23:24:07.682309 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:07.682207    2565 scope.go:117] "RemoveContainer" containerID="fb93133af29b31c3b21a8ad675bdcd5db2968f16d66dbe49fa26e818826aab7a"
Jul 16 23:24:07.682463 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:24:07.682443    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx_odh-ai-gateway-infra(26b3be06-71b1-4549-95c5-0bf6b3299463)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx" podUID="26b3be06-71b1-4549-95c5-0bf6b3299463"
Jul 16 23:24:08.718440 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:08.718401    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx"]
Jul 16 23:24:08.857379 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:08.857330    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx"
Jul 16 23:24:08.985557 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:08.985451    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b3be06-71b1-4549-95c5-0bf6b3299463-maas-api-service-ca\") pod \"26b3be06-71b1-4549-95c5-0bf6b3299463\" (UID: \"26b3be06-71b1-4549-95c5-0bf6b3299463\") "
Jul 16 23:24:08.985830 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:08.985804    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b3be06-71b1-4549-95c5-0bf6b3299463-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "26b3be06-71b1-4549-95c5-0bf6b3299463" (UID: "26b3be06-71b1-4549-95c5-0bf6b3299463"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:24:09.086513 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.086466    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26b3be06-71b1-4549-95c5-0bf6b3299463-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:24:09.666012 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.665968    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"]
Jul 16 23:24:09.666376 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.666360    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26b3be06-71b1-4549-95c5-0bf6b3299463" containerName="revoke-keys"
Jul 16 23:24:09.666376 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.666375    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="26b3be06-71b1-4549-95c5-0bf6b3299463" containerName="revoke-keys"
Jul 16 23:24:09.666524 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.666386    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26b3be06-71b1-4549-95c5-0bf6b3299463" containerName="revoke-keys"
Jul 16 23:24:09.666524 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.666392    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="26b3be06-71b1-4549-95c5-0bf6b3299463" containerName="revoke-keys"
Jul 16 23:24:09.666524 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.666410    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" containerName="istio-proxy"
Jul 16 23:24:09.666524 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.666415    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" containerName="istio-proxy"
Jul 16 23:24:09.666524 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.666424    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26b3be06-71b1-4549-95c5-0bf6b3299463" containerName="revoke-keys"
Jul 16 23:24:09.666524 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.666430    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="26b3be06-71b1-4549-95c5-0bf6b3299463" containerName="revoke-keys"
Jul 16 23:24:09.666524 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.666436    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" containerName="istio-proxy"
Jul 16 23:24:09.666524 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.666441    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" containerName="istio-proxy"
Jul 16 23:24:09.666524 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.666503    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" containerName="istio-proxy"
Jul 16 23:24:09.666524 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.666511    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b6e953b-8160-4e8d-9c4a-81c6c4ac095e" containerName="istio-proxy"
Jul 16 23:24:09.666524 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.666518    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="26b3be06-71b1-4549-95c5-0bf6b3299463" containerName="revoke-keys"
Jul 16 23:24:09.666524 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.666527    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="26b3be06-71b1-4549-95c5-0bf6b3299463" containerName="revoke-keys"
Jul 16 23:24:09.669818 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.669794    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.672437 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.672411    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-b4ab52-openshift-default-dockercfg-cvkfv\""
Jul 16 23:24:09.683714 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.683685    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"]
Jul 16 23:24:09.696372 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.696315    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx" event={"ID":"26b3be06-71b1-4549-95c5-0bf6b3299463","Type":"ContainerDied","Data":"ec73c8e50cb24d86d9d20214c1bb31c82bf2a429b45ca437e62b8ea4da9ff189"}
Jul 16 23:24:09.696582 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.696381    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx"
Jul 16 23:24:09.696582 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.696399    2565 scope.go:117] "RemoveContainer" containerID="fb93133af29b31c3b21a8ad675bdcd5db2968f16d66dbe49fa26e818826aab7a"
Jul 16 23:24:09.720211 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.720152    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx"]
Jul 16 23:24:09.727491 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.727465    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-f58rx"]
Jul 16 23:24:09.793789 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.793742    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istiod-ca-cert\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.793789 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.793792    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-data\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.794065 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.793850    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-workload-socket\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.794065 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.793881    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-credential-socket\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.794065 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.793905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-envoy\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.794065 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.793935    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2fk5\" (UniqueName: \"kubernetes.io/projected/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-kube-api-access-t2fk5\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.794065 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.793958    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-token\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.794065 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.793989    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-podinfo\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.794419 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.794073    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-workload-certs\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.894804 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.894764    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2fk5\" (UniqueName: \"kubernetes.io/projected/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-kube-api-access-t2fk5\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.895041 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.894829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-token\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.895041 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.895020    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-podinfo\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.895188 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.895140    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-workload-certs\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.895284 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.895227    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istiod-ca-cert\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.895362 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.895287    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-data\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.895362 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.895338    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-workload-socket\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.895473 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.895374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-credential-socket\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.895473 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.895413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-envoy\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.895578 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.895546    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-workload-certs\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.895637 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.895616    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-data\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.895797 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.895755    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-credential-socket\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.895864 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.895796    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-workload-socket\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.896004 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.895985    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istiod-ca-cert\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.897862 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.897846    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-envoy\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.898192 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.898173    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-podinfo\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.902939 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.902912    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-token\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.903059 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.903038    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2fk5\" (UniqueName: \"kubernetes.io/projected/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-kube-api-access-t2fk5\") pod \"e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") " pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:09.981157 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:09.981066    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:10.128818 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:10.128701    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"]
Jul 16 23:24:10.135758 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:10.135719    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:24:10.135908 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:10.135795    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:24:10.135908 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:10.135830    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:24:10.703341 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:10.703299    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx" event={"ID":"a8b1b47a-d704-46ed-b3a3-ec9f324d4224","Type":"ContainerStarted","Data":"546ed43b169b22a06668b9026c50592408775c5de9874c4ad61a954f7f196b94"}
Jul 16 23:24:10.703546 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:10.703385    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx" event={"ID":"a8b1b47a-d704-46ed-b3a3-ec9f324d4224","Type":"ContainerStarted","Data":"73726d7b8d08bcc43fd7dd8bda867712ba5f224f80fa3ede94028333a1459561"}
Jul 16 23:24:10.752661 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:10.752606    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx" podStartSLOduration=1.752590059 podStartE2EDuration="1.752590059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:24:10.749177506 +0000 UTC m=+3061.497661576" watchObservedRunningTime="2026-07-16 23:24:10.752590059 +0000 UTC m=+3061.501074130"
Jul 16 23:24:10.981309 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:10.981188    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:11.726672 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:11.726635    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26b3be06-71b1-4549-95c5-0bf6b3299463" path="/var/lib/kubelet/pods/26b3be06-71b1-4549-95c5-0bf6b3299463/volumes"
Jul 16 23:24:11.987159 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:11.987074    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:12.688284 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:12.688235    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"
Jul 16 23:24:12.691394 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:12.691363    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"]
Jul 16 23:24:12.691907 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:12.691888    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="26b3be06-71b1-4549-95c5-0bf6b3299463" containerName="revoke-keys"
Jul 16 23:24:12.695456 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:12.695434    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"
Jul 16 23:24:12.698409 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:12.698387    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-b4ab52-serving-cert\""
Jul 16 23:24:12.707685 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:12.707662    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"]
Jul 16 23:24:12.714894 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:12.714862    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:12.716132 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:12.716109    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:12.823156 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:12.823114    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gslq\" (UniqueName: \"kubernetes.io/projected/a4975fd1-65d7-45e2-8f0b-c940c23a5606-kube-api-access-8gslq\") pod \"maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt\" (UID: \"a4975fd1-65d7-45e2-8f0b-c940c23a5606\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"
Jul 16 23:24:12.823394 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:12.823185    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4975fd1-65d7-45e2-8f0b-c940c23a5606-maas-api-tls\") pod \"maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt\" (UID: \"a4975fd1-65d7-45e2-8f0b-c940c23a5606\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"
Jul 16 23:24:12.823394 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:12.823210    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4975fd1-65d7-45e2-8f0b-c940c23a5606-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt\" (UID: \"a4975fd1-65d7-45e2-8f0b-c940c23a5606\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"
Jul 16 23:24:12.924976 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:12.924937    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gslq\" (UniqueName: \"kubernetes.io/projected/a4975fd1-65d7-45e2-8f0b-c940c23a5606-kube-api-access-8gslq\") pod \"maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt\" (UID: \"a4975fd1-65d7-45e2-8f0b-c940c23a5606\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"
Jul 16 23:24:12.925161 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:12.925000    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4975fd1-65d7-45e2-8f0b-c940c23a5606-maas-api-tls\") pod \"maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt\" (UID: \"a4975fd1-65d7-45e2-8f0b-c940c23a5606\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"
Jul 16 23:24:12.925161 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:12.925030    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4975fd1-65d7-45e2-8f0b-c940c23a5606-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt\" (UID: \"a4975fd1-65d7-45e2-8f0b-c940c23a5606\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"
Jul 16 23:24:12.925697 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:12.925672    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4975fd1-65d7-45e2-8f0b-c940c23a5606-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt\" (UID: \"a4975fd1-65d7-45e2-8f0b-c940c23a5606\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"
Jul 16 23:24:12.927436 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:12.927417    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4975fd1-65d7-45e2-8f0b-c940c23a5606-maas-api-tls\") pod \"maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt\" (UID: \"a4975fd1-65d7-45e2-8f0b-c940c23a5606\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"
Jul 16 23:24:12.932913 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:12.932887    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gslq\" (UniqueName: \"kubernetes.io/projected/a4975fd1-65d7-45e2-8f0b-c940c23a5606-kube-api-access-8gslq\") pod \"maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt\" (UID: \"a4975fd1-65d7-45e2-8f0b-c940c23a5606\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"
Jul 16 23:24:13.007465 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:13.007375    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"
Jul 16 23:24:13.191698 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:13.191657    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"]
Jul 16 23:24:13.192665 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:24:13.192635    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4975fd1_65d7_45e2_8f0b_c940c23a5606.slice/crio-d2040c2d9d9dd5e23305e831e228d659b0c8ac416c9ed5c1a5a9b712ad4e9c67 WatchSource:0}: Error finding container d2040c2d9d9dd5e23305e831e228d659b0c8ac416c9ed5c1a5a9b712ad4e9c67: Status 404 returned error can't find the container with id d2040c2d9d9dd5e23305e831e228d659b0c8ac416c9ed5c1a5a9b712ad4e9c67
Jul 16 23:24:13.727364 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:13.727333    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt" event={"ID":"a4975fd1-65d7-45e2-8f0b-c940c23a5606","Type":"ContainerStarted","Data":"7ba1657590893e4bb7461b619ae8cf4006a6ed3d29dfcbece7f0b1c03838eb7a"}
Jul 16 23:24:13.727535 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:13.727384    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt" event={"ID":"a4975fd1-65d7-45e2-8f0b-c940c23a5606","Type":"ContainerStarted","Data":"d2040c2d9d9dd5e23305e831e228d659b0c8ac416c9ed5c1a5a9b712ad4e9c67"}
Jul 16 23:24:13.740003 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:13.739880    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt" podStartSLOduration=1.445420261 podStartE2EDuration="1.739857087s" totalImagesPullingTime="294.436826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:24:13.736885619 +0000 UTC m=+3064.485369690" watchObservedRunningTime="2026-07-16 23:24:13.739857087 +0000 UTC m=+3064.488341163"
Jul 16 23:24:14.727212 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:14.727179    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"
Jul 16 23:24:18.652347 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:18.652303    2565 patch_prober.go:28] interesting pod/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Jul 16 23:24:18.652842 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:18.652379    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g" podUID="eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Jul 16 23:24:18.748682 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:18.748646    2565 generic.go:356] "Generic (PLEG): container finished" podID="eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" containerID="38d85c9eb45021204d97762633ae9f8cd6d51ed7ca17395fd40816a9fba3edfb" exitCode=0
Jul 16 23:24:18.748866 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:18.748718    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g" event={"ID":"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b","Type":"ContainerDied","Data":"38d85c9eb45021204d97762633ae9f8cd6d51ed7ca17395fd40816a9fba3edfb"}
Jul 16 23:24:18.749139 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:18.749121    2565 scope.go:117] "RemoveContainer" containerID="38d85c9eb45021204d97762633ae9f8cd6d51ed7ca17395fd40816a9fba3edfb"
Jul 16 23:24:18.750220 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:18.750190    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:24:18.750323 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:18.750306    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:24:18.750363 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:18.750344    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:24:19.765212 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:19.765163    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g" event={"ID":"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b","Type":"ContainerStarted","Data":"2e7de02366795aa28c509ea0edecb73122f0ed201ae0c363c83c012ad149a47a"}
Jul 16 23:24:20.448514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:20.448051    2565 patch_prober.go:28] interesting pod/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Jul 16 23:24:20.448514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:20.448120    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g" podUID="eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Jul 16 23:24:20.448514 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:20.448436    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:20.746145 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:20.746045    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"
Jul 16 23:24:21.448829 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:21.448360    2565 patch_prober.go:28] interesting pod/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Jul 16 23:24:21.448829 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:21.448424    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g" podUID="eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Jul 16 23:24:22.448363 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:22.448300    2565 patch_prober.go:28] interesting pod/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Jul 16 23:24:22.448689 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:22.448398    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g" podUID="eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Jul 16 23:24:23.448020 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:23.447972    2565 patch_prober.go:28] interesting pod/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Jul 16 23:24:23.448590 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:23.448061    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g" podUID="eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Jul 16 23:24:24.452681 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:24.452639    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:24.453104 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:24.453021    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:24.453764 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:24.453748    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:24:27.716673 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:27.716632    2565 patch_prober.go:28] interesting pod/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 16 23:24:27.717119 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:27.716694    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx" podUID="a8b1b47a-d704-46ed-b3a3-ec9f324d4224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 16 23:24:27.816601 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:27.816565    2565 generic.go:356] "Generic (PLEG): container finished" podID="a8b1b47a-d704-46ed-b3a3-ec9f324d4224" containerID="546ed43b169b22a06668b9026c50592408775c5de9874c4ad61a954f7f196b94" exitCode=0
Jul 16 23:24:27.816794 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:27.816638    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx" event={"ID":"a8b1b47a-d704-46ed-b3a3-ec9f324d4224","Type":"ContainerDied","Data":"546ed43b169b22a06668b9026c50592408775c5de9874c4ad61a954f7f196b94"}
Jul 16 23:24:27.817082 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:27.817068    2565 scope.go:117] "RemoveContainer" containerID="546ed43b169b22a06668b9026c50592408775c5de9874c4ad61a954f7f196b94"
Jul 16 23:24:27.818109 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:27.818082    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:24:27.818166 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:27.818150    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:24:27.818205 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:27.818182    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 23:24:28.824981 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:28.824948    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx" event={"ID":"a8b1b47a-d704-46ed-b3a3-ec9f324d4224","Type":"ContainerStarted","Data":"487bd85812fc614334d6de3076babdfef00026b4ac7e0355910a15921365652e"}
Jul 16 23:24:28.982131 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:28.982087    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:28.982416 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:28.982389    2565 patch_prober.go:28] interesting pod/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 16 23:24:28.982468 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:28.982446    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx" podUID="a8b1b47a-d704-46ed-b3a3-ec9f324d4224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 16 23:24:29.982531 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:29.982486    2565 patch_prober.go:28] interesting pod/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 16 23:24:29.982954 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:29.982553    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx" podUID="a8b1b47a-d704-46ed-b3a3-ec9f324d4224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 16 23:24:30.982466 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:30.982424    2565 patch_prober.go:28] interesting pod/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 16 23:24:30.982710 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:30.982486    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx" podUID="a8b1b47a-d704-46ed-b3a3-ec9f324d4224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 16 23:24:31.982107 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:31.982067    2565 patch_prober.go:28] interesting pod/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 16 23:24:31.982327 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:31.982140    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx" podUID="a8b1b47a-d704-46ed-b3a3-ec9f324d4224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 16 23:24:32.981600 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:32.981556    2565 patch_prober.go:28] interesting pod/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 16 23:24:32.982023 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:32.981639    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx" podUID="a8b1b47a-d704-46ed-b3a3-ec9f324d4224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 16 23:24:33.986352 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:33.986318    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:33.986741 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:33.986718    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:24:33.987398 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:24:33.987377    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:25:41.066432 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:41.066343    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp"]
Jul 16 23:25:41.070066 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:41.070044    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp"
Jul 16 23:25:41.072176 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:41.072149    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:25:41.076852 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:41.076820    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp"]
Jul 16 23:25:41.097105 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:41.097072    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d0fdf4-aeff-42bd-a259-b85b486a15c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp\" (UID: \"d0d0fdf4-aeff-42bd-a259-b85b486a15c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp"
Jul 16 23:25:41.198693 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:41.198655    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d0fdf4-aeff-42bd-a259-b85b486a15c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp\" (UID: \"d0d0fdf4-aeff-42bd-a259-b85b486a15c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp"
Jul 16 23:25:41.199276 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:41.199235    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d0fdf4-aeff-42bd-a259-b85b486a15c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp\" (UID: \"d0d0fdf4-aeff-42bd-a259-b85b486a15c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp"
Jul 16 23:25:41.382999 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:41.382968    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp"
Jul 16 23:25:41.522898 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:41.522861    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp"]
Jul 16 23:25:41.525952 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:25:41.525919    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d0fdf4_aeff_42bd_a259_b85b486a15c3.slice/crio-070420b0f13e6f40d6932eb10bd1b9a694cf9da9cb321682505b0303541d988b WatchSource:0}: Error finding container 070420b0f13e6f40d6932eb10bd1b9a694cf9da9cb321682505b0303541d988b: Status 404 returned error can't find the container with id 070420b0f13e6f40d6932eb10bd1b9a694cf9da9cb321682505b0303541d988b
Jul 16 23:25:42.216112 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:42.216075    2565 generic.go:356] "Generic (PLEG): container finished" podID="d0d0fdf4-aeff-42bd-a259-b85b486a15c3" containerID="06a6c34c6121e622fc9c17590b219b100805bbdcf5e9b532a202bd09e2f6a8f9" exitCode=6
Jul 16 23:25:42.216516 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:42.216157    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp" event={"ID":"d0d0fdf4-aeff-42bd-a259-b85b486a15c3","Type":"ContainerDied","Data":"06a6c34c6121e622fc9c17590b219b100805bbdcf5e9b532a202bd09e2f6a8f9"}
Jul 16 23:25:42.216516 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:42.216212    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp" event={"ID":"d0d0fdf4-aeff-42bd-a259-b85b486a15c3","Type":"ContainerStarted","Data":"070420b0f13e6f40d6932eb10bd1b9a694cf9da9cb321682505b0303541d988b"}
Jul 16 23:25:42.216516 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:42.216433    2565 scope.go:117] "RemoveContainer" containerID="06a6c34c6121e622fc9c17590b219b100805bbdcf5e9b532a202bd09e2f6a8f9"
Jul 16 23:25:43.223048 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:43.222950    2565 generic.go:356] "Generic (PLEG): container finished" podID="d0d0fdf4-aeff-42bd-a259-b85b486a15c3" containerID="b9ce77f77a3b403c39f76e595997e8906148020cedd96660eff2691858f09393" exitCode=6
Jul 16 23:25:43.223048 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:43.223028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp" event={"ID":"d0d0fdf4-aeff-42bd-a259-b85b486a15c3","Type":"ContainerDied","Data":"b9ce77f77a3b403c39f76e595997e8906148020cedd96660eff2691858f09393"}
Jul 16 23:25:43.223580 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:43.223088    2565 scope.go:117] "RemoveContainer" containerID="06a6c34c6121e622fc9c17590b219b100805bbdcf5e9b532a202bd09e2f6a8f9"
Jul 16 23:25:43.223580 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:43.223363    2565 scope.go:117] "RemoveContainer" containerID="b9ce77f77a3b403c39f76e595997e8906148020cedd96660eff2691858f09393"
Jul 16 23:25:43.223680 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:25:43.223628    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp_odh-ai-gateway-infra(d0d0fdf4-aeff-42bd-a259-b85b486a15c3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp" podUID="d0d0fdf4-aeff-42bd-a259-b85b486a15c3"
Jul 16 23:25:44.233098 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:44.233069    2565 scope.go:117] "RemoveContainer" containerID="b9ce77f77a3b403c39f76e595997e8906148020cedd96660eff2691858f09393"
Jul 16 23:25:44.233557 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:25:44.233338    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp_odh-ai-gateway-infra(d0d0fdf4-aeff-42bd-a259-b85b486a15c3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp" podUID="d0d0fdf4-aeff-42bd-a259-b85b486a15c3"
Jul 16 23:25:53.346876 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:53.346830    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk"]
Jul 16 23:25:53.350570 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:53.350543    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk"
Jul 16 23:25:53.358587 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:53.358542    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk"]
Jul 16 23:25:53.414611 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:53.414561    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a8f384-c993-4932-b7dc-df51a768d353-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-s8qlk\" (UID: \"b1a8f384-c993-4932-b7dc-df51a768d353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk"
Jul 16 23:25:53.515923 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:53.515883    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a8f384-c993-4932-b7dc-df51a768d353-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-s8qlk\" (UID: \"b1a8f384-c993-4932-b7dc-df51a768d353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk"
Jul 16 23:25:53.516601 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:53.516576    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a8f384-c993-4932-b7dc-df51a768d353-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-s8qlk\" (UID: \"b1a8f384-c993-4932-b7dc-df51a768d353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk"
Jul 16 23:25:53.663882 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:53.663788    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk"
Jul 16 23:25:54.038412 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:54.038379    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk"]
Jul 16 23:25:54.040626 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:25:54.040601    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a8f384_c993_4932_b7dc_df51a768d353.slice/crio-beaaa372d30bbd9db12757047841f17c44ca65303f4f23c154cf93edcd5cb765 WatchSource:0}: Error finding container beaaa372d30bbd9db12757047841f17c44ca65303f4f23c154cf93edcd5cb765: Status 404 returned error can't find the container with id beaaa372d30bbd9db12757047841f17c44ca65303f4f23c154cf93edcd5cb765
Jul 16 23:25:54.288321 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:54.288206    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk" event={"ID":"b1a8f384-c993-4932-b7dc-df51a768d353","Type":"ContainerStarted","Data":"beaaa372d30bbd9db12757047841f17c44ca65303f4f23c154cf93edcd5cb765"}
Jul 16 23:25:55.296782 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:55.296741    2565 generic.go:356] "Generic (PLEG): container finished" podID="b1a8f384-c993-4932-b7dc-df51a768d353" containerID="3a1fd0830f348eac7899c9b35aa25e3d5315bb8526343e607d0ec7bb0d1e1022" exitCode=6
Jul 16 23:25:55.297324 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:55.296811    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk" event={"ID":"b1a8f384-c993-4932-b7dc-df51a768d353","Type":"ContainerDied","Data":"3a1fd0830f348eac7899c9b35aa25e3d5315bb8526343e607d0ec7bb0d1e1022"}
Jul 16 23:25:55.297324 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:55.297086    2565 scope.go:117] "RemoveContainer" containerID="3a1fd0830f348eac7899c9b35aa25e3d5315bb8526343e607d0ec7bb0d1e1022"
Jul 16 23:25:55.721291 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:55.721241    2565 scope.go:117] "RemoveContainer" containerID="b9ce77f77a3b403c39f76e595997e8906148020cedd96660eff2691858f09393"
Jul 16 23:25:56.304189 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:56.304091    2565 generic.go:356] "Generic (PLEG): container finished" podID="b1a8f384-c993-4932-b7dc-df51a768d353" containerID="6c50035bb821f1de710e7075912b56987c8c89ea88d33fa9a13d5c1c732bb91e" exitCode=6
Jul 16 23:25:56.304718 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:56.304191    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk" event={"ID":"b1a8f384-c993-4932-b7dc-df51a768d353","Type":"ContainerDied","Data":"6c50035bb821f1de710e7075912b56987c8c89ea88d33fa9a13d5c1c732bb91e"}
Jul 16 23:25:56.304718 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:56.304278    2565 scope.go:117] "RemoveContainer" containerID="3a1fd0830f348eac7899c9b35aa25e3d5315bb8526343e607d0ec7bb0d1e1022"
Jul 16 23:25:56.304718 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:56.304401    2565 scope.go:117] "RemoveContainer" containerID="6c50035bb821f1de710e7075912b56987c8c89ea88d33fa9a13d5c1c732bb91e"
Jul 16 23:25:56.304718 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:25:56.304675    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-s8qlk_odh-ai-gateway-infra(b1a8f384-c993-4932-b7dc-df51a768d353)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk" podUID="b1a8f384-c993-4932-b7dc-df51a768d353"
Jul 16 23:25:56.306339 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:56.306316    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp" event={"ID":"d0d0fdf4-aeff-42bd-a259-b85b486a15c3","Type":"ContainerStarted","Data":"938bf83bb94ab3e4ff6efda3ce5c9236d509abfdf8f23cd2a165ae295aaecb4e"}
Jul 16 23:25:56.335431 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:56.335366    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp" podStartSLOduration=15.335346727 podStartE2EDuration="15.335346727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:25:56.334766319 +0000 UTC m=+3167.083250388" watchObservedRunningTime="2026-07-16 23:25:56.335346727 +0000 UTC m=+3167.083830796"
Jul 16 23:25:57.313698 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:57.313660    2565 generic.go:356] "Generic (PLEG): container finished" podID="d0d0fdf4-aeff-42bd-a259-b85b486a15c3" containerID="938bf83bb94ab3e4ff6efda3ce5c9236d509abfdf8f23cd2a165ae295aaecb4e" exitCode=6
Jul 16 23:25:57.314340 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:57.313733    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp" event={"ID":"d0d0fdf4-aeff-42bd-a259-b85b486a15c3","Type":"ContainerDied","Data":"938bf83bb94ab3e4ff6efda3ce5c9236d509abfdf8f23cd2a165ae295aaecb4e"}
Jul 16 23:25:57.314340 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:57.313803    2565 scope.go:117] "RemoveContainer" containerID="b9ce77f77a3b403c39f76e595997e8906148020cedd96660eff2691858f09393"
Jul 16 23:25:57.314340 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:57.314138    2565 scope.go:117] "RemoveContainer" containerID="938bf83bb94ab3e4ff6efda3ce5c9236d509abfdf8f23cd2a165ae295aaecb4e"
Jul 16 23:25:57.314525 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:25:57.314500    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp_odh-ai-gateway-infra(d0d0fdf4-aeff-42bd-a259-b85b486a15c3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp" podUID="d0d0fdf4-aeff-42bd-a259-b85b486a15c3"
Jul 16 23:25:57.315760 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:57.315743    2565 scope.go:117] "RemoveContainer" containerID="6c50035bb821f1de710e7075912b56987c8c89ea88d33fa9a13d5c1c732bb91e"
Jul 16 23:25:57.316030 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:25:57.316009    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-s8qlk_odh-ai-gateway-infra(b1a8f384-c993-4932-b7dc-df51a768d353)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk" podUID="b1a8f384-c993-4932-b7dc-df51a768d353"
Jul 16 23:25:57.353656 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:57.353617    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp"]
Jul 16 23:25:58.317592 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:58.317558    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4"]
Jul 16 23:25:58.322994 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:58.322554    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4"
Jul 16 23:25:58.328712 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:58.328681    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4"]
Jul 16 23:25:58.467908 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:58.467875    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe27043-2333-4987-9333-c26b608be8ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4\" (UID: \"abe27043-2333-4987-9333-c26b608be8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4"
Jul 16 23:25:58.471668 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:58.471644    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp"
Jul 16 23:25:58.568738 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:58.568626    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d0fdf4-aeff-42bd-a259-b85b486a15c3-maas-api-service-ca\") pod \"d0d0fdf4-aeff-42bd-a259-b85b486a15c3\" (UID: \"d0d0fdf4-aeff-42bd-a259-b85b486a15c3\") "
Jul 16 23:25:58.568928 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:58.568884    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe27043-2333-4987-9333-c26b608be8ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4\" (UID: \"abe27043-2333-4987-9333-c26b608be8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4"
Jul 16 23:25:58.569003 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:58.568971    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d0fdf4-aeff-42bd-a259-b85b486a15c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0d0fdf4-aeff-42bd-a259-b85b486a15c3" (UID: "d0d0fdf4-aeff-42bd-a259-b85b486a15c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:25:58.569512 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:58.569493    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe27043-2333-4987-9333-c26b608be8ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4\" (UID: \"abe27043-2333-4987-9333-c26b608be8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4"
Jul 16 23:25:58.637583 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:58.637542    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4"
Jul 16 23:25:58.670412 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:58.670365    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d0fdf4-aeff-42bd-a259-b85b486a15c3-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:25:58.783576 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:58.783548    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4"]
Jul 16 23:25:58.785674 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:25:58.785641    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe27043_2333_4987_9333_c26b608be8ff.slice/crio-783742674ab0bde8bf6a6d3afea64338c36826ffbc2cc00a7981abe2349f2f2b WatchSource:0}: Error finding container 783742674ab0bde8bf6a6d3afea64338c36826ffbc2cc00a7981abe2349f2f2b: Status 404 returned error can't find the container with id 783742674ab0bde8bf6a6d3afea64338c36826ffbc2cc00a7981abe2349f2f2b
Jul 16 23:25:59.329243 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:59.329217    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp"
Jul 16 23:25:59.329741 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:59.329272    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp" event={"ID":"d0d0fdf4-aeff-42bd-a259-b85b486a15c3","Type":"ContainerDied","Data":"070420b0f13e6f40d6932eb10bd1b9a694cf9da9cb321682505b0303541d988b"}
Jul 16 23:25:59.329741 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:59.329340    2565 scope.go:117] "RemoveContainer" containerID="938bf83bb94ab3e4ff6efda3ce5c9236d509abfdf8f23cd2a165ae295aaecb4e"
Jul 16 23:25:59.331455 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:59.331424    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4" event={"ID":"abe27043-2333-4987-9333-c26b608be8ff","Type":"ContainerStarted","Data":"17dce23e31b5aba8367c6fdbc6eed5b32002a9706c08a699d94cdf3fb3129a3e"}
Jul 16 23:25:59.331596 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:59.331483    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4" event={"ID":"abe27043-2333-4987-9333-c26b608be8ff","Type":"ContainerStarted","Data":"783742674ab0bde8bf6a6d3afea64338c36826ffbc2cc00a7981abe2349f2f2b"}
Jul 16 23:25:59.349841 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:59.349790    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4" podStartSLOduration=1.349771062 podStartE2EDuration="1.349771062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:25:59.347455507 +0000 UTC m=+3170.095939580" watchObservedRunningTime="2026-07-16 23:25:59.349771062 +0000 UTC m=+3170.098255133"
Jul 16 23:25:59.359875 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:59.359838    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp"]
Jul 16 23:25:59.363168 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:59.363133    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8w2rp"]
Jul 16 23:25:59.732353 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:25:59.732238    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0d0fdf4-aeff-42bd-a259-b85b486a15c3" path="/var/lib/kubelet/pods/d0d0fdf4-aeff-42bd-a259-b85b486a15c3/volumes"
Jul 16 23:26:00.340260 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:00.340203    2565 generic.go:356] "Generic (PLEG): container finished" podID="abe27043-2333-4987-9333-c26b608be8ff" containerID="17dce23e31b5aba8367c6fdbc6eed5b32002a9706c08a699d94cdf3fb3129a3e" exitCode=6
Jul 16 23:26:00.340819 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:00.340323    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4" event={"ID":"abe27043-2333-4987-9333-c26b608be8ff","Type":"ContainerDied","Data":"17dce23e31b5aba8367c6fdbc6eed5b32002a9706c08a699d94cdf3fb3129a3e"}
Jul 16 23:26:00.340819 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:00.340597    2565 scope.go:117] "RemoveContainer" containerID="17dce23e31b5aba8367c6fdbc6eed5b32002a9706c08a699d94cdf3fb3129a3e"
Jul 16 23:26:01.350431 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:01.350396    2565 generic.go:356] "Generic (PLEG): container finished" podID="abe27043-2333-4987-9333-c26b608be8ff" containerID="703de3bb51148dee562c2eea443c313aa63b77208f30cd00e19451283df09a55" exitCode=6
Jul 16 23:26:01.350901 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:01.350473    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4" event={"ID":"abe27043-2333-4987-9333-c26b608be8ff","Type":"ContainerDied","Data":"703de3bb51148dee562c2eea443c313aa63b77208f30cd00e19451283df09a55"}
Jul 16 23:26:01.350901 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:01.350533    2565 scope.go:117] "RemoveContainer" containerID="17dce23e31b5aba8367c6fdbc6eed5b32002a9706c08a699d94cdf3fb3129a3e"
Jul 16 23:26:01.350901 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:01.350692    2565 scope.go:117] "RemoveContainer" containerID="703de3bb51148dee562c2eea443c313aa63b77208f30cd00e19451283df09a55"
Jul 16 23:26:01.351037 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:26:01.350942    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4_odh-ai-gateway-infra(abe27043-2333-4987-9333-c26b608be8ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4" podUID="abe27043-2333-4987-9333-c26b608be8ff"
Jul 16 23:26:02.358308 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:02.358261    2565 scope.go:117] "RemoveContainer" containerID="703de3bb51148dee562c2eea443c313aa63b77208f30cd00e19451283df09a55"
Jul 16 23:26:02.358730 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:26:02.358490    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4_odh-ai-gateway-infra(abe27043-2333-4987-9333-c26b608be8ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4" podUID="abe27043-2333-4987-9333-c26b608be8ff"
Jul 16 23:26:11.721450 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:11.721413    2565 scope.go:117] "RemoveContainer" containerID="6c50035bb821f1de710e7075912b56987c8c89ea88d33fa9a13d5c1c732bb91e"
Jul 16 23:26:12.414740 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:12.414701    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk" event={"ID":"b1a8f384-c993-4932-b7dc-df51a768d353","Type":"ContainerStarted","Data":"f84ae596f699419581555198d37d93377b6322b93fb34658b19f639432ba2d56"}
Jul 16 23:26:12.430307 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:12.430242    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk" podStartSLOduration=19.430227176 podStartE2EDuration="19.430227176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:26:12.428884397 +0000 UTC m=+3183.177368467" watchObservedRunningTime="2026-07-16 23:26:12.430227176 +0000 UTC m=+3183.178711246"
Jul 16 23:26:13.421807 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:13.421770    2565 generic.go:356] "Generic (PLEG): container finished" podID="b1a8f384-c993-4932-b7dc-df51a768d353" containerID="f84ae596f699419581555198d37d93377b6322b93fb34658b19f639432ba2d56" exitCode=6
Jul 16 23:26:13.422316 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:13.421840    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk" event={"ID":"b1a8f384-c993-4932-b7dc-df51a768d353","Type":"ContainerDied","Data":"f84ae596f699419581555198d37d93377b6322b93fb34658b19f639432ba2d56"}
Jul 16 23:26:13.422316 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:13.421899    2565 scope.go:117] "RemoveContainer" containerID="6c50035bb821f1de710e7075912b56987c8c89ea88d33fa9a13d5c1c732bb91e"
Jul 16 23:26:13.422316 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:13.422158    2565 scope.go:117] "RemoveContainer" containerID="f84ae596f699419581555198d37d93377b6322b93fb34658b19f639432ba2d56"
Jul 16 23:26:13.422471 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:26:13.422450    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-s8qlk_odh-ai-gateway-infra(b1a8f384-c993-4932-b7dc-df51a768d353)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk" podUID="b1a8f384-c993-4932-b7dc-df51a768d353"
Jul 16 23:26:13.445372 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:13.445336    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk"]
Jul 16 23:26:14.570524 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:14.570489    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk"
Jul 16 23:26:14.628699 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:14.628649    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a8f384-c993-4932-b7dc-df51a768d353-maas-api-service-ca\") pod \"b1a8f384-c993-4932-b7dc-df51a768d353\" (UID: \"b1a8f384-c993-4932-b7dc-df51a768d353\") "
Jul 16 23:26:14.629019 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:14.628994    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a8f384-c993-4932-b7dc-df51a768d353-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b1a8f384-c993-4932-b7dc-df51a768d353" (UID: "b1a8f384-c993-4932-b7dc-df51a768d353"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:26:14.720880 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:14.720788    2565 scope.go:117] "RemoveContainer" containerID="703de3bb51148dee562c2eea443c313aa63b77208f30cd00e19451283df09a55"
Jul 16 23:26:14.730018 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:14.729986    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a8f384-c993-4932-b7dc-df51a768d353-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:26:15.438207 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:15.438167    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4" event={"ID":"abe27043-2333-4987-9333-c26b608be8ff","Type":"ContainerStarted","Data":"58f4c3dc8b65c28b133c8e9c9e370e5b3f1cdd0449f73e74c7e2c40fd6bd02ad"}
Jul 16 23:26:15.440154 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:15.439790    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk" event={"ID":"b1a8f384-c993-4932-b7dc-df51a768d353","Type":"ContainerDied","Data":"beaaa372d30bbd9db12757047841f17c44ca65303f4f23c154cf93edcd5cb765"}
Jul 16 23:26:15.440154 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:15.439806    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk"
Jul 16 23:26:15.440154 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:15.439846    2565 scope.go:117] "RemoveContainer" containerID="f84ae596f699419581555198d37d93377b6322b93fb34658b19f639432ba2d56"
Jul 16 23:26:15.500817 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:15.500782    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk"]
Jul 16 23:26:15.504723 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:15.504690    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-s8qlk"]
Jul 16 23:26:15.727442 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:15.727342    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1a8f384-c993-4932-b7dc-df51a768d353" path="/var/lib/kubelet/pods/b1a8f384-c993-4932-b7dc-df51a768d353/volumes"
Jul 16 23:26:16.446710 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:16.446675    2565 generic.go:356] "Generic (PLEG): container finished" podID="abe27043-2333-4987-9333-c26b608be8ff" containerID="58f4c3dc8b65c28b133c8e9c9e370e5b3f1cdd0449f73e74c7e2c40fd6bd02ad" exitCode=6
Jul 16 23:26:16.446880 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:16.446749    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4" event={"ID":"abe27043-2333-4987-9333-c26b608be8ff","Type":"ContainerDied","Data":"58f4c3dc8b65c28b133c8e9c9e370e5b3f1cdd0449f73e74c7e2c40fd6bd02ad"}
Jul 16 23:26:16.446880 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:16.446812    2565 scope.go:117] "RemoveContainer" containerID="703de3bb51148dee562c2eea443c313aa63b77208f30cd00e19451283df09a55"
Jul 16 23:26:16.447168 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:16.447143    2565 scope.go:117] "RemoveContainer" containerID="58f4c3dc8b65c28b133c8e9c9e370e5b3f1cdd0449f73e74c7e2c40fd6bd02ad"
Jul 16 23:26:16.447393 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:26:16.447366    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4_odh-ai-gateway-infra(abe27043-2333-4987-9333-c26b608be8ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4" podUID="abe27043-2333-4987-9333-c26b608be8ff"
Jul 16 23:26:16.463362 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:16.463335    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4"]
Jul 16 23:26:17.602500 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:17.602470    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4"
Jul 16 23:26:17.663442 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:17.663407    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe27043-2333-4987-9333-c26b608be8ff-maas-api-service-ca\") pod \"abe27043-2333-4987-9333-c26b608be8ff\" (UID: \"abe27043-2333-4987-9333-c26b608be8ff\") "
Jul 16 23:26:17.663766 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:17.663740    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe27043-2333-4987-9333-c26b608be8ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "abe27043-2333-4987-9333-c26b608be8ff" (UID: "abe27043-2333-4987-9333-c26b608be8ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:26:17.764558 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:17.764514    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abe27043-2333-4987-9333-c26b608be8ff-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:26:18.461587 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:18.461556    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4"
Jul 16 23:26:18.461587 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:18.461573    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4" event={"ID":"abe27043-2333-4987-9333-c26b608be8ff","Type":"ContainerDied","Data":"783742674ab0bde8bf6a6d3afea64338c36826ffbc2cc00a7981abe2349f2f2b"}
Jul 16 23:26:18.461847 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:18.461637    2565 scope.go:117] "RemoveContainer" containerID="58f4c3dc8b65c28b133c8e9c9e370e5b3f1cdd0449f73e74c7e2c40fd6bd02ad"
Jul 16 23:26:18.478395 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:18.478359    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4"]
Jul 16 23:26:18.484119 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:18.484076    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-9hzg4"]
Jul 16 23:26:19.726945 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:19.726895    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abe27043-2333-4987-9333-c26b608be8ff" path="/var/lib/kubelet/pods/abe27043-2333-4987-9333-c26b608be8ff/volumes"
Jul 16 23:26:31.112049 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112013    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h"]
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112442    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe27043-2333-4987-9333-c26b608be8ff" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112455    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe27043-2333-4987-9333-c26b608be8ff" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112464    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe27043-2333-4987-9333-c26b608be8ff" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112469    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe27043-2333-4987-9333-c26b608be8ff" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112479    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a8f384-c993-4932-b7dc-df51a768d353" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112485    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a8f384-c993-4932-b7dc-df51a768d353" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112498    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d0fdf4-aeff-42bd-a259-b85b486a15c3" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112504    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d0fdf4-aeff-42bd-a259-b85b486a15c3" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112511    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d0fdf4-aeff-42bd-a259-b85b486a15c3" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112516    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d0fdf4-aeff-42bd-a259-b85b486a15c3" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112522    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a8f384-c993-4932-b7dc-df51a768d353" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112528    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a8f384-c993-4932-b7dc-df51a768d353" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112537    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d0fdf4-aeff-42bd-a259-b85b486a15c3" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112542    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d0fdf4-aeff-42bd-a259-b85b486a15c3" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112552    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a8f384-c993-4932-b7dc-df51a768d353" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112557    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a8f384-c993-4932-b7dc-df51a768d353" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112612    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1a8f384-c993-4932-b7dc-df51a768d353" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112618    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe27043-2333-4987-9333-c26b608be8ff" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112625    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe27043-2333-4987-9333-c26b608be8ff" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112633    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1a8f384-c993-4932-b7dc-df51a768d353" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112641    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d0fdf4-aeff-42bd-a259-b85b486a15c3" containerName="revoke-keys"
Jul 16 23:26:31.112636 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.112647    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d0fdf4-aeff-42bd-a259-b85b486a15c3" containerName="revoke-keys"
Jul 16 23:26:31.115739 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.115722    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h"
Jul 16 23:26:31.117807 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.117785    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:26:31.123373 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.123225    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h"]
Jul 16 23:26:31.199589 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.199539    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/871c1c15-4d6e-4846-8a92-9838e9930102-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h\" (UID: \"871c1c15-4d6e-4846-8a92-9838e9930102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h"
Jul 16 23:26:31.301196 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.301151    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/871c1c15-4d6e-4846-8a92-9838e9930102-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h\" (UID: \"871c1c15-4d6e-4846-8a92-9838e9930102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h"
Jul 16 23:26:31.301831 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.301806    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/871c1c15-4d6e-4846-8a92-9838e9930102-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h\" (UID: \"871c1c15-4d6e-4846-8a92-9838e9930102\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h"
Jul 16 23:26:31.428456 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.428355    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h"
Jul 16 23:26:31.566991 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:31.566965    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h"]
Jul 16 23:26:31.569778 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:26:31.569748    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod871c1c15_4d6e_4846_8a92_9838e9930102.slice/crio-48b1053e2f366b0680beb883a49b69fe93c3e0a9171fc8b00458e4add653e295 WatchSource:0}: Error finding container 48b1053e2f366b0680beb883a49b69fe93c3e0a9171fc8b00458e4add653e295: Status 404 returned error can't find the container with id 48b1053e2f366b0680beb883a49b69fe93c3e0a9171fc8b00458e4add653e295
Jul 16 23:26:32.545126 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:32.545031    2565 generic.go:356] "Generic (PLEG): container finished" podID="871c1c15-4d6e-4846-8a92-9838e9930102" containerID="84ea86d17f658ca8fc26f3a45c54b5884e822c5bd9a6ea3e81a7fb3fced310af" exitCode=6
Jul 16 23:26:32.545126 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:32.545074    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h" event={"ID":"871c1c15-4d6e-4846-8a92-9838e9930102","Type":"ContainerDied","Data":"84ea86d17f658ca8fc26f3a45c54b5884e822c5bd9a6ea3e81a7fb3fced310af"}
Jul 16 23:26:32.545563 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:32.545141    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h" event={"ID":"871c1c15-4d6e-4846-8a92-9838e9930102","Type":"ContainerStarted","Data":"48b1053e2f366b0680beb883a49b69fe93c3e0a9171fc8b00458e4add653e295"}
Jul 16 23:26:32.545563 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:32.545342    2565 scope.go:117] "RemoveContainer" containerID="84ea86d17f658ca8fc26f3a45c54b5884e822c5bd9a6ea3e81a7fb3fced310af"
Jul 16 23:26:33.552470 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:33.552366    2565 generic.go:356] "Generic (PLEG): container finished" podID="871c1c15-4d6e-4846-8a92-9838e9930102" containerID="33b064c6e498d4343046d7a6b789ea5e81238b41e56b3d9e721f1b9036740bc0" exitCode=6
Jul 16 23:26:33.552470 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:33.552451    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h" event={"ID":"871c1c15-4d6e-4846-8a92-9838e9930102","Type":"ContainerDied","Data":"33b064c6e498d4343046d7a6b789ea5e81238b41e56b3d9e721f1b9036740bc0"}
Jul 16 23:26:33.552995 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:33.552515    2565 scope.go:117] "RemoveContainer" containerID="84ea86d17f658ca8fc26f3a45c54b5884e822c5bd9a6ea3e81a7fb3fced310af"
Jul 16 23:26:33.552995 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:33.552667    2565 scope.go:117] "RemoveContainer" containerID="33b064c6e498d4343046d7a6b789ea5e81238b41e56b3d9e721f1b9036740bc0"
Jul 16 23:26:33.552995 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:26:33.552923    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h_odh-ai-gateway-infra(871c1c15-4d6e-4846-8a92-9838e9930102)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h" podUID="871c1c15-4d6e-4846-8a92-9838e9930102"
Jul 16 23:26:34.560051 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:34.560024    2565 scope.go:117] "RemoveContainer" containerID="33b064c6e498d4343046d7a6b789ea5e81238b41e56b3d9e721f1b9036740bc0"
Jul 16 23:26:34.560539 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:26:34.560318    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h_odh-ai-gateway-infra(871c1c15-4d6e-4846-8a92-9838e9930102)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h" podUID="871c1c15-4d6e-4846-8a92-9838e9930102"
Jul 16 23:26:45.721167 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:45.721133    2565 scope.go:117] "RemoveContainer" containerID="33b064c6e498d4343046d7a6b789ea5e81238b41e56b3d9e721f1b9036740bc0"
Jul 16 23:26:46.628971 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:46.628923    2565 generic.go:356] "Generic (PLEG): container finished" podID="871c1c15-4d6e-4846-8a92-9838e9930102" containerID="1d6528eb1bd72ee9480313672b0d70b8c278393cb89d0b395146a2ce47b391b2" exitCode=6
Jul 16 23:26:46.628971 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:46.628971    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h" event={"ID":"871c1c15-4d6e-4846-8a92-9838e9930102","Type":"ContainerDied","Data":"1d6528eb1bd72ee9480313672b0d70b8c278393cb89d0b395146a2ce47b391b2"}
Jul 16 23:26:46.629212 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:46.629034    2565 scope.go:117] "RemoveContainer" containerID="33b064c6e498d4343046d7a6b789ea5e81238b41e56b3d9e721f1b9036740bc0"
Jul 16 23:26:46.629434 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:46.629410    2565 scope.go:117] "RemoveContainer" containerID="1d6528eb1bd72ee9480313672b0d70b8c278393cb89d0b395146a2ce47b391b2"
Jul 16 23:26:46.629782 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:26:46.629754    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h_odh-ai-gateway-infra(871c1c15-4d6e-4846-8a92-9838e9930102)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h" podUID="871c1c15-4d6e-4846-8a92-9838e9930102"
Jul 16 23:26:47.665683 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:47.665644    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h"]
Jul 16 23:26:47.806832 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:47.806801    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h"
Jul 16 23:26:47.966164 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:47.966032    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/871c1c15-4d6e-4846-8a92-9838e9930102-maas-api-service-ca\") pod \"871c1c15-4d6e-4846-8a92-9838e9930102\" (UID: \"871c1c15-4d6e-4846-8a92-9838e9930102\") "
Jul 16 23:26:47.966537 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:47.966512    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/871c1c15-4d6e-4846-8a92-9838e9930102-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "871c1c15-4d6e-4846-8a92-9838e9930102" (UID: "871c1c15-4d6e-4846-8a92-9838e9930102"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:26:48.067866 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:48.067825    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/871c1c15-4d6e-4846-8a92-9838e9930102-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:26:48.641738 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:48.641708    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h"
Jul 16 23:26:48.641924 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:48.641744    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h" event={"ID":"871c1c15-4d6e-4846-8a92-9838e9930102","Type":"ContainerDied","Data":"48b1053e2f366b0680beb883a49b69fe93c3e0a9171fc8b00458e4add653e295"}
Jul 16 23:26:48.641924 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:48.641813    2565 scope.go:117] "RemoveContainer" containerID="1d6528eb1bd72ee9480313672b0d70b8c278393cb89d0b395146a2ce47b391b2"
Jul 16 23:26:48.664227 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:48.664190    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h"]
Jul 16 23:26:48.669804 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:48.669771    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-hkw4h"]
Jul 16 23:26:49.726595 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:49.726557    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="871c1c15-4d6e-4846-8a92-9838e9930102" path="/var/lib/kubelet/pods/871c1c15-4d6e-4846-8a92-9838e9930102/volumes"
Jul 16 23:26:53.390831 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.390796    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t"]
Jul 16 23:26:53.391215 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.391177    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="871c1c15-4d6e-4846-8a92-9838e9930102" containerName="revoke-keys"
Jul 16 23:26:53.391215 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.391187    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="871c1c15-4d6e-4846-8a92-9838e9930102" containerName="revoke-keys"
Jul 16 23:26:53.391215 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.391215    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="871c1c15-4d6e-4846-8a92-9838e9930102" containerName="revoke-keys"
Jul 16 23:26:53.391363 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.391221    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="871c1c15-4d6e-4846-8a92-9838e9930102" containerName="revoke-keys"
Jul 16 23:26:53.391363 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.391228    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe27043-2333-4987-9333-c26b608be8ff" containerName="revoke-keys"
Jul 16 23:26:53.391363 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.391234    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe27043-2333-4987-9333-c26b608be8ff" containerName="revoke-keys"
Jul 16 23:26:53.391363 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.391306    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="871c1c15-4d6e-4846-8a92-9838e9930102" containerName="revoke-keys"
Jul 16 23:26:53.391363 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.391314    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe27043-2333-4987-9333-c26b608be8ff" containerName="revoke-keys"
Jul 16 23:26:53.391363 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.391322    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="871c1c15-4d6e-4846-8a92-9838e9930102" containerName="revoke-keys"
Jul 16 23:26:53.391363 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.391329    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1a8f384-c993-4932-b7dc-df51a768d353" containerName="revoke-keys"
Jul 16 23:26:53.391363 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.391336    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d0fdf4-aeff-42bd-a259-b85b486a15c3" containerName="revoke-keys"
Jul 16 23:26:53.391363 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.391341    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="871c1c15-4d6e-4846-8a92-9838e9930102" containerName="revoke-keys"
Jul 16 23:26:53.394363 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.394346    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t"
Jul 16 23:26:53.396704 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.396686    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:26:53.400995 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.400929    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t"]
Jul 16 23:26:53.418320 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.418288    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62fce735-14f0-4b16-b0f5-9abf09c04065-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-vv58t\" (UID: \"62fce735-14f0-4b16-b0f5-9abf09c04065\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t"
Jul 16 23:26:53.519646 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.519592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62fce735-14f0-4b16-b0f5-9abf09c04065-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-vv58t\" (UID: \"62fce735-14f0-4b16-b0f5-9abf09c04065\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t"
Jul 16 23:26:53.520242 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.520219    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62fce735-14f0-4b16-b0f5-9abf09c04065-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-vv58t\" (UID: \"62fce735-14f0-4b16-b0f5-9abf09c04065\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t"
Jul 16 23:26:53.707709 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.707618    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t"
Jul 16 23:26:53.844286 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:53.844242    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t"]
Jul 16 23:26:53.846243 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:26:53.846207    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fce735_14f0_4b16_b0f5_9abf09c04065.slice/crio-7e13e2f39249fc0f2d2777d8d874ad51e2fcd858afc9b1b88cf9268cf0153b6b WatchSource:0}: Error finding container 7e13e2f39249fc0f2d2777d8d874ad51e2fcd858afc9b1b88cf9268cf0153b6b: Status 404 returned error can't find the container with id 7e13e2f39249fc0f2d2777d8d874ad51e2fcd858afc9b1b88cf9268cf0153b6b
Jul 16 23:26:54.675591 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:54.675547    2565 generic.go:356] "Generic (PLEG): container finished" podID="62fce735-14f0-4b16-b0f5-9abf09c04065" containerID="fe879b9f5ad9dcf9bacf71c53242b29ad011e4c329f0de73c9e0e1e97fb8f21e" exitCode=6
Jul 16 23:26:54.675969 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:54.675607    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t" event={"ID":"62fce735-14f0-4b16-b0f5-9abf09c04065","Type":"ContainerDied","Data":"fe879b9f5ad9dcf9bacf71c53242b29ad011e4c329f0de73c9e0e1e97fb8f21e"}
Jul 16 23:26:54.675969 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:54.675653    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t" event={"ID":"62fce735-14f0-4b16-b0f5-9abf09c04065","Type":"ContainerStarted","Data":"7e13e2f39249fc0f2d2777d8d874ad51e2fcd858afc9b1b88cf9268cf0153b6b"}
Jul 16 23:26:54.675969 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:54.675895    2565 scope.go:117] "RemoveContainer" containerID="fe879b9f5ad9dcf9bacf71c53242b29ad011e4c329f0de73c9e0e1e97fb8f21e"
Jul 16 23:26:55.684237 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:55.684205    2565 generic.go:356] "Generic (PLEG): container finished" podID="62fce735-14f0-4b16-b0f5-9abf09c04065" containerID="c575172a8aff916eb0fa654ebc684e41aeab23b6845e48c3e3cdb6199190502f" exitCode=6
Jul 16 23:26:55.684748 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:55.684297    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t" event={"ID":"62fce735-14f0-4b16-b0f5-9abf09c04065","Type":"ContainerDied","Data":"c575172a8aff916eb0fa654ebc684e41aeab23b6845e48c3e3cdb6199190502f"}
Jul 16 23:26:55.684748 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:55.684378    2565 scope.go:117] "RemoveContainer" containerID="fe879b9f5ad9dcf9bacf71c53242b29ad011e4c329f0de73c9e0e1e97fb8f21e"
Jul 16 23:26:55.684748 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:55.684645    2565 scope.go:117] "RemoveContainer" containerID="c575172a8aff916eb0fa654ebc684e41aeab23b6845e48c3e3cdb6199190502f"
Jul 16 23:26:55.684946 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:26:55.684920    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-vv58t_odh-ai-gateway-infra(62fce735-14f0-4b16-b0f5-9abf09c04065)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t" podUID="62fce735-14f0-4b16-b0f5-9abf09c04065"
Jul 16 23:26:56.691430 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:26:56.691395    2565 scope.go:117] "RemoveContainer" containerID="c575172a8aff916eb0fa654ebc684e41aeab23b6845e48c3e3cdb6199190502f"
Jul 16 23:26:56.691812 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:26:56.691615    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-vv58t_odh-ai-gateway-infra(62fce735-14f0-4b16-b0f5-9abf09c04065)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t" podUID="62fce735-14f0-4b16-b0f5-9abf09c04065"
Jul 16 23:27:10.192539 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:27:10.192460    2565 scope.go:117] "RemoveContainer" containerID="978c08c2676007b2a520751ff9d40cc8af2f568ded1c1bbd938802050f56bb1d"
Jul 16 23:27:10.720489 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:27:10.720454    2565 scope.go:117] "RemoveContainer" containerID="c575172a8aff916eb0fa654ebc684e41aeab23b6845e48c3e3cdb6199190502f"
Jul 16 23:27:11.775030 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:27:11.774939    2565 generic.go:356] "Generic (PLEG): container finished" podID="62fce735-14f0-4b16-b0f5-9abf09c04065" containerID="4cd7f8a7d3b9426abb9b49a17bb60eec6ab3db12c08d915ce9715047bb5df6b2" exitCode=6
Jul 16 23:27:11.775030 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:27:11.775012    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t" event={"ID":"62fce735-14f0-4b16-b0f5-9abf09c04065","Type":"ContainerDied","Data":"4cd7f8a7d3b9426abb9b49a17bb60eec6ab3db12c08d915ce9715047bb5df6b2"}
Jul 16 23:27:11.775567 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:27:11.775088    2565 scope.go:117] "RemoveContainer" containerID="c575172a8aff916eb0fa654ebc684e41aeab23b6845e48c3e3cdb6199190502f"
Jul 16 23:27:11.775567 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:27:11.775445    2565 scope.go:117] "RemoveContainer" containerID="4cd7f8a7d3b9426abb9b49a17bb60eec6ab3db12c08d915ce9715047bb5df6b2"
Jul 16 23:27:11.775710 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:27:11.775686    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-vv58t_odh-ai-gateway-infra(62fce735-14f0-4b16-b0f5-9abf09c04065)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t" podUID="62fce735-14f0-4b16-b0f5-9abf09c04065"
Jul 16 23:27:12.801561 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:27:12.801527    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t"]
Jul 16 23:27:12.944960 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:27:12.944931    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t"
Jul 16 23:27:13.016831 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:27:13.016801    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62fce735-14f0-4b16-b0f5-9abf09c04065-maas-api-service-ca\") pod \"62fce735-14f0-4b16-b0f5-9abf09c04065\" (UID: \"62fce735-14f0-4b16-b0f5-9abf09c04065\") "
Jul 16 23:27:13.017131 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:27:13.017107    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62fce735-14f0-4b16-b0f5-9abf09c04065-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62fce735-14f0-4b16-b0f5-9abf09c04065" (UID: "62fce735-14f0-4b16-b0f5-9abf09c04065"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:27:13.118584 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:27:13.118544    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62fce735-14f0-4b16-b0f5-9abf09c04065-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:27:13.788094 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:27:13.788065    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t"
Jul 16 23:27:13.788290 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:27:13.788104    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t" event={"ID":"62fce735-14f0-4b16-b0f5-9abf09c04065","Type":"ContainerDied","Data":"7e13e2f39249fc0f2d2777d8d874ad51e2fcd858afc9b1b88cf9268cf0153b6b"}
Jul 16 23:27:13.788290 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:27:13.788180    2565 scope.go:117] "RemoveContainer" containerID="4cd7f8a7d3b9426abb9b49a17bb60eec6ab3db12c08d915ce9715047bb5df6b2"
Jul 16 23:27:13.803365 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:27:13.803337    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t"]
Jul 16 23:27:13.807450 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:27:13.807427    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-vv58t"]
Jul 16 23:27:15.727379 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:27:15.727341    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62fce735-14f0-4b16-b0f5-9abf09c04065" path="/var/lib/kubelet/pods/62fce735-14f0-4b16-b0f5-9abf09c04065/volumes"
Jul 16 23:28:21.187556 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:21.187510    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll"]
Jul 16 23:28:21.188153 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:21.188102    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="871c1c15-4d6e-4846-8a92-9838e9930102" containerName="revoke-keys"
Jul 16 23:28:21.188153 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:21.188124    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="871c1c15-4d6e-4846-8a92-9838e9930102" containerName="revoke-keys"
Jul 16 23:28:21.188153 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:21.188146    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62fce735-14f0-4b16-b0f5-9abf09c04065" containerName="revoke-keys"
Jul 16 23:28:21.188362 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:21.188155    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62fce735-14f0-4b16-b0f5-9abf09c04065" containerName="revoke-keys"
Jul 16 23:28:21.188362 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:21.188186    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62fce735-14f0-4b16-b0f5-9abf09c04065" containerName="revoke-keys"
Jul 16 23:28:21.188362 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:21.188196    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62fce735-14f0-4b16-b0f5-9abf09c04065" containerName="revoke-keys"
Jul 16 23:28:21.188362 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:21.188313    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62fce735-14f0-4b16-b0f5-9abf09c04065" containerName="revoke-keys"
Jul 16 23:28:21.188362 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:21.188331    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62fce735-14f0-4b16-b0f5-9abf09c04065" containerName="revoke-keys"
Jul 16 23:28:21.192558 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:21.192535    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll"
Jul 16 23:28:21.194958 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:21.194931    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:28:21.197573 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:21.197547    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll"]
Jul 16 23:28:21.288388 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:21.288347    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26ce678-61da-403b-9354-28814dc54463-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll\" (UID: \"e26ce678-61da-403b-9354-28814dc54463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll"
Jul 16 23:28:21.389677 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:21.389638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26ce678-61da-403b-9354-28814dc54463-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll\" (UID: \"e26ce678-61da-403b-9354-28814dc54463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll"
Jul 16 23:28:21.390310 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:21.390284    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26ce678-61da-403b-9354-28814dc54463-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll\" (UID: \"e26ce678-61da-403b-9354-28814dc54463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll"
Jul 16 23:28:21.505964 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:21.505857    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll"
Jul 16 23:28:21.647859 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:21.647652    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll"]
Jul 16 23:28:21.651068 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:28:21.651015    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26ce678_61da_403b_9354_28814dc54463.slice/crio-2bd411c8e82dc41f213ef5052efb9fa32d722e1c0c618e972ba2f47fb040d913 WatchSource:0}: Error finding container 2bd411c8e82dc41f213ef5052efb9fa32d722e1c0c618e972ba2f47fb040d913: Status 404 returned error can't find the container with id 2bd411c8e82dc41f213ef5052efb9fa32d722e1c0c618e972ba2f47fb040d913
Jul 16 23:28:22.145054 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:22.144969    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll" event={"ID":"e26ce678-61da-403b-9354-28814dc54463","Type":"ContainerStarted","Data":"dcd5643476ec9816d80aab178d34a2a885b9e2fc7bb9369ec19ce0dccec99d9f"}
Jul 16 23:28:22.145054 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:22.145027    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll" event={"ID":"e26ce678-61da-403b-9354-28814dc54463","Type":"ContainerStarted","Data":"2bd411c8e82dc41f213ef5052efb9fa32d722e1c0c618e972ba2f47fb040d913"}
Jul 16 23:28:22.160722 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:22.160668    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll" podStartSLOduration=1.1606541 podStartE2EDuration="1.1606541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:28:22.158675668 +0000 UTC m=+3312.907159762" watchObservedRunningTime="2026-07-16 23:28:22.1606541 +0000 UTC m=+3312.909138198"
Jul 16 23:28:23.152578 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:23.152537    2565 generic.go:356] "Generic (PLEG): container finished" podID="e26ce678-61da-403b-9354-28814dc54463" containerID="dcd5643476ec9816d80aab178d34a2a885b9e2fc7bb9369ec19ce0dccec99d9f" exitCode=6
Jul 16 23:28:23.153078 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:23.152661    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll" event={"ID":"e26ce678-61da-403b-9354-28814dc54463","Type":"ContainerDied","Data":"dcd5643476ec9816d80aab178d34a2a885b9e2fc7bb9369ec19ce0dccec99d9f"}
Jul 16 23:28:23.153078 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:23.152837    2565 scope.go:117] "RemoveContainer" containerID="dcd5643476ec9816d80aab178d34a2a885b9e2fc7bb9369ec19ce0dccec99d9f"
Jul 16 23:28:24.160681 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:24.160636    2565 generic.go:356] "Generic (PLEG): container finished" podID="e26ce678-61da-403b-9354-28814dc54463" containerID="8a916bbda98a6b753edac819f78b9b521dfccbc4361d48b80c164ecb803a9cdd" exitCode=6
Jul 16 23:28:24.161154 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:24.160684    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll" event={"ID":"e26ce678-61da-403b-9354-28814dc54463","Type":"ContainerDied","Data":"8a916bbda98a6b753edac819f78b9b521dfccbc4361d48b80c164ecb803a9cdd"}
Jul 16 23:28:24.161154 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:24.160767    2565 scope.go:117] "RemoveContainer" containerID="dcd5643476ec9816d80aab178d34a2a885b9e2fc7bb9369ec19ce0dccec99d9f"
Jul 16 23:28:24.161154 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:24.160933    2565 scope.go:117] "RemoveContainer" containerID="8a916bbda98a6b753edac819f78b9b521dfccbc4361d48b80c164ecb803a9cdd"
Jul 16 23:28:24.161345 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:28:24.161198    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll_odh-ai-gateway-infra(e26ce678-61da-403b-9354-28814dc54463)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll" podUID="e26ce678-61da-403b-9354-28814dc54463"
Jul 16 23:28:25.167445 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:25.167418    2565 scope.go:117] "RemoveContainer" containerID="8a916bbda98a6b753edac819f78b9b521dfccbc4361d48b80c164ecb803a9cdd"
Jul 16 23:28:25.167840 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:28:25.167631    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll_odh-ai-gateway-infra(e26ce678-61da-403b-9354-28814dc54463)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll" podUID="e26ce678-61da-403b-9354-28814dc54463"
Jul 16 23:28:38.720579 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:38.720507    2565 scope.go:117] "RemoveContainer" containerID="8a916bbda98a6b753edac819f78b9b521dfccbc4361d48b80c164ecb803a9cdd"
Jul 16 23:28:39.242587 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:39.242551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll" event={"ID":"e26ce678-61da-403b-9354-28814dc54463","Type":"ContainerStarted","Data":"0e9d8632af4ea61a71b9da84f39415352c668a2ecbe226c4f552f8c9c3af77d6"}
Jul 16 23:28:40.249824 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:40.249789    2565 generic.go:356] "Generic (PLEG): container finished" podID="e26ce678-61da-403b-9354-28814dc54463" containerID="0e9d8632af4ea61a71b9da84f39415352c668a2ecbe226c4f552f8c9c3af77d6" exitCode=6
Jul 16 23:28:40.250309 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:40.249858    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll" event={"ID":"e26ce678-61da-403b-9354-28814dc54463","Type":"ContainerDied","Data":"0e9d8632af4ea61a71b9da84f39415352c668a2ecbe226c4f552f8c9c3af77d6"}
Jul 16 23:28:40.250309 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:40.249918    2565 scope.go:117] "RemoveContainer" containerID="8a916bbda98a6b753edac819f78b9b521dfccbc4361d48b80c164ecb803a9cdd"
Jul 16 23:28:40.250309 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:40.250192    2565 scope.go:117] "RemoveContainer" containerID="0e9d8632af4ea61a71b9da84f39415352c668a2ecbe226c4f552f8c9c3af77d6"
Jul 16 23:28:40.250456 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:28:40.250433    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll_odh-ai-gateway-infra(e26ce678-61da-403b-9354-28814dc54463)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll" podUID="e26ce678-61da-403b-9354-28814dc54463"
Jul 16 23:28:40.271345 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:40.271317    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll"]
Jul 16 23:28:41.394830 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:41.394804    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll"
Jul 16 23:28:41.489696 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:41.489642    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26ce678-61da-403b-9354-28814dc54463-maas-api-service-ca\") pod \"e26ce678-61da-403b-9354-28814dc54463\" (UID: \"e26ce678-61da-403b-9354-28814dc54463\") "
Jul 16 23:28:41.490008 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:41.489985    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e26ce678-61da-403b-9354-28814dc54463-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e26ce678-61da-403b-9354-28814dc54463" (UID: "e26ce678-61da-403b-9354-28814dc54463"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:28:41.590455 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:41.590413    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26ce678-61da-403b-9354-28814dc54463-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:28:42.262716 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:42.262680    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll" event={"ID":"e26ce678-61da-403b-9354-28814dc54463","Type":"ContainerDied","Data":"2bd411c8e82dc41f213ef5052efb9fa32d722e1c0c618e972ba2f47fb040d913"}
Jul 16 23:28:42.262716 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:42.262722    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll"
Jul 16 23:28:42.263004 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:42.262758    2565 scope.go:117] "RemoveContainer" containerID="0e9d8632af4ea61a71b9da84f39415352c668a2ecbe226c4f552f8c9c3af77d6"
Jul 16 23:28:42.279915 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:42.279887    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll"]
Jul 16 23:28:42.285696 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:42.285666    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-z7lll"]
Jul 16 23:28:43.473582 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.473546    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6"]
Jul 16 23:28:43.473979 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.473928    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e26ce678-61da-403b-9354-28814dc54463" containerName="revoke-keys"
Jul 16 23:28:43.473979 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.473940    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e26ce678-61da-403b-9354-28814dc54463" containerName="revoke-keys"
Jul 16 23:28:43.473979 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.473955    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62fce735-14f0-4b16-b0f5-9abf09c04065" containerName="revoke-keys"
Jul 16 23:28:43.473979 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.473961    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62fce735-14f0-4b16-b0f5-9abf09c04065" containerName="revoke-keys"
Jul 16 23:28:43.473979 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.473976    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e26ce678-61da-403b-9354-28814dc54463" containerName="revoke-keys"
Jul 16 23:28:43.473979 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.473981    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e26ce678-61da-403b-9354-28814dc54463" containerName="revoke-keys"
Jul 16 23:28:43.474174 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.473993    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e26ce678-61da-403b-9354-28814dc54463" containerName="revoke-keys"
Jul 16 23:28:43.474174 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.473999    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e26ce678-61da-403b-9354-28814dc54463" containerName="revoke-keys"
Jul 16 23:28:43.474174 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.474054    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62fce735-14f0-4b16-b0f5-9abf09c04065" containerName="revoke-keys"
Jul 16 23:28:43.474174 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.474060    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e26ce678-61da-403b-9354-28814dc54463" containerName="revoke-keys"
Jul 16 23:28:43.474174 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.474068    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e26ce678-61da-403b-9354-28814dc54463" containerName="revoke-keys"
Jul 16 23:28:43.478364 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.478345    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6"
Jul 16 23:28:43.480661 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.480633    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:28:43.483532 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.483502    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6"]
Jul 16 23:28:43.512022 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.511986    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/311352e6-26cf-458b-9471-5fc44347f57b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-jc6t6\" (UID: \"311352e6-26cf-458b-9471-5fc44347f57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6"
Jul 16 23:28:43.613045 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.613004    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/311352e6-26cf-458b-9471-5fc44347f57b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-jc6t6\" (UID: \"311352e6-26cf-458b-9471-5fc44347f57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6"
Jul 16 23:28:43.613713 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.613690    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/311352e6-26cf-458b-9471-5fc44347f57b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-jc6t6\" (UID: \"311352e6-26cf-458b-9471-5fc44347f57b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6"
Jul 16 23:28:43.726911 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.726829    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e26ce678-61da-403b-9354-28814dc54463" path="/var/lib/kubelet/pods/e26ce678-61da-403b-9354-28814dc54463/volumes"
Jul 16 23:28:43.791181 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.791151    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6"
Jul 16 23:28:43.925100 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:43.925071    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6"]
Jul 16 23:28:43.927142 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:28:43.927114    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311352e6_26cf_458b_9471_5fc44347f57b.slice/crio-7d84cf1c307b394bbaaa711a394095537188c1c1a040416627abce8614d4a883 WatchSource:0}: Error finding container 7d84cf1c307b394bbaaa711a394095537188c1c1a040416627abce8614d4a883: Status 404 returned error can't find the container with id 7d84cf1c307b394bbaaa711a394095537188c1c1a040416627abce8614d4a883
Jul 16 23:28:44.274968 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:44.274927    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6" event={"ID":"311352e6-26cf-458b-9471-5fc44347f57b","Type":"ContainerStarted","Data":"8bdf74c0b86c863f24f5e6de51e393cb2974e92ceeec9db47c9bba6c7dfb000f"}
Jul 16 23:28:44.275129 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:44.274985    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6" event={"ID":"311352e6-26cf-458b-9471-5fc44347f57b","Type":"ContainerStarted","Data":"7d84cf1c307b394bbaaa711a394095537188c1c1a040416627abce8614d4a883"}
Jul 16 23:28:44.289415 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:44.289360    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6" podStartSLOduration=1.289344949 podStartE2EDuration="1.289344949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:28:44.287705947 +0000 UTC m=+3335.036190030" watchObservedRunningTime="2026-07-16 23:28:44.289344949 +0000 UTC m=+3335.037829049"
Jul 16 23:28:45.281835 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:45.281797    2565 generic.go:356] "Generic (PLEG): container finished" podID="311352e6-26cf-458b-9471-5fc44347f57b" containerID="8bdf74c0b86c863f24f5e6de51e393cb2974e92ceeec9db47c9bba6c7dfb000f" exitCode=6
Jul 16 23:28:45.282274 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:45.281891    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6" event={"ID":"311352e6-26cf-458b-9471-5fc44347f57b","Type":"ContainerDied","Data":"8bdf74c0b86c863f24f5e6de51e393cb2974e92ceeec9db47c9bba6c7dfb000f"}
Jul 16 23:28:45.282274 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:45.282129    2565 scope.go:117] "RemoveContainer" containerID="8bdf74c0b86c863f24f5e6de51e393cb2974e92ceeec9db47c9bba6c7dfb000f"
Jul 16 23:28:46.289326 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:46.289220    2565 generic.go:356] "Generic (PLEG): container finished" podID="311352e6-26cf-458b-9471-5fc44347f57b" containerID="2cf500b5e21fe005191fe81ea0a4e7cb2604710588b11111a4d3d48f64e82e0f" exitCode=6
Jul 16 23:28:46.289326 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:46.289311    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6" event={"ID":"311352e6-26cf-458b-9471-5fc44347f57b","Type":"ContainerDied","Data":"2cf500b5e21fe005191fe81ea0a4e7cb2604710588b11111a4d3d48f64e82e0f"}
Jul 16 23:28:46.289831 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:46.289372    2565 scope.go:117] "RemoveContainer" containerID="8bdf74c0b86c863f24f5e6de51e393cb2974e92ceeec9db47c9bba6c7dfb000f"
Jul 16 23:28:46.289831 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:46.289620    2565 scope.go:117] "RemoveContainer" containerID="2cf500b5e21fe005191fe81ea0a4e7cb2604710588b11111a4d3d48f64e82e0f"
Jul 16 23:28:46.289940 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:28:46.289920    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-jc6t6_odh-ai-gateway-infra(311352e6-26cf-458b-9471-5fc44347f57b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6" podUID="311352e6-26cf-458b-9471-5fc44347f57b"
Jul 16 23:28:47.296076 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:47.296047    2565 scope.go:117] "RemoveContainer" containerID="2cf500b5e21fe005191fe81ea0a4e7cb2604710588b11111a4d3d48f64e82e0f"
Jul 16 23:28:47.296499 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:28:47.296290    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-jc6t6_odh-ai-gateway-infra(311352e6-26cf-458b-9471-5fc44347f57b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6" podUID="311352e6-26cf-458b-9471-5fc44347f57b"
Jul 16 23:28:48.446160 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:48.446128    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9"]
Jul 16 23:28:48.446606 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:48.446592    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e26ce678-61da-403b-9354-28814dc54463" containerName="revoke-keys"
Jul 16 23:28:48.451081 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:48.451064    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9"
Jul 16 23:28:48.457214 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:48.457188    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9"]
Jul 16 23:28:48.563537 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:48.563505    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74dc3930-bfa8-4193-9ec3-7153feab9d4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9\" (UID: \"74dc3930-bfa8-4193-9ec3-7153feab9d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9"
Jul 16 23:28:48.664768 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:48.664731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74dc3930-bfa8-4193-9ec3-7153feab9d4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9\" (UID: \"74dc3930-bfa8-4193-9ec3-7153feab9d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9"
Jul 16 23:28:48.665380 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:48.665361    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74dc3930-bfa8-4193-9ec3-7153feab9d4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9\" (UID: \"74dc3930-bfa8-4193-9ec3-7153feab9d4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9"
Jul 16 23:28:48.763498 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:48.763409    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9"
Jul 16 23:28:48.890970 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:48.890947    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9"]
Jul 16 23:28:48.894173 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:28:48.894135    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74dc3930_bfa8_4193_9ec3_7153feab9d4a.slice/crio-8d4520a5c1bc6c54885d4f581dcc3025dce2c76462e6349441c5fcd3b5ffe8f7 WatchSource:0}: Error finding container 8d4520a5c1bc6c54885d4f581dcc3025dce2c76462e6349441c5fcd3b5ffe8f7: Status 404 returned error can't find the container with id 8d4520a5c1bc6c54885d4f581dcc3025dce2c76462e6349441c5fcd3b5ffe8f7
Jul 16 23:28:49.307680 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:49.307591    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9" event={"ID":"74dc3930-bfa8-4193-9ec3-7153feab9d4a","Type":"ContainerStarted","Data":"f6e56745f58bbdec6b4f5ae299226e67e2fd97dcf7a6434cbad70a25e7d66de9"}
Jul 16 23:28:49.307680 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:49.307644    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9" event={"ID":"74dc3930-bfa8-4193-9ec3-7153feab9d4a","Type":"ContainerStarted","Data":"8d4520a5c1bc6c54885d4f581dcc3025dce2c76462e6349441c5fcd3b5ffe8f7"}
Jul 16 23:28:49.324317 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:49.324256    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9" podStartSLOduration=1.324228194 podStartE2EDuration="1.324228194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:28:49.321938554 +0000 UTC m=+3340.070422626" watchObservedRunningTime="2026-07-16 23:28:49.324228194 +0000 UTC m=+3340.072712292"
Jul 16 23:28:50.313673 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:50.313633    2565 generic.go:356] "Generic (PLEG): container finished" podID="74dc3930-bfa8-4193-9ec3-7153feab9d4a" containerID="f6e56745f58bbdec6b4f5ae299226e67e2fd97dcf7a6434cbad70a25e7d66de9" exitCode=6
Jul 16 23:28:50.314123 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:50.313724    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9" event={"ID":"74dc3930-bfa8-4193-9ec3-7153feab9d4a","Type":"ContainerDied","Data":"f6e56745f58bbdec6b4f5ae299226e67e2fd97dcf7a6434cbad70a25e7d66de9"}
Jul 16 23:28:50.314123 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:50.313866    2565 scope.go:117] "RemoveContainer" containerID="f6e56745f58bbdec6b4f5ae299226e67e2fd97dcf7a6434cbad70a25e7d66de9"
Jul 16 23:28:51.320350 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:51.320230    2565 generic.go:356] "Generic (PLEG): container finished" podID="74dc3930-bfa8-4193-9ec3-7153feab9d4a" containerID="79e07adb83f0b4f4207a1419920c544d81283fd289276d369ae3297bd32d825d" exitCode=6
Jul 16 23:28:51.320350 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:51.320306    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9" event={"ID":"74dc3930-bfa8-4193-9ec3-7153feab9d4a","Type":"ContainerDied","Data":"79e07adb83f0b4f4207a1419920c544d81283fd289276d369ae3297bd32d825d"}
Jul 16 23:28:51.320806 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:51.320357    2565 scope.go:117] "RemoveContainer" containerID="f6e56745f58bbdec6b4f5ae299226e67e2fd97dcf7a6434cbad70a25e7d66de9"
Jul 16 23:28:51.320806 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:51.320581    2565 scope.go:117] "RemoveContainer" containerID="79e07adb83f0b4f4207a1419920c544d81283fd289276d369ae3297bd32d825d"
Jul 16 23:28:51.320879 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:28:51.320814    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9_odh-ai-gateway-infra(74dc3930-bfa8-4193-9ec3-7153feab9d4a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9" podUID="74dc3930-bfa8-4193-9ec3-7153feab9d4a"
Jul 16 23:28:52.292081 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:52.292045    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w"]
Jul 16 23:28:52.295451 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:52.295435    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w"
Jul 16 23:28:52.302548 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:52.302525    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w"]
Jul 16 23:28:52.327466 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:52.327442    2565 scope.go:117] "RemoveContainer" containerID="79e07adb83f0b4f4207a1419920c544d81283fd289276d369ae3297bd32d825d"
Jul 16 23:28:52.327790 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:28:52.327748    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9_odh-ai-gateway-infra(74dc3930-bfa8-4193-9ec3-7153feab9d4a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9" podUID="74dc3930-bfa8-4193-9ec3-7153feab9d4a"
Jul 16 23:28:52.401848 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:52.401811    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf77b1b7-d1e9-4d81-8eca-d9ed361dd293-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w\" (UID: \"cf77b1b7-d1e9-4d81-8eca-d9ed361dd293\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w"
Jul 16 23:28:52.503267 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:52.503213    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf77b1b7-d1e9-4d81-8eca-d9ed361dd293-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w\" (UID: \"cf77b1b7-d1e9-4d81-8eca-d9ed361dd293\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w"
Jul 16 23:28:52.503896 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:52.503869    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf77b1b7-d1e9-4d81-8eca-d9ed361dd293-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w\" (UID: \"cf77b1b7-d1e9-4d81-8eca-d9ed361dd293\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w"
Jul 16 23:28:52.608313 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:52.608223    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w"
Jul 16 23:28:52.742059 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:52.742022    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w"]
Jul 16 23:28:52.744950 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:28:52.744922    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf77b1b7_d1e9_4d81_8eca_d9ed361dd293.slice/crio-f9f858686421a5d5c34ae7e857beef16e22560ac6da948a7a054d1174ec54551 WatchSource:0}: Error finding container f9f858686421a5d5c34ae7e857beef16e22560ac6da948a7a054d1174ec54551: Status 404 returned error can't find the container with id f9f858686421a5d5c34ae7e857beef16e22560ac6da948a7a054d1174ec54551
Jul 16 23:28:53.333384 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:53.333348    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w" event={"ID":"cf77b1b7-d1e9-4d81-8eca-d9ed361dd293","Type":"ContainerStarted","Data":"301a59775fd0e37757d710fb7d046bbbba835837067360b39d555c2b4c7c7bec"}
Jul 16 23:28:53.333790 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:53.333401    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w" event={"ID":"cf77b1b7-d1e9-4d81-8eca-d9ed361dd293","Type":"ContainerStarted","Data":"f9f858686421a5d5c34ae7e857beef16e22560ac6da948a7a054d1174ec54551"}
Jul 16 23:28:53.348523 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:53.348479    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w" podStartSLOduration=1.348465652 podStartE2EDuration="1.348465652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:28:53.346837035 +0000 UTC m=+3344.095321118" watchObservedRunningTime="2026-07-16 23:28:53.348465652 +0000 UTC m=+3344.096949722"
Jul 16 23:28:54.340968 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:54.340934    2565 generic.go:356] "Generic (PLEG): container finished" podID="cf77b1b7-d1e9-4d81-8eca-d9ed361dd293" containerID="301a59775fd0e37757d710fb7d046bbbba835837067360b39d555c2b4c7c7bec" exitCode=6
Jul 16 23:28:54.341413 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:54.340980    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w" event={"ID":"cf77b1b7-d1e9-4d81-8eca-d9ed361dd293","Type":"ContainerDied","Data":"301a59775fd0e37757d710fb7d046bbbba835837067360b39d555c2b4c7c7bec"}
Jul 16 23:28:54.341413 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:54.341184    2565 scope.go:117] "RemoveContainer" containerID="301a59775fd0e37757d710fb7d046bbbba835837067360b39d555c2b4c7c7bec"
Jul 16 23:28:55.355290 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:55.355231    2565 generic.go:356] "Generic (PLEG): container finished" podID="cf77b1b7-d1e9-4d81-8eca-d9ed361dd293" containerID="5b7d1ac60931479ff1681124435506daa4e27a93f9730a4d590023abbb531ac2" exitCode=6
Jul 16 23:28:55.355677 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:55.355316    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w" event={"ID":"cf77b1b7-d1e9-4d81-8eca-d9ed361dd293","Type":"ContainerDied","Data":"5b7d1ac60931479ff1681124435506daa4e27a93f9730a4d590023abbb531ac2"}
Jul 16 23:28:55.355677 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:55.355399    2565 scope.go:117] "RemoveContainer" containerID="301a59775fd0e37757d710fb7d046bbbba835837067360b39d555c2b4c7c7bec"
Jul 16 23:28:55.355677 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:55.355638    2565 scope.go:117] "RemoveContainer" containerID="5b7d1ac60931479ff1681124435506daa4e27a93f9730a4d590023abbb531ac2"
Jul 16 23:28:55.355912 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:28:55.355893    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w_odh-ai-gateway-infra(cf77b1b7-d1e9-4d81-8eca-d9ed361dd293)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w" podUID="cf77b1b7-d1e9-4d81-8eca-d9ed361dd293"
Jul 16 23:28:56.362928 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:28:56.362897    2565 scope.go:117] "RemoveContainer" containerID="5b7d1ac60931479ff1681124435506daa4e27a93f9730a4d590023abbb531ac2"
Jul 16 23:28:56.363416 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:28:56.363104    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w_odh-ai-gateway-infra(cf77b1b7-d1e9-4d81-8eca-d9ed361dd293)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w" podUID="cf77b1b7-d1e9-4d81-8eca-d9ed361dd293"
Jul 16 23:29:02.720881 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:02.720851    2565 scope.go:117] "RemoveContainer" containerID="2cf500b5e21fe005191fe81ea0a4e7cb2604710588b11111a4d3d48f64e82e0f"
Jul 16 23:29:03.400146 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:03.400109    2565 generic.go:356] "Generic (PLEG): container finished" podID="311352e6-26cf-458b-9471-5fc44347f57b" containerID="8e852d987d0eaaabb803a8352aaa55d4eef60d13281bc3c9f8186a136fdf1b0b" exitCode=6
Jul 16 23:29:03.400351 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:03.400189    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6" event={"ID":"311352e6-26cf-458b-9471-5fc44347f57b","Type":"ContainerDied","Data":"8e852d987d0eaaabb803a8352aaa55d4eef60d13281bc3c9f8186a136fdf1b0b"}
Jul 16 23:29:03.400351 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:03.400279    2565 scope.go:117] "RemoveContainer" containerID="2cf500b5e21fe005191fe81ea0a4e7cb2604710588b11111a4d3d48f64e82e0f"
Jul 16 23:29:03.400571 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:03.400555    2565 scope.go:117] "RemoveContainer" containerID="8e852d987d0eaaabb803a8352aaa55d4eef60d13281bc3c9f8186a136fdf1b0b"
Jul 16 23:29:03.400813 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:29:03.400792    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-jc6t6_odh-ai-gateway-infra(311352e6-26cf-458b-9471-5fc44347f57b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6" podUID="311352e6-26cf-458b-9471-5fc44347f57b"
Jul 16 23:29:03.721422 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:03.721331    2565 scope.go:117] "RemoveContainer" containerID="79e07adb83f0b4f4207a1419920c544d81283fd289276d369ae3297bd32d825d"
Jul 16 23:29:04.407456 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:04.407420    2565 generic.go:356] "Generic (PLEG): container finished" podID="74dc3930-bfa8-4193-9ec3-7153feab9d4a" containerID="b775b697841edf03de1d1bb2f63993be8085ec2c0ff813731f574b684dcd3857" exitCode=6
Jul 16 23:29:04.407644 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:04.407490    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9" event={"ID":"74dc3930-bfa8-4193-9ec3-7153feab9d4a","Type":"ContainerDied","Data":"b775b697841edf03de1d1bb2f63993be8085ec2c0ff813731f574b684dcd3857"}
Jul 16 23:29:04.407644 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:04.407552    2565 scope.go:117] "RemoveContainer" containerID="79e07adb83f0b4f4207a1419920c544d81283fd289276d369ae3297bd32d825d"
Jul 16 23:29:04.407860 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:04.407832    2565 scope.go:117] "RemoveContainer" containerID="b775b697841edf03de1d1bb2f63993be8085ec2c0ff813731f574b684dcd3857"
Jul 16 23:29:04.408120 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:29:04.408098    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9_odh-ai-gateway-infra(74dc3930-bfa8-4193-9ec3-7153feab9d4a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9" podUID="74dc3930-bfa8-4193-9ec3-7153feab9d4a"
Jul 16 23:29:04.425609 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:04.425581    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6"]
Jul 16 23:29:04.568633 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:04.568606    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6"
Jul 16 23:29:04.728463 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:04.728351    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/311352e6-26cf-458b-9471-5fc44347f57b-maas-api-service-ca\") pod \"311352e6-26cf-458b-9471-5fc44347f57b\" (UID: \"311352e6-26cf-458b-9471-5fc44347f57b\") "
Jul 16 23:29:04.728917 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:04.728711    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311352e6-26cf-458b-9471-5fc44347f57b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "311352e6-26cf-458b-9471-5fc44347f57b" (UID: "311352e6-26cf-458b-9471-5fc44347f57b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:29:04.728917 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:04.728789    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/311352e6-26cf-458b-9471-5fc44347f57b-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:29:05.416108 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:05.416079    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6"
Jul 16 23:29:05.416108 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:05.416092    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6" event={"ID":"311352e6-26cf-458b-9471-5fc44347f57b","Type":"ContainerDied","Data":"7d84cf1c307b394bbaaa711a394095537188c1c1a040416627abce8614d4a883"}
Jul 16 23:29:05.416403 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:05.416154    2565 scope.go:117] "RemoveContainer" containerID="8e852d987d0eaaabb803a8352aaa55d4eef60d13281bc3c9f8186a136fdf1b0b"
Jul 16 23:29:05.432671 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:05.432641    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9"]
Jul 16 23:29:05.442642 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:05.442601    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6"]
Jul 16 23:29:05.444730 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:05.444702    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-jc6t6"]
Jul 16 23:29:05.577167 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:05.577130    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9"
Jul 16 23:29:05.726934 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:05.726850    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="311352e6-26cf-458b-9471-5fc44347f57b" path="/var/lib/kubelet/pods/311352e6-26cf-458b-9471-5fc44347f57b/volumes"
Jul 16 23:29:05.743037 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:05.743003    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74dc3930-bfa8-4193-9ec3-7153feab9d4a-maas-api-service-ca\") pod \"74dc3930-bfa8-4193-9ec3-7153feab9d4a\" (UID: \"74dc3930-bfa8-4193-9ec3-7153feab9d4a\") "
Jul 16 23:29:05.743626 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:05.743600    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74dc3930-bfa8-4193-9ec3-7153feab9d4a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74dc3930-bfa8-4193-9ec3-7153feab9d4a" (UID: "74dc3930-bfa8-4193-9ec3-7153feab9d4a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:29:05.845079 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:05.845041    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74dc3930-bfa8-4193-9ec3-7153feab9d4a-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:29:06.425149 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:06.425113    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9"
Jul 16 23:29:06.425372 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:06.425145    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9" event={"ID":"74dc3930-bfa8-4193-9ec3-7153feab9d4a","Type":"ContainerDied","Data":"8d4520a5c1bc6c54885d4f581dcc3025dce2c76462e6349441c5fcd3b5ffe8f7"}
Jul 16 23:29:06.425372 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:06.425232    2565 scope.go:117] "RemoveContainer" containerID="b775b697841edf03de1d1bb2f63993be8085ec2c0ff813731f574b684dcd3857"
Jul 16 23:29:06.450600 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:06.450564    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9"]
Jul 16 23:29:06.455697 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:06.455661    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-nwzb9"]
Jul 16 23:29:07.726418 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:07.726381    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74dc3930-bfa8-4193-9ec3-7153feab9d4a" path="/var/lib/kubelet/pods/74dc3930-bfa8-4193-9ec3-7153feab9d4a/volumes"
Jul 16 23:29:10.720604 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:10.720567    2565 scope.go:117] "RemoveContainer" containerID="5b7d1ac60931479ff1681124435506daa4e27a93f9730a4d590023abbb531ac2"
Jul 16 23:29:11.453008 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:11.452976    2565 generic.go:356] "Generic (PLEG): container finished" podID="cf77b1b7-d1e9-4d81-8eca-d9ed361dd293" containerID="523cb7371963f0c82d79434bf8035a79faebcfddc74f19017c99eb157e862a3d" exitCode=6
Jul 16 23:29:11.453189 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:11.453055    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w" event={"ID":"cf77b1b7-d1e9-4d81-8eca-d9ed361dd293","Type":"ContainerDied","Data":"523cb7371963f0c82d79434bf8035a79faebcfddc74f19017c99eb157e862a3d"}
Jul 16 23:29:11.453189 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:11.453122    2565 scope.go:117] "RemoveContainer" containerID="5b7d1ac60931479ff1681124435506daa4e27a93f9730a4d590023abbb531ac2"
Jul 16 23:29:11.453480 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:11.453464    2565 scope.go:117] "RemoveContainer" containerID="523cb7371963f0c82d79434bf8035a79faebcfddc74f19017c99eb157e862a3d"
Jul 16 23:29:11.453775 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:29:11.453749    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w_odh-ai-gateway-infra(cf77b1b7-d1e9-4d81-8eca-d9ed361dd293)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w" podUID="cf77b1b7-d1e9-4d81-8eca-d9ed361dd293"
Jul 16 23:29:12.477776 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:12.477736    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w"]
Jul 16 23:29:12.634193 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:12.634159    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w"
Jul 16 23:29:12.712460 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:12.712419    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf77b1b7-d1e9-4d81-8eca-d9ed361dd293-maas-api-service-ca\") pod \"cf77b1b7-d1e9-4d81-8eca-d9ed361dd293\" (UID: \"cf77b1b7-d1e9-4d81-8eca-d9ed361dd293\") "
Jul 16 23:29:12.712857 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:12.712830    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf77b1b7-d1e9-4d81-8eca-d9ed361dd293-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cf77b1b7-d1e9-4d81-8eca-d9ed361dd293" (UID: "cf77b1b7-d1e9-4d81-8eca-d9ed361dd293"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:29:12.813783 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:12.813680    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf77b1b7-d1e9-4d81-8eca-d9ed361dd293-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:29:13.465900 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:13.465867    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w"
Jul 16 23:29:13.466081 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:13.465908    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w" event={"ID":"cf77b1b7-d1e9-4d81-8eca-d9ed361dd293","Type":"ContainerDied","Data":"f9f858686421a5d5c34ae7e857beef16e22560ac6da948a7a054d1174ec54551"}
Jul 16 23:29:13.466081 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:13.465970    2565 scope.go:117] "RemoveContainer" containerID="523cb7371963f0c82d79434bf8035a79faebcfddc74f19017c99eb157e862a3d"
Jul 16 23:29:13.490734 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:13.490702    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w"]
Jul 16 23:29:13.494162 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:13.494137    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-brj2w"]
Jul 16 23:29:13.729215 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:13.729095    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf77b1b7-d1e9-4d81-8eca-d9ed361dd293" path="/var/lib/kubelet/pods/cf77b1b7-d1e9-4d81-8eca-d9ed361dd293/volumes"
Jul 16 23:29:43.511686 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.511651    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d"]
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512031    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf77b1b7-d1e9-4d81-8eca-d9ed361dd293" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512044    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf77b1b7-d1e9-4d81-8eca-d9ed361dd293" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512057    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311352e6-26cf-458b-9471-5fc44347f57b" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512063    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="311352e6-26cf-458b-9471-5fc44347f57b" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512070    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74dc3930-bfa8-4193-9ec3-7153feab9d4a" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512076    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="74dc3930-bfa8-4193-9ec3-7153feab9d4a" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512081    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf77b1b7-d1e9-4d81-8eca-d9ed361dd293" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512088    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf77b1b7-d1e9-4d81-8eca-d9ed361dd293" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512093    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf77b1b7-d1e9-4d81-8eca-d9ed361dd293" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512098    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf77b1b7-d1e9-4d81-8eca-d9ed361dd293" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512108    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74dc3930-bfa8-4193-9ec3-7153feab9d4a" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512114    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="74dc3930-bfa8-4193-9ec3-7153feab9d4a" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512126    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311352e6-26cf-458b-9471-5fc44347f57b" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512132    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="311352e6-26cf-458b-9471-5fc44347f57b" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512142    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311352e6-26cf-458b-9471-5fc44347f57b" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512147    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="311352e6-26cf-458b-9471-5fc44347f57b" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512202    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="311352e6-26cf-458b-9471-5fc44347f57b" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512209    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf77b1b7-d1e9-4d81-8eca-d9ed361dd293" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512215    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="311352e6-26cf-458b-9471-5fc44347f57b" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512222    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="311352e6-26cf-458b-9471-5fc44347f57b" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512229    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="74dc3930-bfa8-4193-9ec3-7153feab9d4a" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512235    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf77b1b7-d1e9-4d81-8eca-d9ed361dd293" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512241    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="74dc3930-bfa8-4193-9ec3-7153feab9d4a" containerName="revoke-keys"
Jul 16 23:29:43.514043 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.512262    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="74dc3930-bfa8-4193-9ec3-7153feab9d4a" containerName="revoke-keys"
Jul 16 23:29:43.516018 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.516001    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d"
Jul 16 23:29:43.518281 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.518243    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:29:43.522098 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.522075    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d"]
Jul 16 23:29:43.613737 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.613698    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a068194-55ac-4b35-9432-d9bbdeb97602-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-lgz6d\" (UID: \"1a068194-55ac-4b35-9432-d9bbdeb97602\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d"
Jul 16 23:29:43.714402 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.714366    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a068194-55ac-4b35-9432-d9bbdeb97602-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-lgz6d\" (UID: \"1a068194-55ac-4b35-9432-d9bbdeb97602\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d"
Jul 16 23:29:43.714978 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.714958    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a068194-55ac-4b35-9432-d9bbdeb97602-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-lgz6d\" (UID: \"1a068194-55ac-4b35-9432-d9bbdeb97602\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d"
Jul 16 23:29:43.828412 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.828380    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d"
Jul 16 23:29:43.959661 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:43.959478    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d"]
Jul 16 23:29:43.962660 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:29:43.962625    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a068194_55ac_4b35_9432_d9bbdeb97602.slice/crio-91890dc74cf1e5deb318b956c6b602170d786e9a14791d5abf9def68eb4117d0 WatchSource:0}: Error finding container 91890dc74cf1e5deb318b956c6b602170d786e9a14791d5abf9def68eb4117d0: Status 404 returned error can't find the container with id 91890dc74cf1e5deb318b956c6b602170d786e9a14791d5abf9def68eb4117d0
Jul 16 23:29:44.624343 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:44.624306    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d" event={"ID":"1a068194-55ac-4b35-9432-d9bbdeb97602","Type":"ContainerStarted","Data":"bf15a72c58808fd73a264a400c8a131d6a1a310e9ea7622f28b834c907f680bf"}
Jul 16 23:29:44.624748 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:44.624392    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d" event={"ID":"1a068194-55ac-4b35-9432-d9bbdeb97602","Type":"ContainerStarted","Data":"91890dc74cf1e5deb318b956c6b602170d786e9a14791d5abf9def68eb4117d0"}
Jul 16 23:29:44.639916 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:44.639865    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d" podStartSLOduration=1.6398505669999999 podStartE2EDuration="1.639850567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:29:44.637726853 +0000 UTC m=+3395.386210927" watchObservedRunningTime="2026-07-16 23:29:44.639850567 +0000 UTC m=+3395.388334686"
Jul 16 23:29:45.632197 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:45.632156    2565 generic.go:356] "Generic (PLEG): container finished" podID="1a068194-55ac-4b35-9432-d9bbdeb97602" containerID="bf15a72c58808fd73a264a400c8a131d6a1a310e9ea7622f28b834c907f680bf" exitCode=6
Jul 16 23:29:45.632623 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:45.632242    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d" event={"ID":"1a068194-55ac-4b35-9432-d9bbdeb97602","Type":"ContainerDied","Data":"bf15a72c58808fd73a264a400c8a131d6a1a310e9ea7622f28b834c907f680bf"}
Jul 16 23:29:45.632623 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:45.632492    2565 scope.go:117] "RemoveContainer" containerID="bf15a72c58808fd73a264a400c8a131d6a1a310e9ea7622f28b834c907f680bf"
Jul 16 23:29:46.639756 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:46.639664    2565 generic.go:356] "Generic (PLEG): container finished" podID="1a068194-55ac-4b35-9432-d9bbdeb97602" containerID="8bb18ccf632ec60f9d0a09febb56c68f44add624e2d1b0644a033c655f17c4c9" exitCode=6
Jul 16 23:29:46.639756 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:46.639741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d" event={"ID":"1a068194-55ac-4b35-9432-d9bbdeb97602","Type":"ContainerDied","Data":"8bb18ccf632ec60f9d0a09febb56c68f44add624e2d1b0644a033c655f17c4c9"}
Jul 16 23:29:46.640201 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:46.639787    2565 scope.go:117] "RemoveContainer" containerID="bf15a72c58808fd73a264a400c8a131d6a1a310e9ea7622f28b834c907f680bf"
Jul 16 23:29:46.640201 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:46.640059    2565 scope.go:117] "RemoveContainer" containerID="8bb18ccf632ec60f9d0a09febb56c68f44add624e2d1b0644a033c655f17c4c9"
Jul 16 23:29:46.640359 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:29:46.640332    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-lgz6d_odh-ai-gateway-infra(1a068194-55ac-4b35-9432-d9bbdeb97602)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d" podUID="1a068194-55ac-4b35-9432-d9bbdeb97602"
Jul 16 23:29:47.647331 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:47.647306    2565 scope.go:117] "RemoveContainer" containerID="8bb18ccf632ec60f9d0a09febb56c68f44add624e2d1b0644a033c655f17c4c9"
Jul 16 23:29:47.647728 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:29:47.647494    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-lgz6d_odh-ai-gateway-infra(1a068194-55ac-4b35-9432-d9bbdeb97602)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d" podUID="1a068194-55ac-4b35-9432-d9bbdeb97602"
Jul 16 23:29:52.353498 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:52.353466    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll"]
Jul 16 23:29:52.353890 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:52.353861    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74dc3930-bfa8-4193-9ec3-7153feab9d4a" containerName="revoke-keys"
Jul 16 23:29:52.353890 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:52.353873    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="74dc3930-bfa8-4193-9ec3-7153feab9d4a" containerName="revoke-keys"
Jul 16 23:29:52.353981 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:52.353945    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf77b1b7-d1e9-4d81-8eca-d9ed361dd293" containerName="revoke-keys"
Jul 16 23:29:52.356986 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:52.356967    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll"
Jul 16 23:29:52.364974 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:52.364948    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll"]
Jul 16 23:29:52.500838 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:52.500785    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7294ab-8657-4194-a9c5-ba1ff2da1734-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll\" (UID: \"cd7294ab-8657-4194-a9c5-ba1ff2da1734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll"
Jul 16 23:29:52.602214 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:52.602172    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7294ab-8657-4194-a9c5-ba1ff2da1734-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll\" (UID: \"cd7294ab-8657-4194-a9c5-ba1ff2da1734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll"
Jul 16 23:29:52.602853 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:52.602832    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7294ab-8657-4194-a9c5-ba1ff2da1734-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll\" (UID: \"cd7294ab-8657-4194-a9c5-ba1ff2da1734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll"
Jul 16 23:29:52.668721 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:52.668624    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll"
Jul 16 23:29:52.807135 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:52.807110    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll"]
Jul 16 23:29:52.809160 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:29:52.809133    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7294ab_8657_4194_a9c5_ba1ff2da1734.slice/crio-ceb6e9dd4c2a007f4d33a0fdd4c6a3da2613827b6191e586b35681c1c4e4065d WatchSource:0}: Error finding container ceb6e9dd4c2a007f4d33a0fdd4c6a3da2613827b6191e586b35681c1c4e4065d: Status 404 returned error can't find the container with id ceb6e9dd4c2a007f4d33a0fdd4c6a3da2613827b6191e586b35681c1c4e4065d
Jul 16 23:29:53.680076 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:53.680035    2565 generic.go:356] "Generic (PLEG): container finished" podID="cd7294ab-8657-4194-a9c5-ba1ff2da1734" containerID="64eae4eae5778940831349f3e50b8128e4d9bd1331976f10398ad1066c362397" exitCode=6
Jul 16 23:29:53.680535 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:53.680124    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll" event={"ID":"cd7294ab-8657-4194-a9c5-ba1ff2da1734","Type":"ContainerDied","Data":"64eae4eae5778940831349f3e50b8128e4d9bd1331976f10398ad1066c362397"}
Jul 16 23:29:53.680535 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:53.680187    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll" event={"ID":"cd7294ab-8657-4194-a9c5-ba1ff2da1734","Type":"ContainerStarted","Data":"ceb6e9dd4c2a007f4d33a0fdd4c6a3da2613827b6191e586b35681c1c4e4065d"}
Jul 16 23:29:53.680535 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:53.680337    2565 scope.go:117] "RemoveContainer" containerID="64eae4eae5778940831349f3e50b8128e4d9bd1331976f10398ad1066c362397"
Jul 16 23:29:54.687063 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:54.686967    2565 generic.go:356] "Generic (PLEG): container finished" podID="cd7294ab-8657-4194-a9c5-ba1ff2da1734" containerID="45db8198fd143d8f063befcb5cfa30118fdf2de19b33353a51b059f1c46b6b26" exitCode=6
Jul 16 23:29:54.687525 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:54.687056    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll" event={"ID":"cd7294ab-8657-4194-a9c5-ba1ff2da1734","Type":"ContainerDied","Data":"45db8198fd143d8f063befcb5cfa30118fdf2de19b33353a51b059f1c46b6b26"}
Jul 16 23:29:54.687525 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:54.687111    2565 scope.go:117] "RemoveContainer" containerID="64eae4eae5778940831349f3e50b8128e4d9bd1331976f10398ad1066c362397"
Jul 16 23:29:54.687525 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:54.687270    2565 scope.go:117] "RemoveContainer" containerID="45db8198fd143d8f063befcb5cfa30118fdf2de19b33353a51b059f1c46b6b26"
Jul 16 23:29:54.687659 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:29:54.687572    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll_odh-ai-gateway-infra(cd7294ab-8657-4194-a9c5-ba1ff2da1734)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll" podUID="cd7294ab-8657-4194-a9c5-ba1ff2da1734"
Jul 16 23:29:55.694106 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:55.694078    2565 scope.go:117] "RemoveContainer" containerID="45db8198fd143d8f063befcb5cfa30118fdf2de19b33353a51b059f1c46b6b26"
Jul 16 23:29:55.694542 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:29:55.694293    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll_odh-ai-gateway-infra(cd7294ab-8657-4194-a9c5-ba1ff2da1734)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll" podUID="cd7294ab-8657-4194-a9c5-ba1ff2da1734"
Jul 16 23:29:58.720971 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:58.720938    2565 scope.go:117] "RemoveContainer" containerID="8bb18ccf632ec60f9d0a09febb56c68f44add624e2d1b0644a033c655f17c4c9"
Jul 16 23:29:59.717027 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:59.716929    2565 generic.go:356] "Generic (PLEG): container finished" podID="1a068194-55ac-4b35-9432-d9bbdeb97602" containerID="417d2ba492e36dbf154b3e413f06f222412cbbc30eb66ff502a4f4206c57a303" exitCode=6
Jul 16 23:29:59.717027 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:59.716995    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d" event={"ID":"1a068194-55ac-4b35-9432-d9bbdeb97602","Type":"ContainerDied","Data":"417d2ba492e36dbf154b3e413f06f222412cbbc30eb66ff502a4f4206c57a303"}
Jul 16 23:29:59.717230 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:59.717057    2565 scope.go:117] "RemoveContainer" containerID="8bb18ccf632ec60f9d0a09febb56c68f44add624e2d1b0644a033c655f17c4c9"
Jul 16 23:29:59.717386 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:29:59.717370    2565 scope.go:117] "RemoveContainer" containerID="417d2ba492e36dbf154b3e413f06f222412cbbc30eb66ff502a4f4206c57a303"
Jul 16 23:29:59.717663 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:29:59.717639    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-lgz6d_odh-ai-gateway-infra(1a068194-55ac-4b35-9432-d9bbdeb97602)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d" podUID="1a068194-55ac-4b35-9432-d9bbdeb97602"
Jul 16 23:30:00.138487 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.138449    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-48992b-29737410-mz27d"]
Jul 16 23:30:00.143035 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.143011    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b4ab52-29737410-5g4pk"]
Jul 16 23:30:00.143144 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.143125    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-48992b-29737410-mz27d"
Jul 16 23:30:00.148522 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.148493    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737410-pvxnd"]
Jul 16 23:30:00.148702 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.148572    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b4ab52-29737410-5g4pk"
Jul 16 23:30:00.155119 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.155092    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737410-pvxnd"]
Jul 16 23:30:00.155493 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.155474    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-48992b-29737410-mz27d"]
Jul 16 23:30:00.155688 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.155414    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737410-pvxnd"
Jul 16 23:30:00.160353 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.160329    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b4ab52-29737410-5g4pk"]
Jul 16 23:30:00.176294 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.176242    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-48992b-29737410-mz27d"
Jul 16 23:30:00.183967 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.183941    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b4ab52-29737410-5g4pk"
Jul 16 23:30:00.216921 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.216882    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737410-pvxnd"
Jul 16 23:30:00.344487 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.344391    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-48992b-29737410-mz27d"]
Jul 16 23:30:00.349349 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:30:00.349219    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08986c7_a429_460f_bb25_6fb4730ac349.slice/crio-432f45f772336c1aad60dc77e003d09362af29b70e6ad3bdddaaa78c3151145d WatchSource:0}: Error finding container 432f45f772336c1aad60dc77e003d09362af29b70e6ad3bdddaaa78c3151145d: Status 404 returned error can't find the container with id 432f45f772336c1aad60dc77e003d09362af29b70e6ad3bdddaaa78c3151145d
Jul 16 23:30:00.383088 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.383045    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b4ab52-29737410-5g4pk"]
Jul 16 23:30:00.390797 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:30:00.390762    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7493e669_3670_452b_9a49_54ff0dd8546c.slice/crio-22fe8a6607b5d0e7c5edbd30b4fc3e5bcf62716d23dbeef742097b33b22929d8 WatchSource:0}: Error finding container 22fe8a6607b5d0e7c5edbd30b4fc3e5bcf62716d23dbeef742097b33b22929d8: Status 404 returned error can't find the container with id 22fe8a6607b5d0e7c5edbd30b4fc3e5bcf62716d23dbeef742097b33b22929d8
Jul 16 23:30:00.416066 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.416003    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737410-pvxnd"]
Jul 16 23:30:00.418640 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:30:00.418611    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b29a52_7376_407f_a096_7e2ba4b3101d.slice/crio-6eae1a1137412223f8064e26429ae70ef58f064774217d99fc3db51179f390c4 WatchSource:0}: Error finding container 6eae1a1137412223f8064e26429ae70ef58f064774217d99fc3db51179f390c4: Status 404 returned error can't find the container with id 6eae1a1137412223f8064e26429ae70ef58f064774217d99fc3db51179f390c4
Jul 16 23:30:00.723986 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.723949    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737410-pvxnd" event={"ID":"04b29a52-7376-407f-a096-7e2ba4b3101d","Type":"ContainerStarted","Data":"6eae1a1137412223f8064e26429ae70ef58f064774217d99fc3db51179f390c4"}
Jul 16 23:30:00.726619 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.726591    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b4ab52-29737410-5g4pk" event={"ID":"7493e669-3670-452b-9a49-54ff0dd8546c","Type":"ContainerStarted","Data":"22fe8a6607b5d0e7c5edbd30b4fc3e5bcf62716d23dbeef742097b33b22929d8"}
Jul 16 23:30:00.727804 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.727781    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-48992b-29737410-mz27d" event={"ID":"e08986c7-a429-460f-bb25-6fb4730ac349","Type":"ContainerStarted","Data":"4f80bb8093a363b90bec46eba042cc37526c4bac1e7333a0c83d0198b7f5d176"}
Jul 16 23:30:00.727912 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.727819    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-48992b-29737410-mz27d" event={"ID":"e08986c7-a429-460f-bb25-6fb4730ac349","Type":"ContainerStarted","Data":"432f45f772336c1aad60dc77e003d09362af29b70e6ad3bdddaaa78c3151145d"}
Jul 16 23:30:00.741372 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.741344    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d"]
Jul 16 23:30:00.743692 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.743552    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-48992b-29737410-mz27d" podStartSLOduration=0.743535202 podStartE2EDuration="743.535202ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:30:00.740498771 +0000 UTC m=+3411.488982835" watchObservedRunningTime="2026-07-16 23:30:00.743535202 +0000 UTC m=+3411.492019272"
Jul 16 23:30:00.868671 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.868643    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d"
Jul 16 23:30:00.979404 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.979310    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a068194-55ac-4b35-9432-d9bbdeb97602-maas-api-service-ca\") pod \"1a068194-55ac-4b35-9432-d9bbdeb97602\" (UID: \"1a068194-55ac-4b35-9432-d9bbdeb97602\") "
Jul 16 23:30:00.979678 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:00.979655    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a068194-55ac-4b35-9432-d9bbdeb97602-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1a068194-55ac-4b35-9432-d9bbdeb97602" (UID: "1a068194-55ac-4b35-9432-d9bbdeb97602"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:30:01.080862 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:01.080829    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a068194-55ac-4b35-9432-d9bbdeb97602-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:30:01.733548 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:01.733508    2565 generic.go:356] "Generic (PLEG): container finished" podID="04b29a52-7376-407f-a096-7e2ba4b3101d" containerID="a0fc7ea704d40ce6acfde6406f7de0df20e85c5a6dd2de76a0f53815344368bf" exitCode=0
Jul 16 23:30:01.733973 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:01.733583    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737410-pvxnd" event={"ID":"04b29a52-7376-407f-a096-7e2ba4b3101d","Type":"ContainerDied","Data":"a0fc7ea704d40ce6acfde6406f7de0df20e85c5a6dd2de76a0f53815344368bf"}
Jul 16 23:30:01.735129 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:01.735111    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d"
Jul 16 23:30:01.735237 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:01.735140    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d" event={"ID":"1a068194-55ac-4b35-9432-d9bbdeb97602","Type":"ContainerDied","Data":"91890dc74cf1e5deb318b956c6b602170d786e9a14791d5abf9def68eb4117d0"}
Jul 16 23:30:01.735237 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:01.735197    2565 scope.go:117] "RemoveContainer" containerID="417d2ba492e36dbf154b3e413f06f222412cbbc30eb66ff502a4f4206c57a303"
Jul 16 23:30:01.736869 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:01.736843    2565 generic.go:356] "Generic (PLEG): container finished" podID="7493e669-3670-452b-9a49-54ff0dd8546c" containerID="e2a0a7abba43787d41156eeea89f8336b49dc88965fa1471ed5ef4b3e30367eb" exitCode=0
Jul 16 23:30:01.736959 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:01.736934    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b4ab52-29737410-5g4pk" event={"ID":"7493e669-3670-452b-9a49-54ff0dd8546c","Type":"ContainerDied","Data":"e2a0a7abba43787d41156eeea89f8336b49dc88965fa1471ed5ef4b3e30367eb"}
Jul 16 23:30:01.738747 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:01.738727    2565 generic.go:356] "Generic (PLEG): container finished" podID="e08986c7-a429-460f-bb25-6fb4730ac349" containerID="4f80bb8093a363b90bec46eba042cc37526c4bac1e7333a0c83d0198b7f5d176" exitCode=0
Jul 16 23:30:01.738860 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:01.738767    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-48992b-29737410-mz27d" event={"ID":"e08986c7-a429-460f-bb25-6fb4730ac349","Type":"ContainerDied","Data":"4f80bb8093a363b90bec46eba042cc37526c4bac1e7333a0c83d0198b7f5d176"}
Jul 16 23:30:01.784913 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:01.784825    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d"]
Jul 16 23:30:01.786899 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:01.786876    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-lgz6d"]
Jul 16 23:30:02.925723 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:02.925685    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737410-pvxnd"
Jul 16 23:30:02.927901 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:02.927877    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b4ab52-29737410-5g4pk"
Jul 16 23:30:02.958945 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:02.958913    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-48992b-29737410-mz27d"
Jul 16 23:30:03.727155 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:03.727121    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a068194-55ac-4b35-9432-d9bbdeb97602" path="/var/lib/kubelet/pods/1a068194-55ac-4b35-9432-d9bbdeb97602/volumes"
Jul 16 23:30:03.751461 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:03.751434    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737410-pvxnd"
Jul 16 23:30:03.751623 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:03.751472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737410-pvxnd" event={"ID":"04b29a52-7376-407f-a096-7e2ba4b3101d","Type":"ContainerDied","Data":"6eae1a1137412223f8064e26429ae70ef58f064774217d99fc3db51179f390c4"}
Jul 16 23:30:03.751623 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:03.751555    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eae1a1137412223f8064e26429ae70ef58f064774217d99fc3db51179f390c4"
Jul 16 23:30:03.753051 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:03.753023    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b4ab52-29737410-5g4pk"
Jul 16 23:30:03.753179 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:03.753070    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b4ab52-29737410-5g4pk" event={"ID":"7493e669-3670-452b-9a49-54ff0dd8546c","Type":"ContainerDied","Data":"22fe8a6607b5d0e7c5edbd30b4fc3e5bcf62716d23dbeef742097b33b22929d8"}
Jul 16 23:30:03.753179 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:03.753111    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22fe8a6607b5d0e7c5edbd30b4fc3e5bcf62716d23dbeef742097b33b22929d8"
Jul 16 23:30:03.754591 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:03.754571    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-48992b-29737410-mz27d" event={"ID":"e08986c7-a429-460f-bb25-6fb4730ac349","Type":"ContainerDied","Data":"432f45f772336c1aad60dc77e003d09362af29b70e6ad3bdddaaa78c3151145d"}
Jul 16 23:30:03.754710 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:03.754609    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="432f45f772336c1aad60dc77e003d09362af29b70e6ad3bdddaaa78c3151145d"
Jul 16 23:30:03.754710 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:03.754612    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-48992b-29737410-mz27d"
Jul 16 23:30:03.980549 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:03.980453    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737365-wbzqw"]
Jul 16 23:30:03.984482 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:03.984452    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737365-wbzqw"]
Jul 16 23:30:05.729044 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:05.729011    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1aa0e44-2b47-4ef3-9010-c32b4a31a612" path="/var/lib/kubelet/pods/a1aa0e44-2b47-4ef3-9010-c32b4a31a612/volumes"
Jul 16 23:30:06.721221 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:06.721194    2565 scope.go:117] "RemoveContainer" containerID="45db8198fd143d8f063befcb5cfa30118fdf2de19b33353a51b059f1c46b6b26"
Jul 16 23:30:07.777202 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:07.777114    2565 generic.go:356] "Generic (PLEG): container finished" podID="cd7294ab-8657-4194-a9c5-ba1ff2da1734" containerID="6555a1a51305d034804388b2753d1dbdeae3758acb67b481a1ab8382d4b3927b" exitCode=6
Jul 16 23:30:07.777202 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:07.777190    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll" event={"ID":"cd7294ab-8657-4194-a9c5-ba1ff2da1734","Type":"ContainerDied","Data":"6555a1a51305d034804388b2753d1dbdeae3758acb67b481a1ab8382d4b3927b"}
Jul 16 23:30:07.777709 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:07.777264    2565 scope.go:117] "RemoveContainer" containerID="45db8198fd143d8f063befcb5cfa30118fdf2de19b33353a51b059f1c46b6b26"
Jul 16 23:30:07.777709 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:07.777560    2565 scope.go:117] "RemoveContainer" containerID="6555a1a51305d034804388b2753d1dbdeae3758acb67b481a1ab8382d4b3927b"
Jul 16 23:30:07.777814 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:30:07.777787    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll_odh-ai-gateway-infra(cd7294ab-8657-4194-a9c5-ba1ff2da1734)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll" podUID="cd7294ab-8657-4194-a9c5-ba1ff2da1734"
Jul 16 23:30:08.801342 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:08.801309    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll"]
Jul 16 23:30:08.941677 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:08.941651    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll"
Jul 16 23:30:09.062083 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:09.061986    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7294ab-8657-4194-a9c5-ba1ff2da1734-maas-api-service-ca\") pod \"cd7294ab-8657-4194-a9c5-ba1ff2da1734\" (UID: \"cd7294ab-8657-4194-a9c5-ba1ff2da1734\") "
Jul 16 23:30:09.062391 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:09.062332    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7294ab-8657-4194-a9c5-ba1ff2da1734-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cd7294ab-8657-4194-a9c5-ba1ff2da1734" (UID: "cd7294ab-8657-4194-a9c5-ba1ff2da1734"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:30:09.163639 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:09.163605    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7294ab-8657-4194-a9c5-ba1ff2da1734-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:30:09.790433 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:09.790400    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll"
Jul 16 23:30:09.790433 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:09.790423    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll" event={"ID":"cd7294ab-8657-4194-a9c5-ba1ff2da1734","Type":"ContainerDied","Data":"ceb6e9dd4c2a007f4d33a0fdd4c6a3da2613827b6191e586b35681c1c4e4065d"}
Jul 16 23:30:09.790665 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:09.790480    2565 scope.go:117] "RemoveContainer" containerID="6555a1a51305d034804388b2753d1dbdeae3758acb67b481a1ab8382d4b3927b"
Jul 16 23:30:09.805747 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:09.805720    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll"]
Jul 16 23:30:09.811677 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:09.811654    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-72xll"]
Jul 16 23:30:10.255362 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:10.255330    2565 scope.go:117] "RemoveContainer" containerID="fd71e2585dcf862f6bb9c1e5651508a8b02517718d97f25bb28f9b202229a111"
Jul 16 23:30:11.726436 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:11.726400    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd7294ab-8657-4194-a9c5-ba1ff2da1734" path="/var/lib/kubelet/pods/cd7294ab-8657-4194-a9c5-ba1ff2da1734/volumes"
Jul 16 23:30:17.728053 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:17.728025    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"]
Jul 16 23:30:17.728534 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:17.728199    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv" podUID="3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f" containerName="maas-api" containerID="cri-o://fa5e157d2a337b735c1b6b79a2691a3f2b7aaa6dec660a6e8d105521f6ef26a6" gracePeriod=30
Jul 16 23:30:17.754393 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:17.754359    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-48992b-29737410-mz27d"]
Jul 16 23:30:17.757211 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:17.757187    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-48992b-29737410-mz27d"]
Jul 16 23:30:18.178074 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.178047    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"
Jul 16 23:30:18.253352 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.253320    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pk6x\" (UniqueName: \"kubernetes.io/projected/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f-kube-api-access-7pk6x\") pod \"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f\" (UID: \"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f\") "
Jul 16 23:30:18.253522 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.253362    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f-openshift-service-ca\") pod \"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f\" (UID: \"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f\") "
Jul 16 23:30:18.253522 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.253393    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f-maas-api-tls\") pod \"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f\" (UID: \"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f\") "
Jul 16 23:30:18.253773 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.253747    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f" (UID: "3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:30:18.255540 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.255517    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f-kube-api-access-7pk6x" (OuterVolumeSpecName: "kube-api-access-7pk6x") pod "3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f" (UID: "3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f"). InnerVolumeSpecName "kube-api-access-7pk6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:30:18.255618 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.255566    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f" (UID: "3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:30:18.354573 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.354533    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pk6x\" (UniqueName: \"kubernetes.io/projected/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f-kube-api-access-7pk6x\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:30:18.354573 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.354567    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:30:18.354573 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.354577    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:30:18.836551 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.836508    2565 generic.go:356] "Generic (PLEG): container finished" podID="3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f" containerID="fa5e157d2a337b735c1b6b79a2691a3f2b7aaa6dec660a6e8d105521f6ef26a6" exitCode=0
Jul 16 23:30:18.836979 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.836580    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"
Jul 16 23:30:18.836979 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.836597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv" event={"ID":"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f","Type":"ContainerDied","Data":"fa5e157d2a337b735c1b6b79a2691a3f2b7aaa6dec660a6e8d105521f6ef26a6"}
Jul 16 23:30:18.836979 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.836664    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv" event={"ID":"3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f","Type":"ContainerDied","Data":"b6e304174a9bf4ed0a5393fc20f3726b88583bc64fd61068d52a668e3a4c30cd"}
Jul 16 23:30:18.836979 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.836687    2565 scope.go:117] "RemoveContainer" containerID="fa5e157d2a337b735c1b6b79a2691a3f2b7aaa6dec660a6e8d105521f6ef26a6"
Jul 16 23:30:18.846907 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.846888    2565 scope.go:117] "RemoveContainer" containerID="fa5e157d2a337b735c1b6b79a2691a3f2b7aaa6dec660a6e8d105521f6ef26a6"
Jul 16 23:30:18.847201 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:30:18.847179    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa5e157d2a337b735c1b6b79a2691a3f2b7aaa6dec660a6e8d105521f6ef26a6\": container with ID starting with fa5e157d2a337b735c1b6b79a2691a3f2b7aaa6dec660a6e8d105521f6ef26a6 not found: ID does not exist" containerID="fa5e157d2a337b735c1b6b79a2691a3f2b7aaa6dec660a6e8d105521f6ef26a6"
Jul 16 23:30:18.847295 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.847212    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa5e157d2a337b735c1b6b79a2691a3f2b7aaa6dec660a6e8d105521f6ef26a6"} err="failed to get container status \"fa5e157d2a337b735c1b6b79a2691a3f2b7aaa6dec660a6e8d105521f6ef26a6\": rpc error: code = NotFound desc = could not find container \"fa5e157d2a337b735c1b6b79a2691a3f2b7aaa6dec660a6e8d105521f6ef26a6\": container with ID starting with fa5e157d2a337b735c1b6b79a2691a3f2b7aaa6dec660a6e8d105521f6ef26a6 not found: ID does not exist"
Jul 16 23:30:18.858378 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.858350    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"]
Jul 16 23:30:18.861801 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:18.861777    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-48992b-54857bf6d6-b99nv"]
Jul 16 23:30:19.726702 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:19.726668    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f" path="/var/lib/kubelet/pods/3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f/volumes"
Jul 16 23:30:19.726974 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:19.726961    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e08986c7-a429-460f-bb25-6fb4730ac349" path="/var/lib/kubelet/pods/e08986c7-a429-460f-bb25-6fb4730ac349/volumes"
Jul 16 23:30:33.124524 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:33.124482    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"]
Jul 16 23:30:33.125067 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:33.124708    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g" podUID="eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" containerName="istio-proxy" containerID="cri-o://2e7de02366795aa28c509ea0edecb73122f0ed201ae0c363c83c012ad149a47a" gracePeriod=30
Jul 16 23:30:38.384064 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.384038    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:30:38.441694 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.441615    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-token\") pod \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") "
Jul 16 23:30:38.441694 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.441683    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-credential-socket\") pod \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") "
Jul 16 23:30:38.441946 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.441703    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istiod-ca-cert\") pod \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") "
Jul 16 23:30:38.441946 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.441722    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-envoy\") pod \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") "
Jul 16 23:30:38.441946 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.441740    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-workload-certs\") pod \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") "
Jul 16 23:30:38.441946 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.441849    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6dr4\" (UniqueName: \"kubernetes.io/projected/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-kube-api-access-c6dr4\") pod \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") "
Jul 16 23:30:38.441946 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.441927    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-podinfo\") pod \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") "
Jul 16 23:30:38.442216 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.441988    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-data\") pod \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") "
Jul 16 23:30:38.442216 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.442016    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-workload-socket\") pod \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\" (UID: \"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b\") "
Jul 16 23:30:38.442216 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.442105    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" (UID: "eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:30:38.442216 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.442116    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" (UID: "eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:30:38.442216 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.442200    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" (UID: "eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:30:38.442533 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.442319    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" (UID: "eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:30:38.442533 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.442501    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-workload-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:30:38.442533 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.442518    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-credential-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:30:38.442533 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.442532    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istiod-ca-cert\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:30:38.442746 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.442545    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-workload-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:30:38.442800 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.442776    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-data" (OuterVolumeSpecName: "istio-data") pod "eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" (UID: "eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:30:38.444157 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.444127    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-token" (OuterVolumeSpecName: "istio-token") pod "eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" (UID: "eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:30:38.444356 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.444154    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" (UID: "eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 23:30:38.444436 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.444365    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-kube-api-access-c6dr4" (OuterVolumeSpecName: "kube-api-access-c6dr4") pod "eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" (UID: "eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b"). InnerVolumeSpecName "kube-api-access-c6dr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:30:38.444436 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.444421    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" (UID: "eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:30:38.543668 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.543636    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-data\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:30:38.543668 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.543666    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-token\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:30:38.543668 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.543674    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-envoy\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:30:38.543897 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.543683    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6dr4\" (UniqueName: \"kubernetes.io/projected/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-kube-api-access-c6dr4\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:30:38.543897 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.543693    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b-istio-podinfo\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:30:38.937568 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.937536    2565 generic.go:356] "Generic (PLEG): container finished" podID="eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" containerID="2e7de02366795aa28c509ea0edecb73122f0ed201ae0c363c83c012ad149a47a" exitCode=0
Jul 16 23:30:38.937744 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.937582    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g" event={"ID":"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b","Type":"ContainerDied","Data":"2e7de02366795aa28c509ea0edecb73122f0ed201ae0c363c83c012ad149a47a"}
Jul 16 23:30:38.937744 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.937621    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g" event={"ID":"eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b","Type":"ContainerDied","Data":"1a42b7e2643c5e1803aaab47b846a28b001813fb37668a7665eafaee48acaf57"}
Jul 16 23:30:38.937744 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.937642    2565 scope.go:117] "RemoveContainer" containerID="2e7de02366795aa28c509ea0edecb73122f0ed201ae0c363c83c012ad149a47a"
Jul 16 23:30:38.937744 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.937582    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"
Jul 16 23:30:38.947769 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.947751    2565 scope.go:117] "RemoveContainer" containerID="38d85c9eb45021204d97762633ae9f8cd6d51ed7ca17395fd40816a9fba3edfb"
Jul 16 23:30:38.955584 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.955569    2565 scope.go:117] "RemoveContainer" containerID="2e7de02366795aa28c509ea0edecb73122f0ed201ae0c363c83c012ad149a47a"
Jul 16 23:30:38.955844 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:30:38.955819    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7de02366795aa28c509ea0edecb73122f0ed201ae0c363c83c012ad149a47a\": container with ID starting with 2e7de02366795aa28c509ea0edecb73122f0ed201ae0c363c83c012ad149a47a not found: ID does not exist" containerID="2e7de02366795aa28c509ea0edecb73122f0ed201ae0c363c83c012ad149a47a"
Jul 16 23:30:38.955894 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.955857    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7de02366795aa28c509ea0edecb73122f0ed201ae0c363c83c012ad149a47a"} err="failed to get container status \"2e7de02366795aa28c509ea0edecb73122f0ed201ae0c363c83c012ad149a47a\": rpc error: code = NotFound desc = could not find container \"2e7de02366795aa28c509ea0edecb73122f0ed201ae0c363c83c012ad149a47a\": container with ID starting with 2e7de02366795aa28c509ea0edecb73122f0ed201ae0c363c83c012ad149a47a not found: ID does not exist"
Jul 16 23:30:38.955894 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.955881    2565 scope.go:117] "RemoveContainer" containerID="38d85c9eb45021204d97762633ae9f8cd6d51ed7ca17395fd40816a9fba3edfb"
Jul 16 23:30:38.956148 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:30:38.956128    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d85c9eb45021204d97762633ae9f8cd6d51ed7ca17395fd40816a9fba3edfb\": container with ID starting with 38d85c9eb45021204d97762633ae9f8cd6d51ed7ca17395fd40816a9fba3edfb not found: ID does not exist" containerID="38d85c9eb45021204d97762633ae9f8cd6d51ed7ca17395fd40816a9fba3edfb"
Jul 16 23:30:38.956271 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.956150    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d85c9eb45021204d97762633ae9f8cd6d51ed7ca17395fd40816a9fba3edfb"} err="failed to get container status \"38d85c9eb45021204d97762633ae9f8cd6d51ed7ca17395fd40816a9fba3edfb\": rpc error: code = NotFound desc = could not find container \"38d85c9eb45021204d97762633ae9f8cd6d51ed7ca17395fd40816a9fba3edfb\": container with ID starting with 38d85c9eb45021204d97762633ae9f8cd6d51ed7ca17395fd40816a9fba3edfb not found: ID does not exist"
Jul 16 23:30:38.959023 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.958998    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"]
Jul 16 23:30:38.962955 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:38.962936    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-48992b-openshift-default-679dfb5f9f-td66g"]
Jul 16 23:30:39.726313 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:39.726280    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" path="/var/lib/kubelet/pods/eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b/volumes"
Jul 16 23:30:43.877469 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:43.877437    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"]
Jul 16 23:30:43.877922 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:43.877648    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt" podUID="a4975fd1-65d7-45e2-8f0b-c940c23a5606" containerName="maas-api" containerID="cri-o://7ba1657590893e4bb7461b619ae8cf4006a6ed3d29dfcbece7f0b1c03838eb7a" gracePeriod=30
Jul 16 23:30:43.907481 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:43.907456    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b4ab52-29737410-5g4pk"]
Jul 16 23:30:43.910564 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:43.910540    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b4ab52-29737410-5g4pk"]
Jul 16 23:30:44.128734 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.128668    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"
Jul 16 23:30:44.196206 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.196163    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gslq\" (UniqueName: \"kubernetes.io/projected/a4975fd1-65d7-45e2-8f0b-c940c23a5606-kube-api-access-8gslq\") pod \"a4975fd1-65d7-45e2-8f0b-c940c23a5606\" (UID: \"a4975fd1-65d7-45e2-8f0b-c940c23a5606\") "
Jul 16 23:30:44.196206 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.196218    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4975fd1-65d7-45e2-8f0b-c940c23a5606-maas-api-tls\") pod \"a4975fd1-65d7-45e2-8f0b-c940c23a5606\" (UID: \"a4975fd1-65d7-45e2-8f0b-c940c23a5606\") "
Jul 16 23:30:44.196492 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.196342    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4975fd1-65d7-45e2-8f0b-c940c23a5606-openshift-service-ca\") pod \"a4975fd1-65d7-45e2-8f0b-c940c23a5606\" (UID: \"a4975fd1-65d7-45e2-8f0b-c940c23a5606\") "
Jul 16 23:30:44.196735 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.196710    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4975fd1-65d7-45e2-8f0b-c940c23a5606-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4975fd1-65d7-45e2-8f0b-c940c23a5606" (UID: "a4975fd1-65d7-45e2-8f0b-c940c23a5606"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:30:44.198399 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.198378    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4975fd1-65d7-45e2-8f0b-c940c23a5606-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4975fd1-65d7-45e2-8f0b-c940c23a5606" (UID: "a4975fd1-65d7-45e2-8f0b-c940c23a5606"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:30:44.198468 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.198451    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4975fd1-65d7-45e2-8f0b-c940c23a5606-kube-api-access-8gslq" (OuterVolumeSpecName: "kube-api-access-8gslq") pod "a4975fd1-65d7-45e2-8f0b-c940c23a5606" (UID: "a4975fd1-65d7-45e2-8f0b-c940c23a5606"). InnerVolumeSpecName "kube-api-access-8gslq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:30:44.297128 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.297088    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4975fd1-65d7-45e2-8f0b-c940c23a5606-openshift-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:30:44.297128 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.297122    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gslq\" (UniqueName: \"kubernetes.io/projected/a4975fd1-65d7-45e2-8f0b-c940c23a5606-kube-api-access-8gslq\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:30:44.297364 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.297153    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4975fd1-65d7-45e2-8f0b-c940c23a5606-maas-api-tls\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:30:44.968727 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.968691    2565 generic.go:356] "Generic (PLEG): container finished" podID="a4975fd1-65d7-45e2-8f0b-c940c23a5606" containerID="7ba1657590893e4bb7461b619ae8cf4006a6ed3d29dfcbece7f0b1c03838eb7a" exitCode=0
Jul 16 23:30:44.969139 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.968733    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"
Jul 16 23:30:44.969139 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.968765    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt" event={"ID":"a4975fd1-65d7-45e2-8f0b-c940c23a5606","Type":"ContainerDied","Data":"7ba1657590893e4bb7461b619ae8cf4006a6ed3d29dfcbece7f0b1c03838eb7a"}
Jul 16 23:30:44.969139 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.968809    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt" event={"ID":"a4975fd1-65d7-45e2-8f0b-c940c23a5606","Type":"ContainerDied","Data":"d2040c2d9d9dd5e23305e831e228d659b0c8ac416c9ed5c1a5a9b712ad4e9c67"}
Jul 16 23:30:44.969139 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.968829    2565 scope.go:117] "RemoveContainer" containerID="7ba1657590893e4bb7461b619ae8cf4006a6ed3d29dfcbece7f0b1c03838eb7a"
Jul 16 23:30:44.977991 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.977971    2565 scope.go:117] "RemoveContainer" containerID="7ba1657590893e4bb7461b619ae8cf4006a6ed3d29dfcbece7f0b1c03838eb7a"
Jul 16 23:30:44.978229 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:30:44.978213    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba1657590893e4bb7461b619ae8cf4006a6ed3d29dfcbece7f0b1c03838eb7a\": container with ID starting with 7ba1657590893e4bb7461b619ae8cf4006a6ed3d29dfcbece7f0b1c03838eb7a not found: ID does not exist" containerID="7ba1657590893e4bb7461b619ae8cf4006a6ed3d29dfcbece7f0b1c03838eb7a"
Jul 16 23:30:44.978290 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.978237    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba1657590893e4bb7461b619ae8cf4006a6ed3d29dfcbece7f0b1c03838eb7a"} err="failed to get container status \"7ba1657590893e4bb7461b619ae8cf4006a6ed3d29dfcbece7f0b1c03838eb7a\": rpc error: code = NotFound desc = could not find container \"7ba1657590893e4bb7461b619ae8cf4006a6ed3d29dfcbece7f0b1c03838eb7a\": container with ID starting with 7ba1657590893e4bb7461b619ae8cf4006a6ed3d29dfcbece7f0b1c03838eb7a not found: ID does not exist"
Jul 16 23:30:44.989560 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.989537    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"]
Jul 16 23:30:44.993339 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:44.993317    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b4ab52-64dc646dcc-5pjzt"]
Jul 16 23:30:45.726838 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:45.726807    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7493e669-3670-452b-9a49-54ff0dd8546c" path="/var/lib/kubelet/pods/7493e669-3670-452b-9a49-54ff0dd8546c/volumes"
Jul 16 23:30:45.727184 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:45.727171    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4975fd1-65d7-45e2-8f0b-c940c23a5606" path="/var/lib/kubelet/pods/a4975fd1-65d7-45e2-8f0b-c940c23a5606/volumes"
Jul 16 23:30:59.223331 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:59.223288    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"]
Jul 16 23:30:59.223847 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:30:59.223547    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx" podUID="a8b1b47a-d704-46ed-b3a3-ec9f324d4224" containerName="istio-proxy" containerID="cri-o://487bd85812fc614334d6de3076babdfef00026b4ac7e0355910a15921365652e" gracePeriod=30
Jul 16 23:31:03.988938 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:03.988895    2565 patch_prober.go:28] interesting pod/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 23:31:03.989432 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:03.988958    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx" podUID="a8b1b47a-d704-46ed-b3a3-ec9f324d4224" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 23:31:04.487828 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.487797    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:31:04.584646 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.584606    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-podinfo\") pod \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") "
Jul 16 23:31:04.584846 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.584661    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-workload-socket\") pod \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") "
Jul 16 23:31:04.584846 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.584681    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-credential-socket\") pod \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") "
Jul 16 23:31:04.584846 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.584711    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-token\") pod \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") "
Jul 16 23:31:04.584846 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.584734    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-data\") pod \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") "
Jul 16 23:31:04.584846 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.584833    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-envoy\") pod \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") "
Jul 16 23:31:04.585123 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.584867    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istiod-ca-cert\") pod \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") "
Jul 16 23:31:04.585123 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.584930    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a8b1b47a-d704-46ed-b3a3-ec9f324d4224" (UID: "a8b1b47a-d704-46ed-b3a3-ec9f324d4224"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:31:04.585123 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.584941    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-workload-certs\") pod \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") "
Jul 16 23:31:04.585123 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.585036    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2fk5\" (UniqueName: \"kubernetes.io/projected/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-kube-api-access-t2fk5\") pod \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\" (UID: \"a8b1b47a-d704-46ed-b3a3-ec9f324d4224\") "
Jul 16 23:31:04.585123 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.585039    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a8b1b47a-d704-46ed-b3a3-ec9f324d4224" (UID: "a8b1b47a-d704-46ed-b3a3-ec9f324d4224"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:31:04.585430 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.585226    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a8b1b47a-d704-46ed-b3a3-ec9f324d4224" (UID: "a8b1b47a-d704-46ed-b3a3-ec9f324d4224"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:31:04.585430 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.585347    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a8b1b47a-d704-46ed-b3a3-ec9f324d4224" (UID: "a8b1b47a-d704-46ed-b3a3-ec9f324d4224"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:31:04.585520 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.585498    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-workload-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:31:04.585565 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.585517    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-credential-socket\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:31:04.585565 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.585530    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istiod-ca-cert\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:31:04.585565 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.585542    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-workload-certs\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:31:04.585704 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.585675    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-data" (OuterVolumeSpecName: "istio-data") pod "a8b1b47a-d704-46ed-b3a3-ec9f324d4224" (UID: "a8b1b47a-d704-46ed-b3a3-ec9f324d4224"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:31:04.587120 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.587091    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-kube-api-access-t2fk5" (OuterVolumeSpecName: "kube-api-access-t2fk5") pod "a8b1b47a-d704-46ed-b3a3-ec9f324d4224" (UID: "a8b1b47a-d704-46ed-b3a3-ec9f324d4224"). InnerVolumeSpecName "kube-api-access-t2fk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:31:04.587120 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.587104    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a8b1b47a-d704-46ed-b3a3-ec9f324d4224" (UID: "a8b1b47a-d704-46ed-b3a3-ec9f324d4224"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:31:04.587343 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.587319    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a8b1b47a-d704-46ed-b3a3-ec9f324d4224" (UID: "a8b1b47a-d704-46ed-b3a3-ec9f324d4224"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 23:31:04.587419 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.587399    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-token" (OuterVolumeSpecName: "istio-token") pod "a8b1b47a-d704-46ed-b3a3-ec9f324d4224" (UID: "a8b1b47a-d704-46ed-b3a3-ec9f324d4224"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:31:04.686390 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.686351    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2fk5\" (UniqueName: \"kubernetes.io/projected/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-kube-api-access-t2fk5\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:31:04.686390 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.686383    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-podinfo\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:31:04.686390 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.686393    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-token\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:31:04.686390 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.686402    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-data\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:31:04.686672 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:04.686409    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b1b47a-d704-46ed-b3a3-ec9f324d4224-istio-envoy\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:31:05.069384 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:05.069290    2565 generic.go:356] "Generic (PLEG): container finished" podID="a8b1b47a-d704-46ed-b3a3-ec9f324d4224" containerID="487bd85812fc614334d6de3076babdfef00026b4ac7e0355910a15921365652e" exitCode=0
Jul 16 23:31:05.069384 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:05.069343    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"
Jul 16 23:31:05.069384 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:05.069373    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx" event={"ID":"a8b1b47a-d704-46ed-b3a3-ec9f324d4224","Type":"ContainerDied","Data":"487bd85812fc614334d6de3076babdfef00026b4ac7e0355910a15921365652e"}
Jul 16 23:31:05.069871 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:05.069424    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx" event={"ID":"a8b1b47a-d704-46ed-b3a3-ec9f324d4224","Type":"ContainerDied","Data":"73726d7b8d08bcc43fd7dd8bda867712ba5f224f80fa3ede94028333a1459561"}
Jul 16 23:31:05.069871 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:05.069448    2565 scope.go:117] "RemoveContainer" containerID="487bd85812fc614334d6de3076babdfef00026b4ac7e0355910a15921365652e"
Jul 16 23:31:05.078964 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:05.078944    2565 scope.go:117] "RemoveContainer" containerID="546ed43b169b22a06668b9026c50592408775c5de9874c4ad61a954f7f196b94"
Jul 16 23:31:05.086865 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:05.086846    2565 scope.go:117] "RemoveContainer" containerID="487bd85812fc614334d6de3076babdfef00026b4ac7e0355910a15921365652e"
Jul 16 23:31:05.087118 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:31:05.087100    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"487bd85812fc614334d6de3076babdfef00026b4ac7e0355910a15921365652e\": container with ID starting with 487bd85812fc614334d6de3076babdfef00026b4ac7e0355910a15921365652e not found: ID does not exist" containerID="487bd85812fc614334d6de3076babdfef00026b4ac7e0355910a15921365652e"
Jul 16 23:31:05.087162 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:05.087126    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"487bd85812fc614334d6de3076babdfef00026b4ac7e0355910a15921365652e"} err="failed to get container status \"487bd85812fc614334d6de3076babdfef00026b4ac7e0355910a15921365652e\": rpc error: code = NotFound desc = could not find container \"487bd85812fc614334d6de3076babdfef00026b4ac7e0355910a15921365652e\": container with ID starting with 487bd85812fc614334d6de3076babdfef00026b4ac7e0355910a15921365652e not found: ID does not exist"
Jul 16 23:31:05.087162 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:05.087145    2565 scope.go:117] "RemoveContainer" containerID="546ed43b169b22a06668b9026c50592408775c5de9874c4ad61a954f7f196b94"
Jul 16 23:31:05.087410 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:31:05.087391    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546ed43b169b22a06668b9026c50592408775c5de9874c4ad61a954f7f196b94\": container with ID starting with 546ed43b169b22a06668b9026c50592408775c5de9874c4ad61a954f7f196b94 not found: ID does not exist" containerID="546ed43b169b22a06668b9026c50592408775c5de9874c4ad61a954f7f196b94"
Jul 16 23:31:05.087454 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:05.087416    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546ed43b169b22a06668b9026c50592408775c5de9874c4ad61a954f7f196b94"} err="failed to get container status \"546ed43b169b22a06668b9026c50592408775c5de9874c4ad61a954f7f196b94\": rpc error: code = NotFound desc = could not find container \"546ed43b169b22a06668b9026c50592408775c5de9874c4ad61a954f7f196b94\": container with ID starting with 546ed43b169b22a06668b9026c50592408775c5de9874c4ad61a954f7f196b94 not found: ID does not exist"
Jul 16 23:31:05.090959 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:05.090937    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"]
Jul 16 23:31:05.095049 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:05.095028    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-b4ab52-openshift-default-cc95bb967-dpmhx"]
Jul 16 23:31:05.604206 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:05.604181    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv_32bfa590-021b-49b5-a94a-10661dd15166/copy-manifests/0.log"
Jul 16 23:31:05.616854 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:05.616828    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv_32bfa590-021b-49b5-a94a-10661dd15166/manager/0.log"
Jul 16 23:31:05.726544 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:05.726511    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8b1b47a-d704-46ed-b3a3-ec9f324d4224" path="/var/lib/kubelet/pods/a8b1b47a-d704-46ed-b3a3-ec9f324d4224/volumes"
Jul 16 23:31:06.200779 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:06.200750    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d6d9dd47b-hznll_53925486-8958-4f47-8274-1afd2e6cfce4/maas-api/0.log"
Jul 16 23:31:06.309558 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:06.309527    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737380-d66wg_1c880144-9643-4922-bd49-8a30c9f98b8c/cleanup/0.log"
Jul 16 23:31:06.522150 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:06.522071    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737410-pvxnd_04b29a52-7376-407f-a096-7e2ba4b3101d/cleanup/0.log"
Jul 16 23:31:06.836180 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:06.836146    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-8a2515-7d2hx_60692b60-a957-4bb8-9b92-737f7afd01f6/revoke-keys/0.log"
Jul 16 23:31:07.151442 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:07.151362    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-9752734e-js9t8_25aa5168-250d-4b3e-9de8-72282b82a95a/revoke-keys/0.log"
Jul 16 23:31:09.829390 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:09.829360    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4n4xlt_62707e5b-445c-45ac-8d6a-3ef554d6e4b7/istio-proxy/0.log"
Jul 16 23:31:09.936104 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:09.936068    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-rx9kb_4151c7c9-fea2-41d0-93bf-a23d35a1e835/discovery/0.log"
Jul 16 23:31:10.265916 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:10.265840    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-l7jvp_adee9cd9-4926-40e7-a094-7cd9b70591cd/istio-proxy/1.log"
Jul 16 23:31:10.380572 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:10.380541    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-gqxs8_8c7638b3-99dd-4860-9078-26e471cb24a6/payload-pre-processing/0.log"
Jul 16 23:31:11.057187 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:11.057152    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-55dt9_017d0692-e8a1-4857-8e2e-0c422706856d/main/0.log"
Jul 16 23:31:11.277625 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:11.277582    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db89m7t_8bb59881-cd5f-4089-925d-6d0af975f6f5/main/0.log"
Jul 16 23:31:19.492331 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:19.492271    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lgwp5_61e408a2-709b-4928-b6b2-018a505d8d4d/global-pull-secret-syncer/0.log"
Jul 16 23:31:19.676754 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:19.676719    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gkmrf_57b0b632-a518-4519-a7f6-3e18e8cabdb4/konnectivity-agent/0.log"
Jul 16 23:31:19.729075 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:19.729044    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-23.ec2.internal_f6e975acdd98c0499b69a0b9bfc9cbb0/haproxy/0.log"
Jul 16 23:31:21.325464 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325430    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76"]
Jul 16 23:31:21.325874 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325811    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a068194-55ac-4b35-9432-d9bbdeb97602" containerName="revoke-keys"
Jul 16 23:31:21.325874 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325824    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a068194-55ac-4b35-9432-d9bbdeb97602" containerName="revoke-keys"
Jul 16 23:31:21.325874 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325835    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd7294ab-8657-4194-a9c5-ba1ff2da1734" containerName="revoke-keys"
Jul 16 23:31:21.325874 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325840    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd7294ab-8657-4194-a9c5-ba1ff2da1734" containerName="revoke-keys"
Jul 16 23:31:21.325874 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325847    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd7294ab-8657-4194-a9c5-ba1ff2da1734" containerName="revoke-keys"
Jul 16 23:31:21.325874 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325854    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd7294ab-8657-4194-a9c5-ba1ff2da1734" containerName="revoke-keys"
Jul 16 23:31:21.325874 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325864    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a068194-55ac-4b35-9432-d9bbdeb97602" containerName="revoke-keys"
Jul 16 23:31:21.325874 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325869    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a068194-55ac-4b35-9432-d9bbdeb97602" containerName="revoke-keys"
Jul 16 23:31:21.325874 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325875    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f" containerName="maas-api"
Jul 16 23:31:21.325874 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325880    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f" containerName="maas-api"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325886    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" containerName="istio-proxy"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325891    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" containerName="istio-proxy"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325897    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd7294ab-8657-4194-a9c5-ba1ff2da1734" containerName="revoke-keys"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325902    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd7294ab-8657-4194-a9c5-ba1ff2da1734" containerName="revoke-keys"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325911    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4975fd1-65d7-45e2-8f0b-c940c23a5606" containerName="maas-api"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325915    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4975fd1-65d7-45e2-8f0b-c940c23a5606" containerName="maas-api"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325921    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" containerName="istio-proxy"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325925    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" containerName="istio-proxy"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325933    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8b1b47a-d704-46ed-b3a3-ec9f324d4224" containerName="istio-proxy"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325938    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8b1b47a-d704-46ed-b3a3-ec9f324d4224" containerName="istio-proxy"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325944    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e08986c7-a429-460f-bb25-6fb4730ac349" containerName="cleanup"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325949    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e08986c7-a429-460f-bb25-6fb4730ac349" containerName="cleanup"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325956    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7493e669-3670-452b-9a49-54ff0dd8546c" containerName="cleanup"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325961    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7493e669-3670-452b-9a49-54ff0dd8546c" containerName="cleanup"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325968    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b29a52-7376-407f-a096-7e2ba4b3101d" containerName="cleanup"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325974    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b29a52-7376-407f-a096-7e2ba4b3101d" containerName="cleanup"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325983    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8b1b47a-d704-46ed-b3a3-ec9f324d4224" containerName="istio-proxy"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.325988    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8b1b47a-d704-46ed-b3a3-ec9f324d4224" containerName="istio-proxy"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.326039    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4975fd1-65d7-45e2-8f0b-c940c23a5606" containerName="maas-api"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.326045    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a068194-55ac-4b35-9432-d9bbdeb97602" containerName="revoke-keys"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.326051    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8b1b47a-d704-46ed-b3a3-ec9f324d4224" containerName="istio-proxy"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.326056    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a068194-55ac-4b35-9432-d9bbdeb97602" containerName="revoke-keys"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.326062    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd7294ab-8657-4194-a9c5-ba1ff2da1734" containerName="revoke-keys"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.326066    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d9049fc-bd3b-404c-8d9a-85dbb9e6ad1f" containerName="maas-api"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.326071    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd7294ab-8657-4194-a9c5-ba1ff2da1734" containerName="revoke-keys"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.326078    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd7294ab-8657-4194-a9c5-ba1ff2da1734" containerName="revoke-keys"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.326083    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" containerName="istio-proxy"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.326089    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a068194-55ac-4b35-9432-d9bbdeb97602" containerName="revoke-keys"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.326095    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb7352f0-4d62-4dd6-aaae-d1fc9a8c740b" containerName="istio-proxy"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.326101    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7493e669-3670-452b-9a49-54ff0dd8546c" containerName="cleanup"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.326107    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e08986c7-a429-460f-bb25-6fb4730ac349" containerName="cleanup"
Jul 16 23:31:21.326191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.326111    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="04b29a52-7376-407f-a096-7e2ba4b3101d" containerName="cleanup"
Jul 16 23:31:21.331154 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.331128    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76"
Jul 16 23:31:21.334054 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.334029    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dfwcd\""
Jul 16 23:31:21.341511 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.341474    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76"]
Jul 16 23:31:21.439465 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.439414    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd6aa26-8816-424d-a708-27ad317042b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76\" (UID: \"7bd6aa26-8816-424d-a708-27ad317042b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76"
Jul 16 23:31:21.540328 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.540277    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd6aa26-8816-424d-a708-27ad317042b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76\" (UID: \"7bd6aa26-8816-424d-a708-27ad317042b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76"
Jul 16 23:31:21.540887 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.540865    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd6aa26-8816-424d-a708-27ad317042b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76\" (UID: \"7bd6aa26-8816-424d-a708-27ad317042b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76"
Jul 16 23:31:21.644382 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.644286    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76"
Jul 16 23:31:21.774794 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:21.774764    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76"]
Jul 16 23:31:21.777566 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:31:21.777539    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd6aa26_8816_424d_a708_27ad317042b9.slice/crio-055be084aff71875da2dea68507a14853dd8fd204df28a4a56ad711e413bdc66 WatchSource:0}: Error finding container 055be084aff71875da2dea68507a14853dd8fd204df28a4a56ad711e413bdc66: Status 404 returned error can't find the container with id 055be084aff71875da2dea68507a14853dd8fd204df28a4a56ad711e413bdc66
Jul 16 23:31:22.164679 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:22.164642    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76" event={"ID":"7bd6aa26-8816-424d-a708-27ad317042b9","Type":"ContainerStarted","Data":"7a08cacc5d8c06b27aad1322ed1f253a55badcb17bb21a650ef8e7acefd3ad8c"}
Jul 16 23:31:22.164893 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:22.164701    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76" event={"ID":"7bd6aa26-8816-424d-a708-27ad317042b9","Type":"ContainerStarted","Data":"055be084aff71875da2dea68507a14853dd8fd204df28a4a56ad711e413bdc66"}
Jul 16 23:31:22.179854 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:22.179781    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76" podStartSLOduration=1.179759937 podStartE2EDuration="1.179759937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:31:22.179355556 +0000 UTC m=+3492.927839625" watchObservedRunningTime="2026-07-16 23:31:22.179759937 +0000 UTC m=+3492.928244009"
Jul 16 23:31:23.172334 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:23.172286    2565 generic.go:356] "Generic (PLEG): container finished" podID="7bd6aa26-8816-424d-a708-27ad317042b9" containerID="7a08cacc5d8c06b27aad1322ed1f253a55badcb17bb21a650ef8e7acefd3ad8c" exitCode=6
Jul 16 23:31:23.172768 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:23.172390    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76" event={"ID":"7bd6aa26-8816-424d-a708-27ad317042b9","Type":"ContainerDied","Data":"7a08cacc5d8c06b27aad1322ed1f253a55badcb17bb21a650ef8e7acefd3ad8c"}
Jul 16 23:31:23.172768 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:23.172558    2565 scope.go:117] "RemoveContainer" containerID="7a08cacc5d8c06b27aad1322ed1f253a55badcb17bb21a650ef8e7acefd3ad8c"
Jul 16 23:31:23.649752 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:23.649716    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9"]
Jul 16 23:31:23.650179 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:23.650159    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a068194-55ac-4b35-9432-d9bbdeb97602" containerName="revoke-keys"
Jul 16 23:31:23.650298 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:23.650182    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a068194-55ac-4b35-9432-d9bbdeb97602" containerName="revoke-keys"
Jul 16 23:31:23.650381 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:23.650297    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8b1b47a-d704-46ed-b3a3-ec9f324d4224" containerName="istio-proxy"
Jul 16 23:31:23.653264 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:23.653225    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9"
Jul 16 23:31:23.655188 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:23.655158    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6918622a-25c0-42c0-b380-41eb71557283-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-5tlk9\" (UID: \"6918622a-25c0-42c0-b380-41eb71557283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9"
Jul 16 23:31:23.661001 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:23.660981    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9"]
Jul 16 23:31:23.756281 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:23.756151    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6918622a-25c0-42c0-b380-41eb71557283-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-5tlk9\" (UID: \"6918622a-25c0-42c0-b380-41eb71557283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9"
Jul 16 23:31:23.756876 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:23.756851    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6918622a-25c0-42c0-b380-41eb71557283-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3933c193-5tlk9\" (UID: \"6918622a-25c0-42c0-b380-41eb71557283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9"
Jul 16 23:31:23.964783 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:23.964742    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9"
Jul 16 23:31:24.102402 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:24.102378    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9"]
Jul 16 23:31:24.104853 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:31:24.104827    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6918622a_25c0_42c0_b380_41eb71557283.slice/crio-c0128ab60e9be52616500eefddfb228d65b92b342fdee3949aa6a05a7010a8b4 WatchSource:0}: Error finding container c0128ab60e9be52616500eefddfb228d65b92b342fdee3949aa6a05a7010a8b4: Status 404 returned error can't find the container with id c0128ab60e9be52616500eefddfb228d65b92b342fdee3949aa6a05a7010a8b4
Jul 16 23:31:24.178208 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:24.178173    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9" event={"ID":"6918622a-25c0-42c0-b380-41eb71557283","Type":"ContainerStarted","Data":"c0128ab60e9be52616500eefddfb228d65b92b342fdee3949aa6a05a7010a8b4"}
Jul 16 23:31:24.179954 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:24.179926    2565 generic.go:356] "Generic (PLEG): container finished" podID="7bd6aa26-8816-424d-a708-27ad317042b9" containerID="3b894e4a7bfb22ed080e25859b89a370c2b27b96314a323979922398d918e0fd" exitCode=6
Jul 16 23:31:24.180060 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:24.180012    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76" event={"ID":"7bd6aa26-8816-424d-a708-27ad317042b9","Type":"ContainerDied","Data":"3b894e4a7bfb22ed080e25859b89a370c2b27b96314a323979922398d918e0fd"}
Jul 16 23:31:24.180113 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:24.180060    2565 scope.go:117] "RemoveContainer" containerID="7a08cacc5d8c06b27aad1322ed1f253a55badcb17bb21a650ef8e7acefd3ad8c"
Jul 16 23:31:24.180191 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:24.180172    2565 scope.go:117] "RemoveContainer" containerID="3b894e4a7bfb22ed080e25859b89a370c2b27b96314a323979922398d918e0fd"
Jul 16 23:31:24.180473 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:31:24.180453    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76_odh-ai-gateway-infra(7bd6aa26-8816-424d-a708-27ad317042b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76" podUID="7bd6aa26-8816-424d-a708-27ad317042b9"
Jul 16 23:31:25.185940 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:25.185901    2565 generic.go:356] "Generic (PLEG): container finished" podID="6918622a-25c0-42c0-b380-41eb71557283" containerID="72b7fa780cd29948a8f3c617a5b8bf54eadd37a43a8d90c45d461d6fe5e511a0" exitCode=6
Jul 16 23:31:25.186483 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:25.185990    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9" event={"ID":"6918622a-25c0-42c0-b380-41eb71557283","Type":"ContainerDied","Data":"72b7fa780cd29948a8f3c617a5b8bf54eadd37a43a8d90c45d461d6fe5e511a0"}
Jul 16 23:31:25.186483 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:25.186235    2565 scope.go:117] "RemoveContainer" containerID="72b7fa780cd29948a8f3c617a5b8bf54eadd37a43a8d90c45d461d6fe5e511a0"
Jul 16 23:31:25.187872 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:25.187856    2565 scope.go:117] "RemoveContainer" containerID="3b894e4a7bfb22ed080e25859b89a370c2b27b96314a323979922398d918e0fd"
Jul 16 23:31:25.188046 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:31:25.188029    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76_odh-ai-gateway-infra(7bd6aa26-8816-424d-a708-27ad317042b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76" podUID="7bd6aa26-8816-424d-a708-27ad317042b9"
Jul 16 23:31:26.194173 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:26.194084    2565 generic.go:356] "Generic (PLEG): container finished" podID="6918622a-25c0-42c0-b380-41eb71557283" containerID="37e92b0a14873d17c87875d06675171310e5816aa84100cfec4ca1ccd40a0f75" exitCode=6
Jul 16 23:31:26.194652 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:26.194166    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9" event={"ID":"6918622a-25c0-42c0-b380-41eb71557283","Type":"ContainerDied","Data":"37e92b0a14873d17c87875d06675171310e5816aa84100cfec4ca1ccd40a0f75"}
Jul 16 23:31:26.194652 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:26.194232    2565 scope.go:117] "RemoveContainer" containerID="72b7fa780cd29948a8f3c617a5b8bf54eadd37a43a8d90c45d461d6fe5e511a0"
Jul 16 23:31:26.194652 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:26.194492    2565 scope.go:117] "RemoveContainer" containerID="37e92b0a14873d17c87875d06675171310e5816aa84100cfec4ca1ccd40a0f75"
Jul 16 23:31:26.194819 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:31:26.194728    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-5tlk9_odh-ai-gateway-infra(6918622a-25c0-42c0-b380-41eb71557283)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9" podUID="6918622a-25c0-42c0-b380-41eb71557283"
Jul 16 23:31:27.148081 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.148049    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-44bjw_156c60da-5d5c-4d2e-9d91-05d6e4a31cba/cluster-monitoring-operator/0.log"
Jul 16 23:31:27.201321 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.201290    2565 scope.go:117] "RemoveContainer" containerID="37e92b0a14873d17c87875d06675171310e5816aa84100cfec4ca1ccd40a0f75"
Jul 16 23:31:27.201790 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:31:27.201566    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-5tlk9_odh-ai-gateway-infra(6918622a-25c0-42c0-b380-41eb71557283)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9" podUID="6918622a-25c0-42c0-b380-41eb71557283"
Jul 16 23:31:27.261891 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.261856    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-7d6x7_83289e15-a6aa-453a-bac4-355fdadd007e/monitoring-plugin/0.log"
Jul 16 23:31:27.413353 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.413264    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhpdq_bbb767dc-1b1a-4aad-b781-bdd9dcf75e41/node-exporter/0.log"
Jul 16 23:31:27.438021 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.437992    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhpdq_bbb767dc-1b1a-4aad-b781-bdd9dcf75e41/kube-rbac-proxy/0.log"
Jul 16 23:31:27.462042 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.462009    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mhpdq_bbb767dc-1b1a-4aad-b781-bdd9dcf75e41/init-textfile/0.log"
Jul 16 23:31:27.567841 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.567816    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_911afef5-abb0-4869-b331-aed0f5420d2b/prometheus/0.log"
Jul 16 23:31:27.588081 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.588050    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_911afef5-abb0-4869-b331-aed0f5420d2b/config-reloader/0.log"
Jul 16 23:31:27.609463 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.609438    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_911afef5-abb0-4869-b331-aed0f5420d2b/thanos-sidecar/0.log"
Jul 16 23:31:27.631562 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.631534    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_911afef5-abb0-4869-b331-aed0f5420d2b/kube-rbac-proxy-web/0.log"
Jul 16 23:31:27.653733 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.653705    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_911afef5-abb0-4869-b331-aed0f5420d2b/kube-rbac-proxy/0.log"
Jul 16 23:31:27.675459 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.675370    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_911afef5-abb0-4869-b331-aed0f5420d2b/kube-rbac-proxy-thanos/0.log"
Jul 16 23:31:27.698716 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.698685    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_911afef5-abb0-4869-b331-aed0f5420d2b/init-config-reloader/0.log"
Jul 16 23:31:27.724705 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.724674    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-9dmjv_fb078957-0631-4ff9-af8a-78ab0f668f4d/prometheus-operator/0.log"
Jul 16 23:31:27.740700 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.740666    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-9dmjv_fb078957-0631-4ff9-af8a-78ab0f668f4d/kube-rbac-proxy/0.log"
Jul 16 23:31:27.762634 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.762593    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-r4kck_a17fd63f-8198-4623-a983-6577ad6983f4/prometheus-operator-admission-webhook/0.log"
Jul 16 23:31:27.864544 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.864516    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75c84d948b-hw986_3bf1d80d-3994-4c03-a50a-496478abf16e/thanos-query/0.log"
Jul 16 23:31:27.891011 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.890981    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75c84d948b-hw986_3bf1d80d-3994-4c03-a50a-496478abf16e/kube-rbac-proxy-web/0.log"
Jul 16 23:31:27.911343 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.911314    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75c84d948b-hw986_3bf1d80d-3994-4c03-a50a-496478abf16e/kube-rbac-proxy/0.log"
Jul 16 23:31:27.935399 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.935321    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75c84d948b-hw986_3bf1d80d-3994-4c03-a50a-496478abf16e/prom-label-proxy/0.log"
Jul 16 23:31:27.954603 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.954565    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75c84d948b-hw986_3bf1d80d-3994-4c03-a50a-496478abf16e/kube-rbac-proxy-rules/0.log"
Jul 16 23:31:27.978002 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:27.977972    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75c84d948b-hw986_3bf1d80d-3994-4c03-a50a-496478abf16e/kube-rbac-proxy-metrics/0.log"
Jul 16 23:31:28.576590 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.576556    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-c2btj"]
Jul 16 23:31:28.581409 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.581384    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-c2btj"
Jul 16 23:31:28.588027 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.587998    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-c2btj"]
Jul 16 23:31:28.599742 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.599702    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16eddab7-3c0f-4bfd-ad4e-64dd2f4e9947-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-c2btj\" (UID: \"16eddab7-3c0f-4bfd-ad4e-64dd2f4e9947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-c2btj"
Jul 16 23:31:28.700457 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.700416    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16eddab7-3c0f-4bfd-ad4e-64dd2f4e9947-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-c2btj\" (UID: \"16eddab7-3c0f-4bfd-ad4e-64dd2f4e9947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-c2btj"
Jul 16 23:31:28.701063 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.701042    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16eddab7-3c0f-4bfd-ad4e-64dd2f4e9947-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f263f2f2-c2btj\" (UID: \"16eddab7-3c0f-4bfd-ad4e-64dd2f4e9947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-c2btj"
Jul 16 23:31:28.753969 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.753930    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"]
Jul 16 23:31:28.757553 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.757531    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:28.760096 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.760069    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rvb4k\"/\"kube-root-ca.crt\""
Jul 16 23:31:28.760242 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.760095    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rvb4k\"/\"openshift-service-ca.crt\""
Jul 16 23:31:28.760242 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.760111    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rvb4k\"/\"default-dockercfg-8wz9g\""
Jul 16 23:31:28.767299 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.767239    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"]
Jul 16 23:31:28.801218 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.801169    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f499d9c-c9b1-440d-bc80-ec355433452a-lib-modules\") pod \"perf-node-gather-daemonset-b4l56\" (UID: \"0f499d9c-c9b1-440d-bc80-ec355433452a\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:28.801218 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.801219    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f499d9c-c9b1-440d-bc80-ec355433452a-proc\") pod \"perf-node-gather-daemonset-b4l56\" (UID: \"0f499d9c-c9b1-440d-bc80-ec355433452a\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:28.801483 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.801316    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f499d9c-c9b1-440d-bc80-ec355433452a-podres\") pod \"perf-node-gather-daemonset-b4l56\" (UID: \"0f499d9c-c9b1-440d-bc80-ec355433452a\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:28.801483 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.801350    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f499d9c-c9b1-440d-bc80-ec355433452a-sys\") pod \"perf-node-gather-daemonset-b4l56\" (UID: \"0f499d9c-c9b1-440d-bc80-ec355433452a\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:28.801483 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.801374    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfw82\" (UniqueName: \"kubernetes.io/projected/0f499d9c-c9b1-440d-bc80-ec355433452a-kube-api-access-cfw82\") pod \"perf-node-gather-daemonset-b4l56\" (UID: \"0f499d9c-c9b1-440d-bc80-ec355433452a\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:28.893675 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.893575    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-c2btj"
Jul 16 23:31:28.901953 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.901913    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f499d9c-c9b1-440d-bc80-ec355433452a-lib-modules\") pod \"perf-node-gather-daemonset-b4l56\" (UID: \"0f499d9c-c9b1-440d-bc80-ec355433452a\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:28.902105 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.901968    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f499d9c-c9b1-440d-bc80-ec355433452a-proc\") pod \"perf-node-gather-daemonset-b4l56\" (UID: \"0f499d9c-c9b1-440d-bc80-ec355433452a\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:28.902105 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.902017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f499d9c-c9b1-440d-bc80-ec355433452a-podres\") pod \"perf-node-gather-daemonset-b4l56\" (UID: \"0f499d9c-c9b1-440d-bc80-ec355433452a\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:28.902105 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.902041    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f499d9c-c9b1-440d-bc80-ec355433452a-sys\") pod \"perf-node-gather-daemonset-b4l56\" (UID: \"0f499d9c-c9b1-440d-bc80-ec355433452a\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:28.902105 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.902074    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfw82\" (UniqueName: \"kubernetes.io/projected/0f499d9c-c9b1-440d-bc80-ec355433452a-kube-api-access-cfw82\") pod \"perf-node-gather-daemonset-b4l56\" (UID: \"0f499d9c-c9b1-440d-bc80-ec355433452a\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:28.902365 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.902109    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f499d9c-c9b1-440d-bc80-ec355433452a-lib-modules\") pod \"perf-node-gather-daemonset-b4l56\" (UID: \"0f499d9c-c9b1-440d-bc80-ec355433452a\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:28.902365 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.902140    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0f499d9c-c9b1-440d-bc80-ec355433452a-proc\") pod \"perf-node-gather-daemonset-b4l56\" (UID: \"0f499d9c-c9b1-440d-bc80-ec355433452a\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:28.902365 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.902140    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f499d9c-c9b1-440d-bc80-ec355433452a-sys\") pod \"perf-node-gather-daemonset-b4l56\" (UID: \"0f499d9c-c9b1-440d-bc80-ec355433452a\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:28.902365 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.902188    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0f499d9c-c9b1-440d-bc80-ec355433452a-podres\") pod \"perf-node-gather-daemonset-b4l56\" (UID: \"0f499d9c-c9b1-440d-bc80-ec355433452a\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:28.916232 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:28.916195    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfw82\" (UniqueName: \"kubernetes.io/projected/0f499d9c-c9b1-440d-bc80-ec355433452a-kube-api-access-cfw82\") pod \"perf-node-gather-daemonset-b4l56\" (UID: \"0f499d9c-c9b1-440d-bc80-ec355433452a\") " pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:29.045413 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:29.045377    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-c2btj"]
Jul 16 23:31:29.048820 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:31:29.048791    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16eddab7_3c0f_4bfd_ad4e_64dd2f4e9947.slice/crio-37f51d996885ae2333eb86a1dbd5d65894d069ff5dbad7dfc39aead2e2c8ee11 WatchSource:0}: Error finding container 37f51d996885ae2333eb86a1dbd5d65894d069ff5dbad7dfc39aead2e2c8ee11: Status 404 returned error can't find the container with id 37f51d996885ae2333eb86a1dbd5d65894d069ff5dbad7dfc39aead2e2c8ee11
Jul 16 23:31:29.068673 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:29.068652    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:29.202599 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:29.202575    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"]
Jul 16 23:31:29.204741 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:31:29.204715    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0f499d9c_c9b1_440d_bc80_ec355433452a.slice/crio-30970a8fe6fa2a0257b30a45cdff60ef3955985054a7216f90cd586eab373dbc WatchSource:0}: Error finding container 30970a8fe6fa2a0257b30a45cdff60ef3955985054a7216f90cd586eab373dbc: Status 404 returned error can't find the container with id 30970a8fe6fa2a0257b30a45cdff60ef3955985054a7216f90cd586eab373dbc
Jul 16 23:31:29.211135 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:29.211105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-c2btj" event={"ID":"16eddab7-3c0f-4bfd-ad4e-64dd2f4e9947","Type":"ContainerStarted","Data":"37f51d996885ae2333eb86a1dbd5d65894d069ff5dbad7dfc39aead2e2c8ee11"}
Jul 16 23:31:29.212362 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:29.212336    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56" event={"ID":"0f499d9c-c9b1-440d-bc80-ec355433452a","Type":"ContainerStarted","Data":"30970a8fe6fa2a0257b30a45cdff60ef3955985054a7216f90cd586eab373dbc"}
Jul 16 23:31:30.218977 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:30.218870    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56" event={"ID":"0f499d9c-c9b1-440d-bc80-ec355433452a","Type":"ContainerStarted","Data":"5d14c6348cbef568d60ff69a4a7c2ecd51549ff8b149d6fb9ce8c1d729a9c9d8"}
Jul 16 23:31:30.219426 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:30.219027    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:30.220402 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:30.220380    2565 generic.go:356] "Generic (PLEG): container finished" podID="16eddab7-3c0f-4bfd-ad4e-64dd2f4e9947" containerID="afcd647b35fe3be1bf743de71fd86872ff13e3fdfdf9139d63d138ad41e4da71" exitCode=6
Jul 16 23:31:30.220513 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:30.220464    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-c2btj" event={"ID":"16eddab7-3c0f-4bfd-ad4e-64dd2f4e9947","Type":"ContainerDied","Data":"afcd647b35fe3be1bf743de71fd86872ff13e3fdfdf9139d63d138ad41e4da71"}
Jul 16 23:31:30.220595 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:30.220581    2565 scope.go:117] "RemoveContainer" containerID="afcd647b35fe3be1bf743de71fd86872ff13e3fdfdf9139d63d138ad41e4da71"
Jul 16 23:31:30.234958 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:30.234914    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56" podStartSLOduration=2.234893465 podStartE2EDuration="2.234893465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 23:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:31:30.233740652 +0000 UTC m=+3500.982224722" watchObservedRunningTime="2026-07-16 23:31:30.234893465 +0000 UTC m=+3500.983377540"
Jul 16 23:31:31.227779 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:31.227685    2565 generic.go:356] "Generic (PLEG): container finished" podID="16eddab7-3c0f-4bfd-ad4e-64dd2f4e9947" containerID="ca226477b04f4ae427842582d337a1ee944509b9153d7eac1e121a6e86f01f80" exitCode=6
Jul 16 23:31:31.227779 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:31.227750    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-c2btj" event={"ID":"16eddab7-3c0f-4bfd-ad4e-64dd2f4e9947","Type":"ContainerDied","Data":"ca226477b04f4ae427842582d337a1ee944509b9153d7eac1e121a6e86f01f80"}
Jul 16 23:31:31.228361 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:31.227814    2565 scope.go:117] "RemoveContainer" containerID="afcd647b35fe3be1bf743de71fd86872ff13e3fdfdf9139d63d138ad41e4da71"
Jul 16 23:31:31.228361 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:31.227973    2565 scope.go:117] "RemoveContainer" containerID="ca226477b04f4ae427842582d337a1ee944509b9153d7eac1e121a6e86f01f80"
Jul 16 23:31:31.228361 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:31:31.228203    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-c2btj_odh-ai-gateway-infra(16eddab7-3c0f-4bfd-ad4e-64dd2f4e9947)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-c2btj" podUID="16eddab7-3c0f-4bfd-ad4e-64dd2f4e9947"
Jul 16 23:31:31.849296 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:31.849207    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bnnqb_25c597a0-b7b8-4ba5-96d5-e6155af0a3fd/dns/0.log"
Jul 16 23:31:31.868433 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:31.868405    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bnnqb_25c597a0-b7b8-4ba5-96d5-e6155af0a3fd/kube-rbac-proxy/0.log"
Jul 16 23:31:31.971979 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:31.971952    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cgrb2_bba53725-3faf-4d5c-8641-848a055f65df/dns-node-resolver/0.log"
Jul 16 23:31:32.234262 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:32.234171    2565 scope.go:117] "RemoveContainer" containerID="ca226477b04f4ae427842582d337a1ee944509b9153d7eac1e121a6e86f01f80"
Jul 16 23:31:32.234619 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:31:32.234377    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f263f2f2-c2btj_odh-ai-gateway-infra(16eddab7-3c0f-4bfd-ad4e-64dd2f4e9947)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f263f2f2-c2btj" podUID="16eddab7-3c0f-4bfd-ad4e-64dd2f4e9947"
Jul 16 23:31:32.438551 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:32.438515    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pwxmt"]
Jul 16 23:31:32.443281 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:32.443234    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pwxmt"
Jul 16 23:31:32.450992 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:32.450857    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pwxmt"]
Jul 16 23:31:32.536688 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:32.536609    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed77b17-ccb3-4a6b-b187-5cc8cdedb856-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-pwxmt\" (UID: \"fed77b17-ccb3-4a6b-b187-5cc8cdedb856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pwxmt"
Jul 16 23:31:32.596798 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:32.596757    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r7xmd_8bce50b5-5616-4134-b9ee-f1264bd569dd/node-ca/0.log"
Jul 16 23:31:32.637316 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:32.637278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed77b17-ccb3-4a6b-b187-5cc8cdedb856-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-pwxmt\" (UID: \"fed77b17-ccb3-4a6b-b187-5cc8cdedb856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pwxmt"
Jul 16 23:31:32.637951 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:32.637925    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed77b17-ccb3-4a6b-b187-5cc8cdedb856-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06b30c8f-pwxmt\" (UID: \"fed77b17-ccb3-4a6b-b187-5cc8cdedb856\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pwxmt"
Jul 16 23:31:32.755596 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:32.755561    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pwxmt"
Jul 16 23:31:32.899938 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:32.899905    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pwxmt"]
Jul 16 23:31:32.903128 ip-10-0-139-23 kubenswrapper[2565]: W0716 23:31:32.903097    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed77b17_ccb3_4a6b_b187_5cc8cdedb856.slice/crio-a8142f85eac037ca26eab90a8236331e413a6ce8337498cbe814a2f1ef687a7f WatchSource:0}: Error finding container a8142f85eac037ca26eab90a8236331e413a6ce8337498cbe814a2f1ef687a7f: Status 404 returned error can't find the container with id a8142f85eac037ca26eab90a8236331e413a6ce8337498cbe814a2f1ef687a7f
Jul 16 23:31:33.240453 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:33.240422    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pwxmt" event={"ID":"fed77b17-ccb3-4a6b-b187-5cc8cdedb856","Type":"ContainerStarted","Data":"a8142f85eac037ca26eab90a8236331e413a6ce8337498cbe814a2f1ef687a7f"}
Jul 16 23:31:33.544825 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:33.544748    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4n4xlt_62707e5b-445c-45ac-8d6a-3ef554d6e4b7/istio-proxy/0.log"
Jul 16 23:31:33.664127 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:33.664102    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-rx9kb_4151c7c9-fea2-41d0-93bf-a23d35a1e835/discovery/0.log"
Jul 16 23:31:33.734401 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:33.734375    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-l7jvp_adee9cd9-4926-40e7-a094-7cd9b70591cd/istio-proxy/0.log"
Jul 16 23:31:33.764547 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:33.764516    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-l7jvp_adee9cd9-4926-40e7-a094-7cd9b70591cd/istio-proxy/1.log"
Jul 16 23:31:33.796619 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:33.796536    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-gqxs8_8c7638b3-99dd-4860-9078-26e471cb24a6/payload-pre-processing/0.log"
Jul 16 23:31:34.247589 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:34.247554    2565 generic.go:356] "Generic (PLEG): container finished" podID="fed77b17-ccb3-4a6b-b187-5cc8cdedb856" containerID="8117e6a761ab6eddd993f37f7dcb90e49d757282bd01758093c669ebbae537a8" exitCode=6
Jul 16 23:31:34.248006 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:34.247636    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pwxmt" event={"ID":"fed77b17-ccb3-4a6b-b187-5cc8cdedb856","Type":"ContainerDied","Data":"8117e6a761ab6eddd993f37f7dcb90e49d757282bd01758093c669ebbae537a8"}
Jul 16 23:31:34.248006 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:34.247805    2565 scope.go:117] "RemoveContainer" containerID="8117e6a761ab6eddd993f37f7dcb90e49d757282bd01758093c669ebbae537a8"
Jul 16 23:31:34.506308 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:34.506206    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-frld8_2e1f153e-c668-492c-99df-6e50a746cbfb/serve-healthcheck-canary/0.log"
Jul 16 23:31:35.183361 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:35.183291    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dlbxx_86e283ba-586c-45f8-886e-80a3f0a56aa6/kube-rbac-proxy/0.log"
Jul 16 23:31:35.203639 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:35.203612    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dlbxx_86e283ba-586c-45f8-886e-80a3f0a56aa6/exporter/0.log"
Jul 16 23:31:35.225541 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:35.225516    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dlbxx_86e283ba-586c-45f8-886e-80a3f0a56aa6/extractor/0.log"
Jul 16 23:31:35.256538 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:35.256502    2565 generic.go:356] "Generic (PLEG): container finished" podID="fed77b17-ccb3-4a6b-b187-5cc8cdedb856" containerID="b7f421da88c0ec55a0294cbd1bc7850e65f4f3f44976ec07e47aaddf13cf2125" exitCode=6
Jul 16 23:31:35.257024 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:35.256540    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pwxmt" event={"ID":"fed77b17-ccb3-4a6b-b187-5cc8cdedb856","Type":"ContainerDied","Data":"b7f421da88c0ec55a0294cbd1bc7850e65f4f3f44976ec07e47aaddf13cf2125"}
Jul 16 23:31:35.257024 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:35.256604    2565 scope.go:117] "RemoveContainer" containerID="8117e6a761ab6eddd993f37f7dcb90e49d757282bd01758093c669ebbae537a8"
Jul 16 23:31:35.257024 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:35.256860    2565 scope.go:117] "RemoveContainer" containerID="b7f421da88c0ec55a0294cbd1bc7850e65f4f3f44976ec07e47aaddf13cf2125"
Jul 16 23:31:35.257189 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:31:35.257115    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-pwxmt_odh-ai-gateway-infra(fed77b17-ccb3-4a6b-b187-5cc8cdedb856)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pwxmt" podUID="fed77b17-ccb3-4a6b-b187-5cc8cdedb856"
Jul 16 23:31:36.237878 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:36.237853    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rvb4k/perf-node-gather-daemonset-b4l56"
Jul 16 23:31:36.263115 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:36.263091    2565 scope.go:117] "RemoveContainer" containerID="b7f421da88c0ec55a0294cbd1bc7850e65f4f3f44976ec07e47aaddf13cf2125"
Jul 16 23:31:36.263493 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:31:36.263287    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-06b30c8f-pwxmt_odh-ai-gateway-infra(fed77b17-ccb3-4a6b-b187-5cc8cdedb856)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06b30c8f-pwxmt" podUID="fed77b17-ccb3-4a6b-b187-5cc8cdedb856"
Jul 16 23:31:37.720777 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:37.720739    2565 scope.go:117] "RemoveContainer" containerID="3b894e4a7bfb22ed080e25859b89a370c2b27b96314a323979922398d918e0fd"
Jul 16 23:31:38.187437 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:38.187386    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv_32bfa590-021b-49b5-a94a-10661dd15166/manager/0.log"
Jul 16 23:31:38.205270 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:38.205217    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bb6b77ff4-4x8sv_32bfa590-021b-49b5-a94a-10661dd15166/copy-manifests/0.log"
Jul 16 23:31:38.275409 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:38.275321    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76" event={"ID":"7bd6aa26-8816-424d-a708-27ad317042b9","Type":"ContainerStarted","Data":"73cc10414f13f3facec6c121fddcd766483db8558d04d789c164ef064cdefeeb"}
Jul 16 23:31:38.683652 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:38.683623    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4n4xlt_62707e5b-445c-45ac-8d6a-3ef554d6e4b7/istio-proxy/0.log"
Jul 16 23:31:38.800799 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:38.800768    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-rx9kb_4151c7c9-fea2-41d0-93bf-a23d35a1e835/discovery/0.log"
Jul 16 23:31:38.872419 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:38.872386    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-l7jvp_adee9cd9-4926-40e7-a094-7cd9b70591cd/istio-proxy/0.log"
Jul 16 23:31:38.902360 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:38.902324    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-l7jvp_adee9cd9-4926-40e7-a094-7cd9b70591cd/istio-proxy/1.log"
Jul 16 23:31:38.922655 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:38.922570    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-gqxs8_8c7638b3-99dd-4860-9078-26e471cb24a6/payload-pre-processing/0.log"
Jul 16 23:31:39.282209 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:39.282120    2565 generic.go:356] "Generic (PLEG): container finished" podID="7bd6aa26-8816-424d-a708-27ad317042b9" containerID="73cc10414f13f3facec6c121fddcd766483db8558d04d789c164ef064cdefeeb" exitCode=6
Jul 16 23:31:39.282209 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:39.282189    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76" event={"ID":"7bd6aa26-8816-424d-a708-27ad317042b9","Type":"ContainerDied","Data":"73cc10414f13f3facec6c121fddcd766483db8558d04d789c164ef064cdefeeb"}
Jul 16 23:31:39.282478 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:39.282266    2565 scope.go:117] "RemoveContainer" containerID="3b894e4a7bfb22ed080e25859b89a370c2b27b96314a323979922398d918e0fd"
Jul 16 23:31:39.282574 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:39.282556    2565 scope.go:117] "RemoveContainer" containerID="73cc10414f13f3facec6c121fddcd766483db8558d04d789c164ef064cdefeeb"
Jul 16 23:31:39.282810 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:31:39.282789    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76_odh-ai-gateway-infra(7bd6aa26-8816-424d-a708-27ad317042b9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76" podUID="7bd6aa26-8816-424d-a708-27ad317042b9"
Jul 16 23:31:39.307352 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:39.307320    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76"]
Jul 16 23:31:39.561068 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:39.560984    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-84c74b69b8-p9m6n_d85de3f9-6def-4ec6-afb3-b5ca6415b331/manager/0.log"
Jul 16 23:31:40.132395 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:40.132363    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4n4xlt_62707e5b-445c-45ac-8d6a-3ef554d6e4b7/istio-proxy/0.log"
Jul 16 23:31:40.244041 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:40.244011    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-rx9kb_4151c7c9-fea2-41d0-93bf-a23d35a1e835/discovery/0.log"
Jul 16 23:31:40.307727 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:40.307699    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-l7jvp_adee9cd9-4926-40e7-a094-7cd9b70591cd/istio-proxy/0.log"
Jul 16 23:31:40.342582 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:40.342548    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-l7jvp_adee9cd9-4926-40e7-a094-7cd9b70591cd/istio-proxy/1.log"
Jul 16 23:31:40.368560 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:40.368532    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-gqxs8_8c7638b3-99dd-4860-9078-26e471cb24a6/payload-pre-processing/0.log"
Jul 16 23:31:40.448363 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:40.448338    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76"
Jul 16 23:31:40.507771 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:40.507681    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd6aa26-8816-424d-a708-27ad317042b9-maas-api-service-ca\") pod \"7bd6aa26-8816-424d-a708-27ad317042b9\" (UID: \"7bd6aa26-8816-424d-a708-27ad317042b9\") "
Jul 16 23:31:40.508046 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:40.508022    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd6aa26-8816-424d-a708-27ad317042b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7bd6aa26-8816-424d-a708-27ad317042b9" (UID: "7bd6aa26-8816-424d-a708-27ad317042b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:31:40.608695 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:40.608657    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bd6aa26-8816-424d-a708-27ad317042b9-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:31:41.294329 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:41.294300    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76"
Jul 16 23:31:41.294329 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:41.294314    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76" event={"ID":"7bd6aa26-8816-424d-a708-27ad317042b9","Type":"ContainerDied","Data":"055be084aff71875da2dea68507a14853dd8fd204df28a4a56ad711e413bdc66"}
Jul 16 23:31:41.294823 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:41.294378    2565 scope.go:117] "RemoveContainer" containerID="73cc10414f13f3facec6c121fddcd766483db8558d04d789c164ef064cdefeeb"
Jul 16 23:31:41.315454 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:41.315426    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76"]
Jul 16 23:31:41.323391 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:41.323366    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a1d48f5b-8kt76"]
Jul 16 23:31:41.721089 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:41.721050    2565 scope.go:117] "RemoveContainer" containerID="37e92b0a14873d17c87875d06675171310e5816aa84100cfec4ca1ccd40a0f75"
Jul 16 23:31:41.726858 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:41.726833    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bd6aa26-8816-424d-a708-27ad317042b9" path="/var/lib/kubelet/pods/7bd6aa26-8816-424d-a708-27ad317042b9/volumes"
Jul 16 23:31:42.301681 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:42.301647    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9" event={"ID":"6918622a-25c0-42c0-b380-41eb71557283","Type":"ContainerStarted","Data":"0b10ffb4aacb9c80bc0a292711862262109f7734b9733cfce8803139cd1fc73a"}
Jul 16 23:31:42.320078 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:42.320027    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9" podStartSLOduration=19.320010755 podStartE2EDuration="19.320010755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:31:42.317237324 +0000 UTC m=+3513.065721395" watchObservedRunningTime="2026-07-16 23:31:42.320010755 +0000 UTC m=+3513.068494826"
Jul 16 23:31:43.311834 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:43.311802    2565 generic.go:356] "Generic (PLEG): container finished" podID="6918622a-25c0-42c0-b380-41eb71557283" containerID="0b10ffb4aacb9c80bc0a292711862262109f7734b9733cfce8803139cd1fc73a" exitCode=6
Jul 16 23:31:43.312272 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:43.311841    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9" event={"ID":"6918622a-25c0-42c0-b380-41eb71557283","Type":"ContainerDied","Data":"0b10ffb4aacb9c80bc0a292711862262109f7734b9733cfce8803139cd1fc73a"}
Jul 16 23:31:43.312272 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:43.311929    2565 scope.go:117] "RemoveContainer" containerID="37e92b0a14873d17c87875d06675171310e5816aa84100cfec4ca1ccd40a0f75"
Jul 16 23:31:43.312397 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:43.312285    2565 scope.go:117] "RemoveContainer" containerID="0b10ffb4aacb9c80bc0a292711862262109f7734b9733cfce8803139cd1fc73a"
Jul 16 23:31:43.312519 ip-10-0-139-23 kubenswrapper[2565]: E0716 23:31:43.312498    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3933c193-5tlk9_odh-ai-gateway-infra(6918622a-25c0-42c0-b380-41eb71557283)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9" podUID="6918622a-25c0-42c0-b380-41eb71557283"
Jul 16 23:31:43.331517 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:43.331492    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9"]
Jul 16 23:31:44.451353 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:44.451325    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9"
Jul 16 23:31:44.544637 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:44.544606    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6918622a-25c0-42c0-b380-41eb71557283-maas-api-service-ca\") pod \"6918622a-25c0-42c0-b380-41eb71557283\" (UID: \"6918622a-25c0-42c0-b380-41eb71557283\") "
Jul 16 23:31:44.544960 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:44.544938    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6918622a-25c0-42c0-b380-41eb71557283-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6918622a-25c0-42c0-b380-41eb71557283" (UID: "6918622a-25c0-42c0-b380-41eb71557283"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:31:44.555586 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:44.555562    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-pplph_76d9a002-345d-4324-a645-71675213c52d/migrator/0.log"
Jul 16 23:31:44.583503 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:44.583476    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-pplph_76d9a002-345d-4324-a645-71675213c52d/graceful-termination/0.log"
Jul 16 23:31:44.646142 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:44.646110    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6918622a-25c0-42c0-b380-41eb71557283-maas-api-service-ca\") on node \"ip-10-0-139-23.ec2.internal\" DevicePath \"\""
Jul 16 23:31:45.324207 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:45.324177    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9"
Jul 16 23:31:45.324207 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:45.324186    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9" event={"ID":"6918622a-25c0-42c0-b380-41eb71557283","Type":"ContainerDied","Data":"c0128ab60e9be52616500eefddfb228d65b92b342fdee3949aa6a05a7010a8b4"}
Jul 16 23:31:45.324473 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:45.324278    2565 scope.go:117] "RemoveContainer" containerID="0b10ffb4aacb9c80bc0a292711862262109f7734b9733cfce8803139cd1fc73a"
Jul 16 23:31:45.347858 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:45.347830    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9"]
Jul 16 23:31:45.353934 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:45.353905    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3933c193-5tlk9"]
Jul 16 23:31:45.726061 ip-10-0-139-23 kubenswrapper[2565]: I0716 23:31:45.726026    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6918622a-25c0-42c0-b380-41eb71557283" path="/var/lib/kubelet/pods/6918622a-25c0-42c0-b380-41eb71557283/volumes"